From Black-box to Causal-box: Towards Building More Interpretable Models
Authors: Inwoo Hwang
Yushu Pan
Elias Bareinboim
Venue: NeurIPS 2025 poster ()
Keywords: Causal inference
Interpretability
XAI
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We develop a framework for building causally interpretable models where counterfactual queries can be consistently evaluated from a model and observational data.

Abstract: Understanding the predictions made by deep learning models remains a central challenge, especially in high-stakes applications. A promising approach is to equip models with the ability to answer counterfactual questions -- hypothetical ``what if?'' scenarios that go beyond the observed data and provide insight into a model reasoning. In this work, we introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models and observational data. We analyze two common model classes -- blackbox and concept-based predictors -- and show that neither is causally interpretable in general. To address this gap, we develop a framework for building models that are causally interpretable by design. Specifically, we derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query. This leads to a fundamental tradeoff between causal interpretability and predictive accuracy, which we characterize by identifying the unique maximal set of features that yields an interpretable model with maximal predictive expressiveness. Experiments corroborate the theoretical findings.

From Black-box to Causal-box: Towards Building More Interpretable Models
Inwoo Hwang
Yushu Pan Causal Artificial Intelligence Lab Columbia University
Elias Bareinboim
inwoo. www.hhawwnooobbbaaaa.. edu
fyushupan, 33-0cs.columbbln edu
Abstract
Understanding the predictions made by deep learning models remains a central challenge, especially in high-stakes applications. A promising approach is equip models with the ability to answer counterfactual questions - hypothetical "what if?" scenarios that go beyond the observed data and provide insight into a model reasoning, In this work, we introduce the notion of causal interpretability. which formalizes when counterfactual queries can be evaluated from specific class of models and observational data. We analyze two common model classes blackbox and concept- bansept-based predictors and show that neither is causally in- terpretable in general. To address this gap, we develop framework for building models that are causally interpretable by design. Specifically, we derive a complete graphical criterion that determines whether given model architecture supports a given counterfactual query. This leads to a fundamental tradeoff between causal interpretability and predictive accuracy, which we characterize by identifying the unique maximal set of features that yields an interpretable model with maximal predictive expressiveness. Experiments corroborate the theoretical findings.
Introduction
Despite the remarkable success of deep learning models across wide range of tasks 1 including image recognition [7, 16]. natural language processing [3, 35], and reinforcement learning [32, 34] these models remain fundamentally opaque. Although they are highly effective at predicting labels based on statistical correlations the data, they lack the capacity to explain the reasoning behind their predictions, earning them the colloquial label of "black boxes." " In other words, current models difficult to interpret: they lack the ability to justify why a particular decision was made, identify which input factors were most influential, reason about how outcomes might differ under alternative, counterfactual conditions. This interpretability gap raises concerns in high-stakes domains such as healthcare, law, and scientific discovery, where understanding how and why a model makes decision is as important as the decision itself.
A rich body of research explainable AI (XAI) has been developed better understand the behavior of learned models. For instance, post-hoc explanation methods such as LIME [30], SHAP [20], and Grad-CAM [31] generate local visual attributions in terms of pixels extracted features to help interpret predictions. Other approaches aim to build intrinsically interpretable models, such as those that impose sparsity constraints [22], restrict final layers [38], leverage decision tree structures [37], often trading off model complexity for greater transparency. While these techniques offer useful insights, they fail bridge the gap between low-level features and high-level, numan-understandable features that might explain the behavior of model.
One promising avenue for bridging this gap is counterfactual reasoning. Answering what if questions such as "Would the diagnosis have changed if a different treatment had been administered!" or
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Model Class
Attribute Prediction
Counterfactual Prediction
Ctf-consistent Class
Model A
Model B
person attractive?
What the one smiled?
Model
Model
Model
Model
Blackbox, Concept-based
Model
Model
Model C
What the was smiled?
Model
Modal
Model
Model
Model D
Model
Model
Consistent Prediction
Would the person be attractive whet
Consistent Estimation
Inconsistent
...?
Estimation
(a)
(b)
(c)
Figure 1: (a) Illustration different model classes: counterfactually consistent models (blue) and blackbox/concept-base models (yellow). (b) Original input image and corresponding predictions from each model. (c) Counterfactual predictions: models the top row predict consistently across instantiations within the class, while those in the bottom row produce inconsistent predictions.
"Would the person have been classified differently if their income were higher?" plays a central role in human reasoning and forms the basis of many explanatory and decision processes [1, 26, 27J. Enabling Al systems to reason counterfactually opens the door to more interpretable models ones that can not only predict outcomes accurately but also explain their decisions meaningful, human-aligned way.
Recently, concept- based prediction models [15, 23] have been proposed improve interpretability by enabling reasoning over human-ume undrrasadndttt features. These models aim to answer counterfactual queries of the form: "Given an input x, how would the model's prediction change feature W were modified from to w'?" Such queries allow users explore the influence of high-level features like the presence of smile or the existence of tumor on model's prediction, providing possible route to assess whether the model reasoning aligns with human expectations.
Despite their appeal, existing concept-based approaches are oblivious to the causal relationships between features. As result, they may not reflect the real-world mechanisms or incorporate common- sense knowledge faithfully. While some recent methods attempt introduce causal structure into concept-based models [4], they frequently lack guarantees of counterfactual consistency that is, the property that models within the exact class yield consistent answers the same counterfactual query.
To illustrate this limitation, consider a task of predicting facial attractiveness. Suppose two models, C and D, from the same concept-based class, represented by the yellow eircle in Fig. 1-(a), trained on the same dataset. They first will have the identical attribute prediction, for example, both will predict lower attractiveness score for the given image (Fig. 1-(c), yellow). However, when they evaluate the counterfactual question "What would the attractiveness be had the person smiled?", model will maintain the low attractiveness score while model D will raise the attractiveness score (Fig. 1-(c), yellow). This discrepancy reveals a deeper issue: the model class is not counterfactually interpretable, as does not constrain the space of counterfactual responses. In such cases, users have no principled way to determine which answer to trust, rendering the query effectively unanswerable. contrast, the model class in blue desirable since any pair of models such as model A and will give the exact same answer for both attribute and counterfactual predictions. In this case, one can assert that the attractiveness would be raised had the person smiled, which indicates the model made the decision based on the feature "Smile" and this is aligned with human understanding [8].

---

In this work, we introduce the notion of causal interpretability, which concerns whether a prediction model can be interpreted consistently across counterfactual scenarios drawing a connection between XAI and causal inference [1, 26]. Intuitively, a model class is said to be causally interpretable if all models within the class yield consistent predictions under counterfactual interventions, as illustrated in blue in Fig. 1. We then show that a blackbox model, which maps inputs directly to labels, is never causally interpretable. That is, such models fundamentally lack the structure needed to answer counterfactual questions. We also demonstrate theoretically that concept-based models [15], which rely on all observed features for prediction, are also not guaranteed to be causally interpretable. Interestingly, we show that causal interpretability can be recovered by constraining the model to use only a certain subset of features.
Against this background, we develop a general approach for building causally interpretable models that can answer counterfactual queries consistently by design. Specifically, we propose a complete graphical criterion for determining whether model that uses a given set of features for prediction causally interpretable with respect to counterfactual query. This enables the understanding of (i) which counterfactual questions a given model can answer, and (ii) which models can answer given counterfactual question. Our framework also reveals a fundamental tradcoff between causal interpretability and predictive accuracy. We characterize the unique maximal set of features that preserves causal interpretability, thereby providing a principled method for building models with maximal expressive power under interpretability constraints. A notable practical implication is that our approach does not require full specification of the causal graph or modeling of unobserved confounders; it only involves the descendants of the target features in the counterfactual query. Experimental results corroborate the proposed theory. More specifically, our contributions are as follows:
(Sec. 2) We introduce the notion of causal interpretability (Def. 2), which states whether we can evaluate the prediction of the model under counterfactual conditions from observational data. Based on this formulation, we show that blackbox model is never interpretable (Prop. 1), whereas concept-based model is also not interpretable in general, in contrast to prior belief.
(Sec. 3) We develop graphical criterion that determines whether the model is causally interpretable with respect to the query (Thm. 1). We characterize the unique maximal set of features yielding interpretable architecture (Thm. 2) and provide practical way of evaluating such queries from the data (Thm. 3). Finally, these results reveal a fundamental tradeoff between the causal interpretability and predictive accuracy (Thm. 4).
Preliminary. Here, we introduce notations and terminologies used in the paper. We use bold letters to denote a set of random variables or their assignments. We use capital letters to denote a random variable or a random vector (e.g.. X) and lower case letters denote their assignments (c.g.. x). x U Z denotes the subset of x corresponding to variables in Z and x \ Z denotes the value of X Z consistent with x.
We employ a structural causal model [1, 26] as our semantical framework. A structural causal model (SCM) M is a 4-tuple (U, V,F, P(U)), where U is set of exogenous variables, V {Vi,-... , Vn is a set of endogenous variables, F = [Jv, fv,, is a set of functions determining V V, t fv, (Pav,. Uv;). where Pay, S \ {Vy} and Uy; S U for all V, E V, and P(U) is distribution over U. An SCM M induces a causal diagram G and distribution over the endogenous P(V). We use graphical kinship to represent the relationships between the variables. N NDW) denotes non-descendants of a variable W, and ND(W) = Nweww D(W;) denotes non-descendants of set of variables W. We now define an SCM that describes generative process that includes images X and labels prediction Y [24].
Definition (Augmented SCM). An augmented SCM (ASCM) over a generative level SCM My = (U,, Fo. (U))) is a tuple M = (V, P(U)) such that
(1) exogenous variables U = (Uo, Ux}:
(2) V = Vo are labeled observed endogenous variables, X is an m-dimensional mixture variable, and Ý is a (predicted) label;
(3) = {Fo: J where fx maps from (the respective domains of) V U Ux to X and classifier fg maps from (the respective domains of) the subset of (V, X) to Y; and (4) P(U)) = p0 (U)).
3

---

F
S
X
X
(a) Blackbox prediction (BP). (b) Concept- based prediction (CP), (c) Generalized CP (GCP).
Figure 2: Causal diagrams for different types of predictive models.
An ASCM M represents sequential generative procedure of latent generative factors (i.e., concepts) V, the image X, and the label prediction Y. First, the latent features V are generated by the underlying My- The induced causal diagram Gv is called latent causal graph (LCG). The high- dimensional mixture X (e.g., image) is then generated from (and Ux), and subsequently, Y is generated from the subset of { (V, X3,, where fe a classifier that predicts the label. We let Q = {M ASCM over Mo} be the space of ASCMs. Omitted proof' are provided in Appendix A.2.
2 Causal Intepretabilityy Foundations
In this section, we formalize the notion of causal interpretability and examine whether existing approaches could elicit counterfactual questions consistently valid manner.
We start by analyzing two important classes of predictive models: blackbox and concept-based models. As illustrated in Fig. 2a, blackbox prediction (BP) models make prediction the label from the image pixels (i.e., fg= : D(X) + D(Y)). In contrast, concept-based prediction (CP) models predict the label based on the generative factors of the image (i.e., f& D(V) - D(Y)), as illustrated in Fig. 2b. In other words, the classifier of concept- baseed model uses the features to make the predictions, instead of the image itself. Formally, class of BP models and a class of CP
models respectively denoted as Qgp and Ωcp, where Ωgp = {M Ω fp D(X) D(Ý)}
and Ocr = {M E Ω fo : D(V) + D())). The following examples illustrate the generative process of BP and CP models.
Example (Blackbox Model). Consider task of estimating the attractiveness of human face represented in an image X. Augmented generative process (ASCM) of the prediction by a BP model is given as MBP = (U = =U'F, Us, Uc, Uc,, Ux}, LLF, X,,, JBP pBP (U)), where
F Up @ Us
S t Us
J.P
C (
(1)
X t (x(F,S,C,Ux)
Φ K fx(X),
Y is the label (attractiveness) prediction, the exogenous variables Ur. Us, Uc, are independent binary variables, and PBP (Ur = =) 0.4, PBP (Us = 0.6, PBP (Uc, =) = 0.3, p8P (Uc, 1) = 0.6. The exogenous variable Ux (representing other generative factors) can include (or be correlated to) {UF, Us, Uc,) The causal diagram induced by MBP is shown in Fig. 2a.
In terms of prediction. the process of obtaining Ý has three steps. First, latent generative features F (gender). (smiling), and (high cheekbones) are generated. Then, fx maps the observed generative features {F,S, U. C} and unobserved generative factors Ux to the images X in the pixel levels. Finally, the predictor fÿ takes these pixels as input estimate Y the corresponding model. The functions fx and fg can be aggregated as & t- f (x(F, C, Ux). This illustrates that the prediction of & by a BP model is made based on all observed features (F, S, C) and unobserved features Ux.

---

Ω
Ω
2
p.ft putt
Quet
Q
p.d
Q y Q
Data distribution
Query
Data distribution
Query
Figure 3: (Left) O' is causally interpretable if query be uniquely computed from the observa- tional data. (Right) A query cannot be uniquely computed from the observational data if Q' is not causally interpretable.
Example 2 (Concept-basee) Model). The main difference between the class of CP models Slep and the class of BP models Dap is the form of the classifter Jg- Consider the same generative process of observed features Vo = {F,S, C} and the image X in Ex. 1. Let us consider a CP model FCP Mce (U {Up, Us, UC, Uc Ux}, S,,,(F,,,,,,,, F (pC((U)) where the generative process of F, S. C, the same as Eq. (1), Y generated as
+ fy
(2)
and pCr(U)) is equal pBP(() in Ex. 1. In words, this means that instead of predicting Y based on pixels (i.e., image X), the classifier fo directly predicts Y based on observed features F, F,S.C C. The causal diagram induced by Mcr is shown in Fig. 2b.
Examples and 2 illustrate two different types of predictive models, where the classifier predicts the label directly from the image X (i.c.. QDp)) from the generative features V (i.e., ΩCp). While both types have showcased their capability to achieve reasonably high predictive accuracy in many domains [5, 7, 10, 11, 14-16, 23, 33, 40], it is unclear this moment whether we can interpret how they would predict under counterfactual scenarios, such as "how attractive the person would be had the been smiling?" ?". The following notion of causal interpretability formally states whether the counterfactual questions can be answered from the model.
Definition (Causal Interpretability). Consider specific model class Wi C Q where Si is the space of ASCMs. We say the class Q' is causally interpretable w.r.t. query Q if QMi QMa for WM11 S' s.t. P.M. (V, X, X,Y) PMMa(V
In words, S2' denotes certain design choice of the models for predicting the label, that is, it is space of prediction model candidates (i.e., model class). 52', for instance, can be Qap, when we want to predict the label directly from the image (Fig. 2a), or Qcp, when the classifier uses all observed features (Fig. 2b). For a query Q, we are concerned with counterfactual questions such as "What if the person had smiled?", which is written in counterfactual notion as P( X = x), and more generally Q(W) = PfPw X).2
In other words, the notion of causal interpretability states whether of the model under different counterfactual conditions. If the model is causally interpretable, the counterfactuals be evaluated from the observational data (Fig. 3, left). Otherwise, the model fundamentally cannot answer the counterfactual question from observational data, and thus, we cannot interpret their behavior under counterfactual scenarios (Fig. right). We now analyze two types of
understand the behavior
II practice, the annotations of the features are provided in many real- wolldwrrld datasets across various domains, e.g., human face [19], medical images [21]. and animal species [36]. Otherwise, the common practice is extract their annotations with vision-language models [29]. which shown effective [23, 39].
"Note that the definition is general in terms of the query Q. which could vary across different domains, e.g.. natural direct effect in fairness analysis [28].
5

---

predictive models discussed above (i.e., BP model in Ex. I and CP model in Ex. 2) and examine their causal interpretability, i.e., whether they can evaluate counterfactuals from observational data.
Example 3 (Continued from Ex. 1), Consider the BP model MBP in Ex. 1. Let Ux includes another
independent variable Us, namely, Ux= (Us: ) where Us L. U Us; let the observational quantity P(F = 0, S = 1,C =1| XX= = x) = 1, which means that the face is of a male (F = -), who is smiling (S 1), and with the cheekbones high (C = 1), given in an image X
The
generative process of Y is as Y fo o fx(F, Ux) 1[S > 0.5].
Consider another BP model Mar with the same generative process of MMP, but for in Mar the classifier f&g is given by: Y f o Ux) = 1 > 0.5). Since 5 = Us. the two BP models MBP and er agrees with the observational data, i.e.. PMm (V, X. 9) P.M (V.X.Y) XYY)). 8). which will lead to the same predictions (and corresponding accuracy).
Now, consider the counterfactual quantity "Given the image X = =x, would the prediction still be attractive (Y- = 1) had the person not smiled (S = 0)?", namely, Q(S) = P(Ys= = 1 X = =x). Intuitively. smaller value of P(Ys=o- 1 X x) implies the model is more reliable since changing a face to non-smiling reduces the attractiveness in general based on common sense knowledge [8]. For the first BP model MBP. Q(S) evaluates as P-Mar (Ys=0 = X
= x
0.5) = 0. However, for the second BP model M'p. Q(S) evaluates P.Mr (Ys=0- = 1|==00 x = 1 IUs = 1 > 0.5] = Details for these derivations are provided in Appendix A.
Note that each BP model evaluates the counterfactual query in a completely different way, and the two models are somewhat inconsistent. In practice, if one chooses the class of BP models Oer for this prediction task, the above counterfactual question cannot be answered correctly, since two BP models can give an exact opposite answer even if the two models agree perfectly with the observational distribution and their predictions. In other words, the blackbox model class cannot answer counterfactual Q(S) consistently from observational data, and its behavior cannot be interpreted under corresponding counterfactual conditions.
One may surmise that Fx. 3 is a pathological case, which for some reason does not allow the evaluation of counterfactual queries in a consistent manner. The next result shows that this is not the case for an arbitrary query QW) and a latent causal graph Gv-
Proposition 1 (Non-interpretability of BP). For any latent causal graph Gv. Qar is not causally interpretable w.r.t. Q(W) for any W V.
Given this impossibility results for the class of blackbox models, one may be tempted to believe that CP architecture is causally interpretable, as it predicts the label directly from the features where the unobserved factors Ux are filtered out. However, the following illustrates that this is not the case.
Example 4 (Continued from Ex, 2). Consider the CP model Mcr in Ex. 2. Similar to Ex 3, consider an observational quantity P(F = 0,S = 1, 1 X x) = 1. Y is generated as follows:
= 0.3].
(3)
Now consider another CP model Mce that is the same as McP, except for C < ['(() =
(Sv U NUC, and P(Uc, =1)= = 0.5. We have pMo (V,X,Y) pMi (V, YY and M'r compatible with the graphical constraints in Fig, 2b. Now consider the same counterfactual quantity
P(Ys-o x x) in Ex. 3. For Mcr. we have PMMrr PMMcrYSs- = 1| xx)= =P.crr (Cs=o 0.3. However, for the second CP model, PMoooYYs 1
x) (C=-o 0,-
1 F =
S, 1,C = 1) = 0.5. This implies that the two CP models are also inconsistent w.r.t Q(S). In other words, even prediction using features V, not pixels X. counterfactual queries induced by the CP models can still differ from each other.
3 A Causal Approach Towards More Interpretable Models
In this section, we establish principled way of understanding causal interpretability from a graphical point of view and propose generalized framework for building causally interpretable models.

---

3.1 Generalized Concept-based Models
We first define generalized concept-based prediction (GCP) models, a broader class that predicts the label from an arbitrary set of observed features.
Definition 3 (Generalized Concept-based Prediction). Let T C V be set of features that is used as a predictor of the label. That is, a classifier f& makes a prediction on a label based on T. We say such predictive models generalized concept-based models. A class of GCP models that employ the features T for prediction denoted as DCCP(T) = {M 2 fy D(T) - +DY)).
Compared to CP models, GCP models employ selected set of features T S V predictor of the label, which relaxes the requirement of CP where all features are considered.
The selection of the features T in GCP model should be specified during the model building stage, and our goal is to understand the implications of different choices of TT and which ones could lead to causally interpretable models (i.e.. satisfying Def. 2). To answer this question systematically, introduce a graphical criterion for determining whether a model satisties causal interpretability.
Theorem 1 (Graphical Criterion). Consider GCP models that employ a set features T as a predictor of the label. (CCP(T) is causally interpretable w.r.t. query Q(W) if and only if C W U UNW)..
In words, this result says that query Q(W) can be evaluated if the model uses the features among W or non-descendants of W to make a prediction on the label. In other words, the models that use any descendant of W cannot answer counterfactual question and no guarantee can provided on how they would make predictions under the corresponding counterfactual scenarios."
Thm. 1 enables
to identify the architectures (associated with T) that causally interpretable with respect to given counterfactual queries. Interestingly, the models that are potentially causally interpretable are not unique. The following formalizes the notion of admissible architectures.
Definition (T-Admissible Set). We say T is T-admissible w.r.t. W. (W1, W2,"..> if (OCP(T) interpretable w.r.t. Q(W,) for all W. E W.. A set of T-admissible sets w.r.t W, denoted as T....
To illustrate, T-admissible set represents model architectures that can answer (potentially multiple) counterfactual queries Q(W1), Q(W)) For example, in Fig. 2, eligible models that can evaluate Q(5) is GCP models whose classifier employs {S), {F), or {S. F} as a predictor of the ((S), (F), (S, F)).
label, i.e., T-admissible set corresponds to the query Q((S)) is T-Ad(
Given the multiplicity of admissible models, our goal to find the models that use as many features as possible to predict the label Y, i.c., maximal T, as it would be beneficial terms of predictive accuracy. We denote it maximal T-admissible set, which is formally defined below.
Definition 5 (Maximal T-Admissible Set). Suppose S E T-Ad(W.) and S' & T-Ad(W-)) for any S' 2 S. We denote such S as Max-T-Ad(W,, (.).
In other words, a maximal T-admissible set is a T-admissible set that would cease to be T-admissible if any additional variable were added to it. Note that once set is not T-admissible, adding more variables never makes it T-admissible again by Thm. 1. Identifying a maximal T-admissible set would lead to a model with maximal predictive power while retaining causal interpretability. One might suspect that multiple maximal T-admissible sets could exist, making it unclear which to select to maximize the predictive expressiveness. However, the next result says that this is not the case, since we can establish the uniqueness of the maximal T-admissible set.
Theorem 2 (Uniqueness of Maximal T-Admissible Set). For the queries Q(W.), a maximal T- admissible set unique and can be written as:
W.) = Nw.ww. (W;
(4)
Also, T E T-Ad(W.)) if and only T MMx-T-Ad(W.).
'Note that for the case of X = T, Qep is not interpretable w.r.t. any Q(W) since X is a descendant of W for any W C V, generalizing Prop. 1. Similarly, OoCrTT is also never interpretable if X E T, i.e. hybrid models that make predictions based on the combination of the image and features.
7

---

T (accuracy)
(accuracy)
Max- M
W.Ad(T:)
W-Ad(T:) - (ICC//1111C))))
M
TAs = 2
TI=
T-AMIN - 2/47
W-AaT)
T= (8.F)
I =
W. (interpretabilit))
W, (interpretabillt)
w! ((P)) w: ((S).
Figure 4: (Left) As we want a model to answer more counterfactual queries (W) C w3). the predic- tive power would decrease (Max-T-Ad E Max-T-Ad(W!.). (Right) As the predictive power increases (T1 C T2), interpretable counterfactuals would decrease (W-Ad(T2) C W-Ad(T,)).
To illustrate, for the group of queries Q(Wi), Q(W2),... ..., the maximal T-admissible set is unique and it is the intersection of non-descendants of W, plus W, Interestingly, identifying a maximal T-admissible set only requires the descendants of W and does not rely on the full specification of the causal graph. For example, given the features [cheekbone, smiling, gender) and the query "What if the person had smiled?". it only requires the knowledge of descendants of "smiling", which is "cheekbone". This does not rely on the full latent causal graph, which is often challenging to obtain.
An important practical implication of Thms and 2 is that, given query Q(W), one could incor- porate additional features as long as they are non-descendants of W, which would help improve accuracy while retaining the causal interpretability w.r.t. P(Yw X). For example, given the T-admissible set (smiling, gender) } and the query "Would the person be attractive had they smiled?",
one can incorporate additional features, e.g., age or hair color, that are non-descendants of smiling.
So far, have described how to find causally interpretable models that answer counterfactual queries. We now describe a practical way of evaluating such queries from the data.
Theorem 3 (Closed Form). If Pecprrr is causally interpretable w.r.t Q(W), the following holds:
P(Pw w x) PP I wT,,, (tt((| x).
(5)
This implies that the counterfactual quantity can be elicited from two-step prediction - (1) a classifier P(Y T) and (2) a feature extractor P(T X). For example, Q(S) introduced in Ex. can be computed using observational data and the maximal T-admissible set (S, F) as: P(Ys=o
X) = E., P( S O, f)ss, f X). Specifically, (S,F) are extracted from P(S, F X) and
the prediction is made by classifying p(ý S 0, F), conditioning $ O. Note that Eq. (5) only holds when the model is causally interpretable, and it does not hold for non-interpretable ones.
3.2 Fundamental Trade-Off between Causal Interpretability and Accuracy
So far, we have developed the machinery for building causally interpretable models that can answer counterfactual queries. Now, discuss which queries can be read from the given predictive model architecture. The following formalizes such notions of admissible queries.
Definition 6 (W-Admissible Set). We say W is W-admissible w.r.t. T if Qocr(T) is causally interpretable w.r.t. Q(W). A set of W-admissible sets w.r.t. T is denoted as W-Ad(T).
For example, in Fig. 2b, CP model that uses the features {F,S, C} as the predictor of the label can answer counterfactual queries ((C)), Q(F S)), (( C}) and Q(F,, C)). i.e. W-Ad([F, S, C}) Fig. 2c, we have W-Ad (=Ad((()C)) (s)) (C), might notice that the model using larger set of features answer smaller number of counterfactual questions. Our next result establishes a trade-off between accuracy and causal interpretability.
[(F)(C)) (F. (F.S.C) (F.. by applying Thm. Similarly, in (F.C). C), (F,S.)) Here, one
Theorem 4 (Causal Interpretability (i) T C T2, then W-Ad(T2) C W-).(TT). (ii) If W C W?. then Mex-T-Ad(Ws Max-T T-Ad(W:)
Trfde-Off). The following holds:
8

---

B
Accuracy (Label)
Est. eroo (Qr)
8
6
X
T=(8,D,C) T=(B.D) T-(D,C)
T=(D)
(b)
(c)
Figure 5: (a) Example images of BarMNIST dataset. (b) Causal diagram of GCP models. Red arrows
represent the possible usage for predicting the label. (c) Causal interpretabliy-aaccraay trade-off.
1.0
1.9
1.0
100
0.5
08
0.8
X
06
Extimated
Eximated Ground
0.0
(BDDC) (BD) (0.4)
(D)
(8.0.) (8.0) (b.C)
(D)
(BDDCC (B.D)
(0.4)
(D)
(8.0.C) (8.0)
(0.C)
(0)
(a) Changing digit.
(b) Changing color.
Figure 6: Estimation of counterfactual queries. Blue dots and orange marks denote estimation of counterfactual queries and ground truth value, respectively.
In other words, Thm. 4-(i) states that the counterfactuals that be evaluated from the model decrease (W-Ad(T,)) S W-Ad(T,)) as the predictors increase (T₁ C T2). Similarly, Thm. 4-(ii) states that the predictive power would decrease MMx-TT-AdWW (W)) S Max-T-Ad( as we want the models to answer more counterfactual queries (W) S W This reveals fundamental trade-off between causal interpretability and accuracy, where better predictive power would compromise the interpretability, and vice versa, as illustrated in Fig. 4.
4 Experiments
In this section, we evaluate our framework for estimating counterfactuals and compare it with prior approaches. Experimental details and additional experimental results are provided in Appendix B.
4.1 Synthetic datasets
We design the BarMNIST dataset [17, 24] where the digits are colored and bar appears at the top of the image, shown in Fig. 5a. Specifically, "color" (C), where D, C. are correlated and D has a direct causal effect on B, as illustrated in Fig. 5b. The true label is generated from all of the features and unobserved factors.
consider the features "bar" (B), "digit" (D), and
The dataset allows us compare the estimation of counterfactuals from each model with the ground- truth. We trained 4 different models, each using T = {B, D, C). {B. {B. D). (D, C), and (D) as predictor the label. shown Fig. 5c, the model using T (B. D, C} achieves the best accuracy. followed by T (B, D} and T (D, C) and the model using T = (D) shows the lowest accuracy. On the other hand, the best model (T = {B, D, CT) in terms of accuracy shows a high estimation eтrоr the counterfactual query of changing the digit. Thm. suggests that any estimation using observed data cannot capture the true counterfactual prediction of this model, since uses B, which is the descendant of D. For the same reason, T {B, D} is not causally interpretable, in contrast to T = {D.Cr C} and T = {D}. Our theory (Thm. 2) also suggests that there exists a unique maximal set of features that maintains causal interpretability, in this case, T = (D. C).
In Fig. 6, we take a closer look how these models estimate counterfactuals. As shown in Fig. 6a, T = (D,C) C} and T (D) are admissible models for the counterfactual query of changing the digit. On the other hand, for changing color (Fig. 6b), models are admissible and output a correct estimate of the counterfactual query, since C' is not descendant of any other features.

---

Obs
Ctf
Obs
Ctf
10
1.9
0.55
0.5
0.5
0.11
0.0
Obs
Ctf
Obs
Ctr
Figure 7: Visualization of interpreting counterfactual predictions CelebA examples.
4.2 Real-world datasets
CelebA dataset [19] contains human face images with the annotations on facial expressions and attributes, such "smiling". "age", "gender", etc. We consider model predicting the label "attractiveness" and examine how a model makes prediction under counterfactual conditions "Would the person look attractive had they smiled?". In the real world, is impossible to observe counterfactual outcome, but our theory allows us to interpret the behavior of (causally interpretable) models under counterfactual conditions. Based on Thm. 1, we choose the features that are not the descendants of smiling. Fig. 7 illustrates the counterfactual prediction of the model using non-descendant features (i.e.. "smiling" and "gender"), We can interpret its behavior under the counterfactual condition that it predicts higher attractiveness had the one smiled, which is aligned with human common sense.
5 Conclusion
In this work, we introduced the notion of causal interpretability, which states whether counterfactual queries can be evaluated from a model and observational data. By examining commonly used model classes blackbox and concept-based models we demonstrated that neither is causally interpretable. this end, we developed a graphical criterion that determines whether the model is causally interpretable with respect the query (Thm. 1). We characterize the unique maximal set of features yielding interpretable architecture (Thm. 2) and provide a practical way of evaluating such queries from the data (Thm. 3). Our results reveal fundamental tradeoff between the causal interpretability and predictive accuracy (Thm. 4). Theoretical findings are corroborated by the experimental results. Additional discussions and limitations provided in Appendix C.
Acknowledgments and Disclosure of Funding
We thank anonymous reviewers for their constructive comments. This research supported in part by the NSF, ONR, AFOSR, DoE, Amazon, JP Morgan, and The Alfred P. Sloan Foundation.
References
[1] Elias Bareinboim, Juan D. Correa, Duligur lbeling, and Thomas Icard. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl, page 507-556. Association for Computing Machinery, New York, NY, USA, 2022.
[2] Juan Correa, Sanghack Lee, and Elias Bareinboim. Nested counterfactual identification from arbitrary surrogate experiments. Advances in Neural Information Processing Systems, 34: 6856-6867, 2021.
[3] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.dm 04805, 2018.
[4] Gabriele Dominici, Pietro Barbiero, Mateo Espinosa Zarlenga, Alberto Termine, Martin Gjoreski, Giuseppe Marra, and Mare Langheinrich. Causal concept graph models: Beyond causal opacity in deep learning. arXiv preprint 2055 2024.
10

---

[5] Gabriele Dominici, Pietro Barbiero, Francesco Giannini, Martin Gjoreski, Giuseppe Marra, and Mare Langheinrich. Counterfactual concept bottleneck models. In The Thirteenth International Conference Learning Representations, 2025.
[6] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models: Beyond the eecuracy-epplainablii trade-off. Advances in Neural Information Processing Systems,
2022.
[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[8] Simone Horn, Natalia Matuszewska, Nikolaos Gkantidis, Carlalberta Verna, and Georgios Kanavakis. Smile dimensions affect selfeeeree smile attractiveness. Scientifie reports. 11 (1):2779, 2021.
[9] Inwoo Hwang, Yesong Choe, Yeahoon Kwon, and Sanghack Lee. On positivity condition for causal inference. In International Conference on Machine Learning, pages 20818-20841. PMLR, 2024.
[10] Aya Abdelsalam Ismail, Tuomas Oikarinen, Amy Wang, Julius Adebayo, Samuel Don Stanton, Hector Corrada Bravo, Kyunghyun Cho, and Nathan C. Frey. Concept bottleneck language mod- els for protein design. In The Thirteenth International Conference on Learning Representations, 2025.
[11] Sujin Jeon, Hyundo Lee, Eungseo Kim. Sanghack Lee, Byoung-Tae Tak Zhang, and Inwoo Hwang. Locality-aware concept bottleneck model. arXiv preprint arXiv:2508. 14562, 2025.
[12] Yonghan Jung, Jin Tian, and Elias Bareinboim. Estimating identifiable causal effects through double machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 121 1211222 2021.
[13] Yonghan Jung, Iván Díaz, Jin Tian, and Elias Bareinboim. Estimating causal effects identifi- able from a combination of observations and experiments. Advances Neural Information Processing Systems, 36:46446 2023.
[14] Eunji Kim, Dahuin Jung, Sangha Park. Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. In International Conference on Machine Learning, pages 16521-16540. PMLR, 2023.
[15] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference machine learning, pages 5338-5348. PMLR, 2020.
[16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, 25, 2012.
[17] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun, com/exab/mnistt/ 1998.
[18] Adam Li, Yushu Pan, and Elias Bareinboim. Disentangled representation learning in non- markovian causal systems. Advances in Neural Information Processing Systems, 37:104843- 104903, 2024.
[19] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, (5(208)):11, 2018
[20] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[21] Michael Nevitt, David Felson, and Gayle Lester. The osteoarthritis initiative. Protocol for the cohort study, 1:2, 2006.

---

[22] Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19) 2011.
[23] Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023.
[24] Yushu Pan and Elias Bareinboim. Counterfactual image editing. In International Conference on Machine Learning, pages 39087-39 PMLR, 2024.
[25] Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669-688, 1995.
[26] Judea Pearl. Causality, Cambridge university press, 2009,
[27] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
[28] Drago Plecko and Elias Bareinboim. Causal fairness analysis. arXiv preprint arXiv:2207.11385, 2022.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 0763.773..26 PmLR, 2021.
[30] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
[31] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[32] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):4844455) 2016.
[33] Chung-En Sun, Tuomas Oikarinen, Berk Ustun, and Tsui Wei Weng. Concept bottleneck large language models. In The Thirteenth International Conference Learning Representations, 2025.
[34] Richard S Sutton, Andrew G Barto, al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, pages 5998-6008 2017.
[36] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 20rd5-200-2011 dataset. 2011.
[37] Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, and Joseph E Gonzalez. Nbdt: Neural-backed decision trees. arXiv preprint 2tt.::2004000221.. 2020.
[38] Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In International Conference on Machine Learning, pages I 11205- 11216. PMLR, 2021.
[39] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19187-19197 2023.
[40] Mert Yuksekgonul, Maggie Wang, and James Zou, Post-hoc concept bottleneck models. The Eleventh International Conference Learning Representations, 2023.
12

---

Appendix
Proofs and Additional Examples
13
A.1 Derivations Examples
13
A.2 Omitted Proofs
14
A.3 Additional Examples
17
Experiments
18
B.1 Dataset
18
B.2 Experimental Details
19
B.3 Additional Experimental Results
19
C Additional Discussions, Limitations, and Future Work
21
A Proofs and Additional Examples
A.1 Derivations Examples
A.1.1 Derivation in Ex. 3
In Ex. 3, for the first BP model MBP. we evaluate Q(S) from MBP as follows:
MM (Ys Ys
X X=
Σ P.Mar = F [aa
1.e.0
=p-Mar
=p-Mar
=1 0 0.5] =0
However, for the second BP model, we evaluate Q(S) from M
(5s- = X
-PMin (1..
-P-Mor (1[Us > =
Σ PMMs > 0.5]s=0 0 Mer -
(summing over U)
- (1[Us > 0.5] = Us = 1)
(S = Us)
=1(Us = >>>000 0.5 1.
13

---

A.1.2 Derivation in Ex. 4
In Ex. 4, for McP,
pMo (Yso = X X x)
=PMo
F -,5--,,--1,,--
=PMo = == =
Y V)
E pMar (Y5=0= = Cs=o [0
-
(Ys=0 PMoo
Cs=o c -
-PMe F 0,5 1 1,0
(Eq. 3)
=0.3
A.2
Omitted Proofs
In this section, we present the proofs of our theoretical results in Sec. and 3. We first formally introduce the causal diagram induced by an SCM.
Definition 7 (Causal Diagram [1, Def. 13)). Consider an SCM M (U, V P(U)). We construct a graph 9 using M as follows:
(1) add vertex for every variable in
(2) add a directed edge (V, + Vi) for every V4, V, V if V, appears as an argument of fv, E F,
(3) add bidirected edge (V, h- Vor every Vi, V1,,, E V if the corresponding Uv, Uv; S are not independent or if fv; and Jv, share some U E as an argument.
We refer to G the causal diagram induced by M (or "causal diagram of M" for short).
We then formally introduce the identifiability of counterfactual query given an observational distribution and a causal diagram G.
Definition (Counterfactual Identification). counterfactual query P(yi(x)): 92[xa]:--) ...) said be identifiable from P(V) and if ( 32.x uniquely computable from the distributions P(V) in any SCM that induces G.
Then start from two lemmas
tool for the proof of Thm. 1.
Lemma 1. Consider an SCM M over V. Suppose that there exists a path made entirely of bi-directed = 0. Let the intervened
edges between VG, V, E V in G. Consider two sets A, B C and values are not consistent with the factual values, namely, b g Then the query P(ap v) is identifiable from P(V) and G and only if C ND()), where ND(B) = (
Proof. (=) Suppose A C (BB). We have P(ab v) P(a v) 1/a v] which implies that P(ab v) is uniquely computable.
(+) Suppose there exists A E A such that A E Des(B). By Correa et al. [2, Thm. 3], P(ap v) is an inconsistent factor since C and b v, and thus, is not identifiable from P(V).
Lemma (Correa al. [2, Lemma. I]). Consider an SCM over V induce observational distribution P(V) and diagram
G. Suppose Ag takes input as A₁. Then En, P(Ai A2bb))...) is identifiable
if and only if P(A A2]b2])...) is identifiable.
Now, we are ready proceed to the proof of Thm.
Theorem 1 (Graphical Criterion). Consider GCP models that employ set ofatures T as a predictor of the label. P is causally interpretable w.r.t. query Q(W) if and only if WUND
14

---

V
T
Figure 8: Diagrams used in the proof of Thm.
X
Figure 9: Diagrams used in the proof of Prop. 1.
Proof. According Defs 2, and this is equivalent to prove that query P(Tw: x) is identifiable if C (D(W) U W given the observational distribution P(V, X, 9) and the diagram GAuE over [V, X, Y} (shown in Fig. 8). To illustrate, the diagram & over V is an arbitrary given DAG; for any V1 V, V, point to X and bi-directed connected to X; only subset T V point to Y. Denote
Z = T.W.
P(Ww' [x)
P(Dw v.x))((v Ix)
(summing over V)
Σ P(Ow V,
Pvv x)
(summing over Tw in Mw world)
v
Σ P(Ow (.))((t [v,x) |x)
(Yw 1 (V.X) Tw
(6)
v,8"
Σ P(Tw )P(")(' /v,x)((v x)
(consistency)
Σ "00lz"'''(Pe"" w') \v, x) V.2"
(do-calculus [25])
(8)
Eq. (6) holds since P,r are independent with X and V since all parents of fw (which is Tw) are conditioned Eq. (7) holds since the T W should be consistent with the intervened value in
(and the remaining variables Z in taking z". Eq. (8) holds due Y 7 W T in Gw, where Gw the graph removing outgoing edge of W. Using do-calculus, have:
P(ow (=w)) = P(7 z" w').
(9)
We will prove that Eq. (8) is identifiable and only if S ND(W)UW, which is equivalent prove Eq. (8) is identifiable iff Z C N ((W) since Z = T W. According to Eq. (8), the only undermined term P(z v. x). Since V and X are bi-directly connected, Lemma suggests P(* x) is identifiable iff Z C D(W). Then, P( z". P') P(v x) is identifiable iff C NDW.). According Lemma Eq. (8) identifiable iff C ND(W)) U.
Proposition (Non-interpretabiliiyy of BP). For any latent causal graph Gv. Dap causally interpretable w.z.. Q(W)foo any W V.
15

---

Proof. Since the observational P(X) is identifiable, we will prove that P(ow, x) is not identifiable given blackbox model structure and observational distribution P(X, Y V).
p(Iw' x)
Σ P(O x x)
(summing over Xw
Σ P(Ox) ) ;(x
(consistency and Yx 1 Xw
Σ P(Dx))
Σ P(xw'; X, w)
(summing over W)
(10)
P( w) is not identifiable according Lemma Then Lemma 2 suggests that P(ow x) is not identifiable.
Theorem 2 (Uniqueness of Maximal T-Admissible Set). For the queries Q(W+), maximal T admissible set unique and can be written as:
Max-T-A Max-T-Ad(W.y, (W.) = -wwww. (W,UND(W..).
(4)
Also, T E (-..(W.) if and only T Max-T-Ad(W.)..
Proof. First, we will show that Ow.ww, (W;UND(W.)) is T-admissible set w.r.t Q(W.). For each W, W., we have
Ow.Ew. (W:UND(W.)) S W;UNDDW..).
Therefore, by Thm. 1. L.,, (W,UND(W:)) is T-admissible set w.r.t Q(W,) for all W; W.. Thus, we have S E T-Ad(W.).
(ii) Now, will show that S is maximal T-admissible w.r.t W.. Suppose there exists S' such that S' E TT-A((W.) and S' 2 S. Since S' E (T-Ad(W.), S' E T-Ad(W,) for all W, E W.. Hence,
S' C WWUNDWW. for all W, W.
Therefore, S' nww.w. (W,UND(W.) S. which contradicts S' 2 S. Therefore, S is maximal T-admissible set w.r. W..
(iii) Now, will show that S is unique maximal T-admissible set. Suppose there exists another maximal T-admissible set S'. Since S' E TT-d(W..). we have S' C S by the same reason in (ii). S' C S, then it contradicts that S' is maximal T-admissible set, since S is T-admissible set. Therefore, we have S = S'. In other words, maximal T-admissible set is unique and can be written Max-T-Ad((.))
Ow.ew, (W; NDD(.))..
(iv) Now, we will show that T E T-Ad(W.) if and only if T C Max-T-Ad(W.... Suppose T E T-Ad(W.). Then, by (ii), we have T S Ow.ew. (W,UND that Max-T-Ad(W:
Also, we showed Nw;ew, (W.UND(W.)) Therefore, we have T C Max-T-Ad(W: (.... Now, suppose that T C Max-T-Ad(W.). We have T S Ow,ew, (W, ND(W,))) and thus, T C W, (DW.) for all W, E W.. Therefore, T E T-Ad(W,) for all W; E W., and thus, T T-Ad(W.).
Theorem 3 (Closed Form). If MoCcrTT is causally interpretable w.n.. Q(W), the following holds:
P(ýw xx) EP w' (T, W)P(t x).
(5)
Proof. From Eq. (8), we have
P(Tw x) Σ P(0 w.r.ww)PP w') (v,x)((v /x) xx.
(1I)
v.x""
16

---

Note that this equation is identifiable if only if Z S (UND.W). Then
Σ V,8"
P z,
P' (v x)
Σ P(O
(Lemma. 1)
v.x"
Σ P(O tb (. w')(((
(where (t) w) E
Σ W. x)
YY W T
Σ w. t, nt) |x).
(12)
This conclude P(Pw. xx)= = E. w w'T P(t x) since Eq. 12 holds for any t, w.
Theorem (Causal Interpretability-Aceeeo Accuracy Trade- Trade-Off). The following holds: (i) IT C 22 then 2- C i).
(ii) If W. C W?, then Max-T-Ad((E)) Max-T-Ad(W!).
Proof. (i) Let T1 C T2. Suppose W 2-) By Def. and Thm. have
T2 S W WUNDDW)
Since T1 S T2, it follows that T1 S W U WWWDDWW. Therefore, by Def. 6 and Thm. 1, W e W-Ad(T1). Thus, for all W E 2-) we have W E W-Ad(T,). Hence, we have
W-Ad(T2) C W-Ad(T,).
(ii) Let W C W?. Then, we have
Ow,ww; (W, UND((W)) E Cww.ew (W,UND Therefore, we have Max-T-Ad W? Max-T-Ad(W!) by Thm. 2.
A.3 Additional Examples
The following example illustrates how GCP and CP models compare.
Example 5 (GCP), Consider the generative process of observed concepts Vo =0= (F,S,C) and the image X, as in Ex. I (BP model) and Ex. (CP model). Consider GCP model MecP {U {UF, UC Uc,, Ux] LLF,S, X, 8), FGCP pocp (U)), where
+ UP
St Us
FGCP
C+ (-SA UC, (S
(13)
X+ (x(F,S,C,Uxx)
8 + (SC)
and PGCP (U) equal to (U) in Ex. The causal diagram induced by GCP model Meer shown in Fig. 2c. To illustrate, instead of predicting the label based pixels in images X (BP models) or all observed features { F, S, C} (CP models), GCP model makes prediction using selected subset of features T (S, F) (i.e., smiling and gender) in this case.
The following example illustrates the case where the GCP model is causal interpretable.
S
P(F,S,C) =1
0
0.168
Example 6 (Continued from Ex. 5). Consider Ocp in Ex. 4. Thm. suggests Qey is not interpretable w.r.t. to query Q(S) P(Y==o X). This is because C E De(S), where
0
0.072
0.096
1 0.144
0
0.112
0 0,048
17
1 0.144
1 0.216
Table Probability table Ex.

---

= i... the prediction Y is made based on C, descendant of S. In contrast, (GCP(( in Ex. 5 is said be causally interpretable w.r.f. to query Po X) since f7 only takes T F} _ S u (D())
imput. To illustrate, let us consider the GCP model Moer Similar to Examples and 4, let the observational
quantity P(F 0, = 1,C 1 x) and let f& be:
GCP FS, 1|S + 0.5). 18
(14)
Now, consider another GCP model
Mcce (U' [U', U', U/, U2, U' U (1P,S,)) x, JOCP pocr (U)), (15)
where
F UF
S (
J.CP
C+ (+
(16)
X (x(,,
p + 1S F> 0.5]
and P(U; 1) P(U', 1) 0.5, P(U)s) =1 = -9/13. P(U,, = =) 0.5, P(U
1) = 0.6. It is verifiable that P.Mer (V) = P.Mer (V) as shown in Table 1. Since fr is the same in both Mger and Maccr P.ace (M f) pMMer (V, Y). Let the distribution of Ux satisifies that PMacr (V.X. ".XYY Y) P.Mar (V.X. Y). MMp is compatible the graphical constraints induced by the model Fig. 2b. Notice that f'p, fs. f' M'acr are totally different to fr. fs. fc in Mocr- For the first GCP model MaCP.
P.Mac (Ysco =
P.Macr FF
Similarly, for the second GCP model M'GcP.
pMacr
= = -
This shows that the two GCP models are consistent with the query. In other words, if one uses the features {S, F} to predict & the model architecture in Fig. 2c is guaranteed to provide a unique answer for the counterfactual question "What would the attractiveness prediction be had the person not smiled?" (i.e., P(Ys=o X)). Then one can trust the counterfactual quantities induced by any model with this architecture.
B Experiments
In this section, we describe the details for the experiments and provide additional experimental results.
B.1 Dataset
B.1.1 BarMNIST
For BarMNIST experiment discussed in Sec. 4.1, the data generating process is as follows:
D +- Up
C + UC
F
B B (UB, A D) @ (UB, AUB)) ( ((-UB,)
(17)
X + D,C,
Y +- ((D@ B) $ Uy
18

---

P(Ù = =
P(ù =0o=:=1|C=1,0-11
C
1.0
1.0
0.6
0.6
x
0.4
0.4
0.2
sttimaedd Truth
0.2
X
0.0
Baseline
0.0
Baseline
Ours
(a) Causal diagram.
(b) Estimation of counterfactuals.
Figure 10: (a) Causal diagram of GCP models. Red arrows represent the possible usage for predicting the label. (b) Estimation of counterfactual queries. Blue dots and orange marks denote estimation of counterfactual queries and ground truth value, respectively.
the exogenous variables Un, Uc, UB, U2 UBg Uy are independent binary variables, and P(Up 1) = 0.5, P(Uc = 1) = 0.4, P(UB, = 1) = 0.9, P(UB, = 1) 1/18, = 1)
=1)
Following this process, we generated 60,000 images and corresponding labels, where each image is annotated with 3 binary features, i.e., bar (B), color (C), and digit (D). Here, D = 0 represents the digits from to 4 and represents the digits from 5 to 9.
B.1.2 CelebA
CelebA dataset [19] contains 202,599 celebrity facial images, where each image is annotated with 40 different attributes. In our experiments, used the attribute "attractiveness" as the label, where the label and all other features are binary.
B.2 Experimental Details
In BarMNIST, used ResNet18 for the feature extractor. For the classifier, used three-layer MLP with the hidden dimension of 32 and leakyrelu activation. We set the batch size to 1024 and trained the models for 100 epoch. We used Adam optimizer with learning rate of 0.0003.
In CelebA, we used ResNet34 for the feature extractor and used linear classifier. We set the batch size to 512 and trained the models for 100 epochs. We used SGD optimizer with the learning rate of 0.001. We resized the image with center crop into 64x64 64 for training.
For the training of our model and baselines, we used binary classification loss for both the feature extractor and the classifier, where they are trained simultancously in an end-to-end manner. All experimental results are averaged over independent runs. We report a standard error as the error bar in Figs. 6, 10 and 11. All experiments are conducted on single NVIDIA A100 GPU. For the implementation, we utilized publicly available code from Espinosa Zarlenga et al. [6]. We used GPT-4o generate the counterfactual images shown in Figs. and 11 to provide an intuitive understanding of the counterfactual questions.
B.3
Additional Experimental Results
B.3.1 BarMNIST
To validate theory with different graph structure, consider causal diagram in Fig. 10a where the goal is to predict the digit D from the image. The data generating process as follows:
UB t BVUG, @ Uc, F
(18)
(BVC) Up
X (x,,,DCC,Ux)
where the exogenous variables Up, Uc, Uc, Up are independent binary variables, where P(UB 0.6, P(Uc, =1) D.5,P(UU 1) 0.1, P(U 0.1.
19

---

(a) Query causal prior
(c) Instance-level examples
Image
Counterfactual
Altrativenees
#itraciiarees
Attrctvarres
Query: P( P = x)
0.66
6.6t
"Would the person look attractive had they been smiling?"
Attactivstae
Atraccharees
Attactivnnae
Smile - Cheekbones
Smile Mouth Opened
0.27
0.42
6:15
3
3.25
0.22
Attrativeress
Atractivnees
Atractvaraa
(b) Population-level statistics
1.42
Alctr factual)
21
0.15
36
0.30
0.5
0:26
increase attractiveness
Figure 11: (a) We examine the prediction of the models under counterfactual condition. We use causal prior knowledge that smiling has causal effects on the features "checkbones" and "opened mouth". (b) Average difference between the estimated counterfactual prediction and the prediction on the observed (factual) image. (c) Qualitative examples for our model and baselines.
The baseline model uses the features B and C' for predicting the label, and model uses B for making a prediction. Our theory (Thm. I) suggests that our model is causally interpretable, but not the baseline which uses C, descendant of B. We compare model and baselines for estimating the counterfactual prediction of the model, where the query is to change the bar, i.e., P(D B=0 x).
Fig. 10b illustrates the estimation of counterfactual queries (blue dots) and ground truth values (orange marks). This shows that our model correctly estimates counterfactual queries. In contrast, the estimation of the baseline significantly differs from the ground truth. This corroborates our theory that estimation can properly interpret the counterfactual behavior of the causally interpretable models, but it is not possible for non-interpretable ones.
B.3.2 CelebA
Here, we provide detailed analysis of CelebA experiments in Sec. 4.2. Fig. 11-(a) illustrates the counterfactual question and causal prior we utilized to construct our model. Specifically, we leverage the common-sense knowledge that smiling has direct causal influence to the features "checkbones" and "opened mouth". To construct our model, we choose features that are non-descendants smiling, specifically "smile" and "gender" as feature set V. Baselines include descendant features. In Fig, 11, baseline uses the features "smiling", "gender", and "cheekbones" and baseline uses the features "smiling", "gender", "cheekbones" and "opened mouth".
Fig. 11-(b) shows the average difference between estimated counterfactual prediction and the prediction the observed image. Fig. 11-(c) shows qualitative examples comparing our method and baselines. The first column in Fig. 11-(c) shows the input image, and the second column illustrates the counterfactual image, reference to provide a better understanding of the counterfactual query.
20

---

The theory suggests that a causally interpretable model can properly estimate its prediction under counterfactual conditions. As shown Fig. 11-(b) and (c), our model, which is causally interpretable, consistently increases the attractiveness across the instances, which is also aligned with human reasoning. In contrast, as illustrated in Fig. II-(c), the estimation of the baselines (which use similar feature set as ours) shows that smiling often does not increase attractiveness (red boxes). In fact, our theory suggests that is not possible to interpret the counterfactual behavior non-interpretable models using only observational data, and any attempts to estimate it would lead to inconsistent results.
C Additional Discussions, Limitations, and Future Work
Estimation of the concepts. In the closed-form formula in Eq. (5), the concepts W and T are ground-truth concepts. Since the labels of ground-truth concepts are available, one can estimate P(T X) over ground truth concept T. For clarity, let denote this estimated distribution as P(T X). In the prediction stage, the true concepts W and T an image instance X are not given directly. Instead, the predicted concepts W and T are sampled through the estimated P(T X). When P(T X) is accurate, the sampled (predicted) concepts are expected to align closely with the ground-truth concepts. However, if the estimation has an error, the predicted concepts may deviate from the true ones, and this error will naturally propagate into the counterfactual evaluation via Eq. (5).
Our goal with this formulation is to formally characterize how these counterfactual quantities can be computed from the observational distribution under ideal conditions (i.e., accurate estimation). The challenge of robustly estimating P(T) X) from finite data is indeed fundamental and highly relevant to practice, but falls outside the scope of this work. Nevertheless, it would be valuable direction for future investigation, particularly in light of ongoing research in counterfactual estimation within the causal inference literature [12, 13] and the importance of creating more interpretable methods in practice.
Causal graph. Our work reveals that understanding and harnessing causal relationships among the generative features are crucial for building interpretable models that properly evaluate counterfactual questions. It is important to note that our framework only requires the causal prior on the descendants, and this is much relaxed assumption compared the conventional assumption in causal inference, where the full specification of the causal graph is needed [9, 18].
Real-world datasets. In real-world datasets, it is infeasible evaluate the actual value of the counterfactual query because the underlying ground-trun truth data-generating process for real-world datasets is not given, specifically, the mechanisms of V are not known. For example, it is unknown how nature decides the generation process of human facial features. Due to this inevitable restriction, thoroughly validated our theory in BarMNIST datasets (where have the ground-truth SCM), including causal interpretability-acoo -ccuurccy tradeoff.
Still, our theory allows us to understand the interplay between causal interpretability and accuracy in real-world datasets. For example, given T-admissible set smiling, gender and the query "Would the person be attractive had they smiled?", if one wants to incorporate additional query "Would the person be attractive had they be a men?", we know the model using this T maintains causal interpretability w.r.t. both queries, and thus, would not compromise accuracy.
21

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction state our main claims. Theoretical results are presented in Sec. and and experimental results are presented in Sec. 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitation section is provided in Appendix C.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: Proofs are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Experimental details provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We provide the experimental details in Appendix B.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wttps.cc/ppblccc//eccccccpcppppp getic//guuee...e CoodeSummii agionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are provided in Appendix B. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Fig. 6 includes error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Experimental details are provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //t///nurspp..//
Answer: [Yes]
Justification:
We reviewed the code of ethics and followed it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss societal impacts in Appendix C.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cited the original paper of the dataset or code.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
27

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https 220 for what should or should not be described.
28


Reviews:
[{"review_id": "TFPaVTmsM7", "reviewer": "Reviewer_ic77", "summary": "The authors present the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a model and observational data. To do so, they introduce a particular framework for testing which family of models is causally interpretable. In their inquiry, they analyse black-box models as well as concept-based models, which turn out to never be causally interpretable (former) and can not be causally interpretable (latter). They then derive the criteria for ensuring causal interpretability and test their theory on two different tasks.", "questions": "Questions are listed in the **Weaknesses** section.", "limitations": "Limitations of the framework are explicit, for they are to be found in the SCM paradigm that is presented.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strenghts**\n\n-I like the presentation of the (A)SCM paradigm. I saw a few of them, but this one is truly clear. On that same line of thought, I feel the quality of the presentation as a whole is excellent, with a good and efficient use of the nine pages.\n\n-Though the accuracy-interpretability tradeoff is a sensible topic ([1], for example), I feel it is well defended here.\n\n[1] Stop explaining black box machine learning models for high-stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 5 (2019), 206–215.\n\n**Weaknesses**\n\n1.1. I feel like sentences such as « In this work, we introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a model and observational data » (lines 5-7) are misleading. One could benefit from counterfactuals that do not always lead to the same outcome in the change of the prediction. For instance, one could examine the effect of the smile on attractiveness, considering that some people have an unattractive smile, teeth, etc. I feel like sentences such as « In this work, we introduce the notion of causal interpretability, which concerns whether a prediction model can be interpreted consistently across counterfactual scenarios » (lines 66-7) are more faithful to what is actually done in this work. So I think every presentation of causal interpretability of the first form should be changed to that of the second form.\n\n1.2. On that same line of thought, one could say that having a counterfactual to always lead to the same outcome in the change of prediction (having the predictions to be monotonic regarding the counterfactual) is somewhat restricting and leads to an interpretation that is somewhat of lesser importance. For instance, in information theory, the relevance of information is inversely proportional to the probability that the event the information conveys is true. Thus, having counterfactuals resulting in similar outcomes is of lesser importance.\n\n2. « This means that a black box does not have access to any of the causal factors that generated the data » (line 125). Wouldn’t it be truer to say that we do not know whether the black-box model has access to any causal factors that generated the data? The black box model could, in its hidden inner workings, retrieve the features before making a prediction accordingly.\n\n3. Why assume that concept-based models require the use of every concept? I could be wrong, but I don’t think this is necessarily a need for concept-based models.\n\n**Typos and such**\n\n- « ND » (line 236), though it is clear from the following that it refers to the concept of « non-descendance », should be defined properly.\n\n- « In other words, a maximal T-admissible set is a T-admissible set that would cease to be T-admissible if any additional variable were added to it » (line 256); I would add for clarity that once a set is not T-admissible, adding one of many variables can never make it T-admissible again.\n\n-Line 243 : « are not be »\n\n-Line 305 : Should cite the original MNIST paper, since the proposed dataset is a derivation of MNIST.\n\n-Figure 4 (c) : Feels weird to have accuracy bars and error bars next to each other on the same graph. I would be better to be consistent with the considered metric.\n\n-At a few places in the article, an « interpretability-accuracy » tradeoff is mentioned and defended (lines 12, 291, 292, Fig. 4 (c) caption), whereas at some other places, a « causal interpretability-accuracy » is defended (lines 80, 94, 280, 298-9, 340). Since the article focuses on causal interpretability, I would only discuss the tradeoff involving this kind of interpretability, since the tradeoff has not been defended for interpretability as a whole and accuracy.\n\n-« a concept-based model is also often not interpretable » (line 89); I believe Example 4. illustrates that this could be the case, not that it is often the case. « We also demonstrate theoretically that concept-based models, which rely on all observed features for prediction, are also not guaranteed to be causally interpretable. » (line 72-3) is more faithful to the work.", "was_revised": true, "final_justification": "I don't feel that the criticisms raised by Reviewer xsrJ are sufficient to reject this article. The assumptions underlying the SCM structures are standard. Concerning the help in understanding the decision, working by elaborating counterfactuals is also standard practice. I stand by my score.", "rebuttal": "We appreciate the reviewer’s time and feedback provided to improve the manuscript. We respond to your comments as follows:\n\n---\n\n> **[W1.1]** I feel like sentences such as « In this work, we introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a model and observational data » (lines 5-7) are misleading. One could benefit from counterfactuals that do not always lead to the same outcome in the change of the prediction. [..] I feel like sentences such as « In this work, we introduce the notion of causal interpretability, which concerns whether a prediction model can be interpreted consistently across counterfactual scenarios » (lines 66-7) are more faithful to what is actually done in this work.\n> \n\nThank you for pointing this out and we appreciate the opportunity to clarify our intent. We agree that the phrasing on lines 66–67 (“whether a prediction model can be interpreted consistently across counterfactual scenarios”) more accurately reflects the core idea behind our notion of *causal interpretability*.\n\nTechnically, causal interpretability refers to the property that any model within a given model class (models using the same set of variables for prediction, i.e., compatible with a specified causal diagram) that matches the observational distribution will also evaluate counterfactuals consistently. We acknowledge that evaluating counterfactuals using an individual fully specified SCM is important in some contexts.\n\nOur original phrasing on lines 5–7 was intended to convey *“when counterfactual queries can be evaluated from a specific model class and observational data.”* To avoid confusion, we have revised this wording throughout the manuscript to better align with the more precise formulation from line 66.\n\n> **[W1.2]** On that same line of thought, one could say that having a counterfactual to always lead to the same outcome in the change of prediction (having the predictions to be monotonic regarding the counterfactual) is somewhat restricting and leads to an interpretation that is somewhat of lesser importance.\n> \n\nWe appreciate this thoughtful comment. We agree with your point that *“the relevance of information is inversely proportional to the probability that the event the information conveys is true.”* However, we believe that the consistent evaluation of counterfactuals within a model class are still very valuable.\n\nAs we illustrate in Example 4, when counterfactuals are not consistently determined by the observational distribution alone, evaluating them requires full specification of the structural mechanisms (e.g., $f_C$) and the distribution over exogenous variables (e.g., $U_C$). In contrast, when *causal interpretability* holds (as formalized in Theorem 3), counterfactuals can be computed using a closed-form expression that depends solely on the observational distribution ($P(\\mathbf{X, V}, \\widehat{Y})$); no knowledge of the SCM’s mechanisms is required.\n\nFrom an information perspective, one could say that causal interpretability allows us to answer counterfactual queries with *less* information. In this sense, the values of causal interpretability here are not maximizing information content, but by minimizing the informational burden required to perform reliable reasoning about counterfactuals.\n\n> **[W2]** « This means that a black box does not have access to any of the causal factors that generated the data » (line 125). Wouldn’t it be truer to say that we do not know whether the black-box model has access to any causal factors that generated the data?\n> \n\nYes, we agree with the reviewer, and we will revise the sentence for better clarity.\n\n> **[W3]** Why assume that concept-based models require the use of every concept? I could be wrong, but I don’t think this is necessarily a need for concept-based models.\n> \n\nWhile this is not the restriction, concept-based models typically use all concepts they collected.  Specifically, concept-based models first predefine the concept set $\\mathbf{V}$ (for example, cheek, smile, and gender in CelebA), and use all these predefined concepts in $\\mathbf{V}$ to predict the label. In our paper, we point out that to ensure causal interpretability, one should take causal relationships among the concepts into account to determine the concept set that is used for classification. We will revise the text for better clarity.\n\n> **[Typos and such]**\n> \n\nThanks for the valuable suggestions! We will incorporate the reviewer’s comments into the revised manuscript, e.g., adding the definition of non-descendants and using the consistent terminology of causal interpretability-accuracy tradeoff.\n\n---\n\nWe appreciate the meaningful discussion and would be happy to provide further clarification. Thank you!"}, {"review_id": "ANFGaj2ERg", "reviewer": "Reviewer_zwPZ", "summary": "The work introduces the concept of counterfactual consistency and show how black-box models but also concept-based models are not counterfactually consistent in general. It introduces conditions on the set of concepts to use that provably guarantee counterfactual consistency.", "questions": "- Ground-truth concepts are typically not available at prediction time. Does this affect the formulation in any way? can you clarify this? \n\n- It would be interesting to relate the t-admissible set to the actionability of the concepts, as in some applications one could be willing to incorporate non-actionable concepts for improving accuracy even if the model would not be counterfactually consistent with respect to them, provided consistency is guaranteed on actionable concepts. Can you comment on this?", "limitations": "Limitations are not discussed in the paper. However, the experimental evaluation is extremely limited, and the impact of the accuracy-interpretability tradeoff in real applications is left to be understood.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths \n\nThis is an interesting contribution in the field of concept-based models, where recent approaches are trying to explore additional conditions for interpretability (e.g. disentanglement). Given the relevance of counterfactual explanations in interactive analysis of predictive models, this contribution can pave the way to the development of more consistent models. \n\n# Weaknesses\n\nOne concern I have about the formalization is the lack of distinction between ground-truth concepts and predicted concepts. The formulation seems to imply the availability of ground-truth concepts at prediction stage.   \n\nThe experimental evaluation is very limited, with quantitative results only provided for a proof-of-concept mnist-based dataset. While this is mostly a theoretical paper, measuring the impact of the accuracy-counterfactual interpretability tradeoff in real datasets would strenghten the contribution.", "was_revised": true, "final_justification": "I believe this is a solid piece of work on a relevant topic, and I am confident that authors can easily address my requests for clarifications and discussion on the limitations concerning real-world experiments.", "rebuttal": "We sincerely thank the reviewer for the constructive and thoughtful feedback. We respond to your comments as follows:\n\n---\n\n> **[W1, Q1]** One concern I have about the formalization is the lack of distinction between ground-truth concepts and predicted concepts. The formulation seems to imply the availability of ground-truth concepts at prediction stage. / Ground-truth concepts are typically not available at prediction time. Does this affect the formulation in any way?\n> \n\nIn the closed-form formula Eq. (5), the concepts $\\mathbf{W}$ and $\\mathbf{T}$ are ground-truth concepts. Since the labels of ground-truth concepts are available, one can estimate $P(\\mathbf{T} \\mid \\mathbf{X})$ over ground truth concept $\\mathbf{T}$. For clarity, let us denote this estimated distribution as $\\widehat{P}(\\mathbf{T} \\mid \\mathbf{X})$.\n\nIn the prediction stage, the true concepts $\\mathbf{W}$ and $\\mathbf{T}$ of an image instance $\\mathbf{X}$ are not given directly. Instead, the predicted concepts $\\mathbf{W}$ and $\\mathbf{T}$ are sampled through the estimated $\\widehat{P}(\\mathbf{T} \\mid \\mathbf{X})$. When $\\widehat{P}(\\mathbf{T} \\mid \\mathbf{X})$ is accurate, the sampled (predicted) concepts are expected to align closely with the ground-truth concepts. However, if the estimation has an error, the predicted concepts may deviate from the true ones, and this error will naturally propagate into the counterfactual evaluation via Equation (5). We will include this discussion in the revised manuscript.\n\nOur goal with this formulation is to formally characterize how these counterfactual quantities can be computed from the observational distribution under ideal conditions (i.e., accurate estimation). The challenge of robustly estimating $P(\\mathbf{T} \\mid \\mathbf{X})$ from finite data is indeed fundamental and highly relevant to practice, but falls outside the scope of this work. That said, we agree it would be a valuable direction for future investigation, particularly in light of ongoing research in counterfactual estimation within the causal inference literature and the importance of creating more interpretable methods in practice.\n\n> **[W2]** While this is mostly a theoretical paper, measuring the impact of the accuracy-counterfactual interpretability tradeoff in real datasets would strengthen the contribution.\n> \n\nIn real-world datasets, it is infeasible to evaluate the actual value of the counterfactual query because the underlying ground-truth data-generating process for real-world datasets is not given, specifically, the mechanisms of $\\mathbf{V}$ are not known. For example, it is unknown how nature decides the generation process of human facial features. Due to this inevitable restriction, we thoroughly validated our theory in BarMNIST datasets (where we have the ground-truth SCM), including causal interpretability-accuracy tradeoff.\n\nStill, our theory allows us to understand the interplay between causal interpretability and accuracy in real-world datasets. For example, given T-admissible set {smiling, gender} and the query \"Would the person be attractive had they smiled?\", if one wants to incorporate additional query \"Would the person be attractive had they be a men?\", we know the model using this $\\mathbf{T}$ maintains causal interpretability w.r.t. both queries, and thus, would not compromise accuracy.\n\n> **[Q2]** It would be interesting to relate the t-admissible set to the actionability of the concepts, as in some applications one could be willing to incorporate non-actionable concepts for improving accuracy even if the model would not be counterfactually consistent with respect to them, provided consistency is guaranteed on actionable concepts.\n> \n\nThanks for this insightful note. The intuition to add non-actionable variables for improving accuracy is correct and can be formally grounded by the T-admissible set and Thm. 1. Our theory suggests that, given the actionable concepts $\\mathbf{W}$, one could incorporate additional non-actionable concepts as long as they are non-descendants of $\\mathbf{W}$, which would help improve accuracy while retaining the causal interpretability w.r.t. $P(\\widehat{Y}_{\\mathbf{w}}\\mid \\mathbf{X})$. For example, given the T-admissible set {smiling, gender} and the query ``Would the person be attractive had they smiled?'', one can incorporate additional concepts (including non-actionable concepts), e.g., age or hair color, that are non-descendants of smiling. We will include this discussion in the revised manuscript.\n\n---\n\nWe appreciate your valuable comments and suggestions. Please let us know if you have further feedback!"}, {"review_id": "ERHmPdq9bQ", "reviewer": "Reviewer_xsrJ", "summary": "This paper introduces a formal notion of **causal interpretability**, which captures when a prediction model can consistently answer counterfactual “what‐if” queries from observational data. The authors demonstrate that standard black-box predictors (which map directly from inputs to labels) and concept-based models (which utilize all observed features) generally fail to meet this interpretability criterion. To address this gap, they propose **Generalized Concept‐based Prediction (GCP)** models, which leverage a carefully selected subset of features for prediction. The key theoretical contribution is a **graphical criterion** that characterizes precisely which feature sets enable causal interpretability with respect to a given counterfactual query. Building on this, the paper proves the **uniqueness of the maximal admissible feature set**, which optimally balances predictive expressiveness under interpretability constraints. It also derives a **closed-form expression** revealing a fundamental tradeoff between interpretability and accuracy. Experiments on two image datasets provide empirical support for the theoretical results.", "questions": "The paper defines causal interpretability as “models within the exact model class yield consistent answers to the same counterfactual query.” What precisely is meant by *model class* in this context? For black-box and concept-based models, how should we determine whether two models belong to the same class? The definition could benefit from further clarification.\n* In Example 4, the reported value of $\\mathcal{M}_{cp}$ seems to be 0.3 rather than 0.2. Please verify this value.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n* The paper rigorously addresses the question: under a counterfactual assumption involving a feature set $W$, will a concept-based model constructed on feature set $T$ behave consistently? Theorem 1 formalizes the necessary condition that $W$ and $T$ must satisfy, offering theoretical insights into concept-based modeling and counterfactual reasoning.\n\nWeaknesses\n\n* While the proposed notion of causal interpretability helps formalize how a model behaves under counterfactual assumptions, it offers limited practical guidance for improving a model’s interpretability. Specifically, it does not explain how the model derives its predictions from inputs, nor does it substantively advance the paper’s stated goal of “bridging the gap between low-level features and high-level concepts” (line 33).\n* The experimental evaluation is relatively simplistic and lacks comparisons with existing black-box models or alternative concept-based methods, weakening the empirical validation.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for spending the time reviewing our paper. We believe that some misunderstandings of our work made the evaluation overly harsh, and we hope that you can reconsider our contributions based on the clarifications provided below. \n\n---\n\n> **[Q1]** The paper defines causal interpretability as “models within the exact model class yield consistent answers to the same counterfactual query.” What precisely is meant by model class in this context? For black-box and concept-based models, how should we determine whether two models belong to the same class?\n> \n\nThe model class in our paper refers to the set of all functions or hypotheses that a particular type of model can represent. $\\Omega_\\text{BP}$ is a model class that includes all black-box models that use the image $\\mathbf{X}$ to predict the label (lines 124-125) (using the image $\\mathbf{X}$ to predict the label is the particular type). A (classic) concept-based model class, $\\Omega_\\text{CP}$, refers to all predictive models that use all concepts $\\mathbf{V}$ for label prediction (lines 126-127). Generalized concept-based model class, $\\Omega_{\\text{GCP}(\\mathbf{T})}$, refers to all predictive models that use the feature set $\\mathbf{T}$ for prediction (lines 227-228). \n\n> **[W1-1]** While the proposed notion of causal interpretability helps formalize how a model behaves under counterfactual assumptions, it offers limited practical guidance for improving a model’s interpretability.\n> \n\nWe respectfully disagree with this point and would like to clarify that our notion of causal interpretability, particularly through Theorem 1, does offer practical guidance for model design. Specifically, Theorem 1 provides a graphical criterion for determining whether a predictor can consistently evaluate counterfactual queries based on observational data.\n\nTo illustrate, causal interpretability formalizes the property that a class of models can evaluate counterfactual questions consistently. Thm. 1 suggests that causal interpretability can be achieved if and only if the set of concepts $\\mathbf{T}$ obeys the given graphical rules (certain causal relationships among the generative features). Thus, when aiming to design a predictor, one should consider involving such $\\mathbf{T}$ in the model and exclude features that violate the graphical conditions. With this design, the predictor is capable of evaluating human-understandable counterfactual questions based on observational distribution and thus is more interpretable. This is a fundamental building block that ties causality, interpretability, and machine learning, and we expect it to be extended to a larger class of models across a wide range of domains.\n\n> **[W1-2]** Specifically, it does not explain how the model derives its predictions from inputs, nor does it substantively advance the paper’s stated goal of “bridging the gap between low-level features and high-level concepts” (line 33).\n> \n\nWe would like to clarify that our method contributes to bridging the gap between low-level features and high-level concepts *for answering counterfactual questions*. Specifically, unlike many post-hoc explanation methods that focus on low-level parameters or raw inputs (such as pixels), existing concept-based models map low-level instances to high-level features and attempt to answer counterfactual questions over high-level concepts. However, these models are not guaranteed to provide consistent answers to counterfactual questions.\n\nAs we illustrated in our paper (lines 53–65 and Figure 1), two models (C and D) can give completely opposite answers to the counterfactual question: “What would the attractiveness be had the person smiled?” In such cases, users cannot determine which answer to trust. Our framework addresses this issue by ensuring that all models within the same generalized concept-based family provide consistent answers to counterfactual queries. This consistency in counterfactuals enables users to assess whether the model reasoning aligns with human expectations.\n\nIn summary, our framework inherits the ability to map pixels (low-level) to concepts (high-level) from existing concept-based models, and further closes the gap by enabling these high-level concepts to be used in answering counterfactual questions.\n\n> **[W2]** The experimental evaluation is relatively simplistic and lacks comparisons with existing black-box models or alternative concept-based methods, weakening the empirical validation.\n> \n\nFirst, for the black-box models, which are the mappings from the image $\\mathbf{X}$ to the (predicted) label $\\widehat{Y}$ that do not involve generative factors $\\mathbf{W}$, it is infeasible to evaluate this query through a black-box model. Even if additional operations are introduced to a black-box model to extract and access $\\mathbf{W}$, Example 3 demonstrates that black-box models can still lead to inconsistent answers.\n\nSecond, our experiments include the comparison with a concept-based model (that uses the features $\\mathbf{V}$ to predict the label) and existing concept-based methods do not fall outside this. We would like to note that our main theoretical results involve how a different choice of a feature set $\\mathbf{T}$ impacts the causal interpretability, and thus, we evaluated the models using different $\\mathbf{T}$, where a concept-based model is a specific case where $\\mathbf{T}=\\mathbf{V}$.\n\nFinally, we would like to note that in real-world datasets, it is infeasible to evaluate the actual value of the counterfactual query because the underlying ground-truth data-generating process for real-world datasets is not given. For example, it is unknown how nature decides the generation process of human facial features. Acknowledging this inevitable restriction, BarMNIST datasets (where we have the ground-truth SCM) allow us to thoroughly validate our theory.\n\n> **[Q2]** In Example 4, the reported value of $\\mathcal{M}_\\text{CP}$ seems to be 0.3 rather than 0.2. Please verify this value.\n> \n\nYes, it is a typo and the value should be 0.3. The two models in the example are inconsistent with the query (0.3 and 0.5). Thanks for pointing it out!\n\n---\n\nWe hope this addresses your questions, and please let us know if there are any remaining concerns!"}]

Author rebuttal:

Meta-review: (a) Theoretical paper formulating when models can be causally interpretable with counterfactual queries, considering prior concepts like concept-based models.  This results in a theorem characterising the result.
(b)  Theoretical formulation of the problem and a theorem about a necessary condition.  Technically solid.
(c)  Limited empirical work of limited value.  How does one use this theory in building better causal models?   Definitions are heavy and need simpler explanation.  Explanations given are minimalist and not always helpful.
(d) Two accepts and the overall notion seems useful, with good rebuttals. Its an interesting characterisation of the problem yielding insights into causality, but needs better explanations.
(e) Healthy discussion with reviewers.  The one rejecting reviewer raised some good questions prompting strong confidential comments by the author, but I think better explanations in the paper would help.
Decision: Accept (poster)