Explaining the Behavior of Black-Box Prediction Algorithms with Causal Learning
Authors: Numair Sani
Daniel Malinsky
Ilya Shpitser
Venue: Accepted by TMLR ()

Abstract: Causal approaches to post-hoc explainability for black-box prediction models (e.g., deep neural networks trained on image pixel data) have become increasingly popular. However, existing approaches have two important shortcomings: (i) the “explanatory units” are micro-level inputs into the relevant prediction model, e.g., image pixels, rather than interpretable macro-level features that are more useful for understanding how to possibly change the algorithm’s behavior, and (ii) existing approaches assume there exists no unmeasured confounding between features and target model predictions, which fails to hold when the explanatory units are macro-level variables. Our focus is on the important setting where the analyst has no access to the inner workings of the target prediction algorithm, rather only the ability to query the output of the model in response to a particular input. To provide causal explanations in such a setting, we propose to learn causal graphical representations that allow for arbitrary unmeasured confounding among features. We demonstrate the resulting graph can differentiate between interpretable features that causally influence model predictions versus those that are merely associated with model predictions due to confounding. Our approach is motivated by a counterfactual theory of causal explanation wherein good explanations point to factors that are “difference-makers” in an interventionist sense.

Published in Transactions on Machine Learning Research (05/2025)
Explaining the Behavior of Black-Box Prediction Algorithms with Causal Learning
Numair Sani*
numairsani@@maail.cmm
Sani Analytics, Mumbai, MH India
Daniel Malinsky'
mmdknsky&rolkubbiic
Department of Biostatistics, Colambia University, New York, NY USA
Ilya Shpitser
ilyasGcs MaaaaOessjh..... edu
Department of Computer Science, Johns Hopkins University, Baltimore, MD USA
Reviewed on OpenReview: https: // optp://eepervvee...mm net/ forum? id ZrqLpXbXuA
Abstract
Causal approaches to post-hoc explainability for black-box prediction models (c.g., deep neural networks trained on image pixel data) have become increasingly popular. How- ever, existing approaches have two important shortcomings: (i) the "explanatory units" are micro-level inputs into the relevant prediction model, e.g.: image pixels, rather than inter- pretable macro-level features that are more useful for understanding how to possibly change the algorithm's behavior, and (ii) existing approaches assume there exists unmeasured confounding between features and target model predictions, which fails to hold when the explanatory units are macro-level variables. Our focus is on the important setting where the analyst has no access to the imner workings of the target prediction algorithm, rather only the ability to query the output of the model in response to a particular input. To provide causal explanations in such a setting, we propose to learn causal graphical represen- tations that allow for arbitrary unmeasured confounding among features. We demonstrate the resulting graph can differentiate between interpretable features that causally influence model prodictions versus those that merely associated with model predictions due to confounding. Our approach is motivated by counterfact ual theory of causal explanation wherein good explanations point to factors that are "difference cdifference-makers" makers" in an interventionist sense.
1 Introduction
In recent years, black-box artificial intelligence (AT) machine learning (ML) prediction algorithms have exhibited impressive performance in wide range of prediction tasks. In particular, methods based on deep neural networks (DNNs) have been successfully used to analyze high-dimensional data in settings such healthcare and social data analytics (Esteva et al. 2019 Guimaraes et al. 2017). An important obstacle to widespread adoption of such methods, particularly in socially-impactful settings, is their black-box nature: it not obvious, in many cases, how to explain the predictions produced by such algorithms when they succeed (or fail), given that they find imperceptible patterns among high-dimensional sets of features. Given the importance and consequences of these decisions, is important audit these algorithms ensure they are fair, safe, and act in the best interests of the communities they affect.
To obtain impartial audits, it is desirable to have independent organizations or teams evaluate proposed prediction hms,, This often means such audits must be performed without access to the inner workings of the target algorithm, due to intellectual property concerns. In such cases, post-hoc explainability methods
"Equal contribution.

---

Published in Transactions on Machine Learning Research (05/2025)
that do not require access to the inner workings of the target algorithm are most appropriate. In this post- hoc setting, existing ML. explainability approaches that involve retraining, modifying, or building newly explainable algorithms (Alvarez Melis & Jaakkola 2018; Koh et al. 2020), or accessing intermediate layers (Kim et al. 2018 Bahadori & Hockerman 2021 Ych et al. 2020), are not directly applicable. "Feature importance" measures are popular; these aim lo capture some generalized notion of association wetn each feature and model's prediction output (or prediction error), but do not generally differentiate between features that are merely associated with the model prediction versus those that have a causal effect on the output. Distinguishing whether some variable is a cause of model outputs rather than only associated due to confounding is crucial to understanding whether manipulations of that feature will make any difference to the prediction model output; intervening on variables associated only due to confounding will have no consequence and provides no explanation of the outcome. We discuss this instinctioon and, more broadly, the importance of causal explainability in Section 2 see also Shin (021) and Moraffah et al. (2020). Existing
causality-awan feature importance approaches (Friedman 2001 Loftus et al. [024] Goldstcin et al. 2015;
Janzing et al. 2020) have important shortcomings in this post-hoc setting: (i) the explanatory features are micro-level inputs into the target prediction model, e.g., image pixels, rather than interpretable macro-level features that are more useful for understanding how to possibly change the algorithm's behavior, and (ii) existing approaches assume there exists no unmeasured confounding between features and target model predictions, which fails to hold when the explanatory units are macro-level variables. Throughout, we use snticro-level" to refer to the raw features directly input into the prediction model, which are typically pixel or voxel values in image analysis and may be words or tokens in text analysis. " "Macro-level" features then correspond to (observed or latent) abstractions of the raw features that are independently manipulable, also described causal representations disentangled representations {Schölkopf et al., )021).
Here we present an approach to post-hoc explanation in terms of user-selected macro-level interpretable features, building on ideas from the literature on causality and graphical models. By employing a causal graphical representation capable of dealing with unobserved confounding, we are able to provide valid causal explanations in terms of macro-level features and distinguish causal features from features merely associated with model predictions due to confounding. This is accomplished through the application of causal discovery methods (a.k.a. causal structure learning) (Spirtes et al. 2000 Pearl 2009 Peters et al.) 2017).
We begin by providing some background on causal explanation and causal modeling, the choice of "ex- planatory units", and the limitations of existing approaches. We then describe our causal discovery-based proposal for explaining the behaviors of black-box prediction algorithms. We present a simulation study that highlights key issues and demonstrates the strength of our approach. We apply a version of our proposal to two datasets: annotated image data for bird classification and annotated chest X-ray images for pneumonia detection. Finally, we discuss some applications, limitations, and future directions of this work.
2 Causal Explanation
There is long history of debate in science and philosophy over what properly constitutes an explanation of some phenomenon. (In our case, the relevant phenomenon will be the output of a prediction algorithm.) A connection between explanation and "investigating causes" has been influential, in Western philosophy. at least since Aristotle| More recently, philosophical scholarship on causal explanation (Cartwright) 1979
Salmon 1984 Woodward 2005 Lombrozo & Vasilyeva 2017) has highlighted various benefits to pursuing understanding of complex systems via causal or counterfactus knowledge, which may be of particular utility to the machine learning community. We focus here primarily on some relevant idcas discussed by Woodward (Woodward 2005) to motivate our perspective in this paper, though similar issues are raised elsewhere in
the literature.
In influential 20th-century philosophical accounts, explanation was construed via applications of deductive logical reasoning (i.e., showing how observations could be derived from physical laws and background condi- tions) or simple probabilistic reasoning [Hempel 1965). One shortcoming- - discussed by several philosophers in the late 20th-century - of all such proposals is that explanation is intuitively asymmetric: the height of a flagpole explains the length of its shadow (given the sun's position in the sky) but not vice versa; the length of classic mechanical pendulum explains the device's period of motion, but not vice versa. Logical and
2

---

Published in Transactions on Machine Learning Research (05/2025)
associational relationships do not exhibit such asymmetries. Morcover, some statements of fact strong associations seem explanatorily irrelevant to a given phenomenon, as when the fact that somebody neglected to supply water to a rock "explains" why it is not living. (An analogous fact may have been more relevant for a plant, which in fact needs water to live.) Woodward argues that "cxplanatory relevance" is best understood via counterfactual contrasts and that. the asymmetry of explanation reflects the role of causality.
On Woodward's counterfactual theory of causal explanation, explanations answer what-would-hare sarullddhave-beer beenn different questions. Specifically, the relevant counterfactuals describe the outcomes of interventions or ma- nipulations. X helps explain Y if, under suitable background conditions, some intervention on X produces change in the distribution of Y. (Here we presume the object of explanation to be the random variable Y, not a specific value or event Y = That is, we choose to focus on type-level explanation rather than token-level explanations of particular events. See Beckers (2022) for related discussion focused on causal explanation at the event level.) This perspective has strong connections to the literature on causal models in artificial intelligence and statistics (Spirtes et al., 2000 Pearl 2009; Peters et al., 2017). A causal model for outcome Y precisely stipulates how Y would change under various interventions. So, to explain black-box algorithms endeavor to build causal models for their behaviors. We propose that such causal explana- tions can be useful for algorithm evaluation and informing decision-making In contrast, purely associational measures will be symmetric, include potentially irrelevant information, and fail to support (interventionist) counterfactual reasoning.
Despite a paucity of causal approaches to explainability in the ML literature (with some exceptions, discussed later), survey research suggests that causal explanations are of particular interest to industry practitioners; Bhatt et al. (2020) quote one chief scientist saying "Figuring out causal factors is the holy grail of explainabilityy" and report similar sentiments expressed by many organizations.
3 Causal Modeling
Here we provide some background on causal modeling to make our proposal more precise. Throughout, we use uppercase letters (e.g., X, Y) to denote random variables or vectors and lowercase (x. y) to denote fixed values.
We use causal graphs to represent causal relations among random variables (Spirtes et al., 2000 Pearl 2009). In causal directed acyelic graph (DAG) g (V, F), directed edge between variables X (X,Y E V) denotes that X is a direct cause of Y, relative to the variables the graph. Direct causation may be explicated via system of nonparametric structural equations (NPSEMs) a.k.a. structural causal model (SCM). The distribution of Y given an intervention that sets X to x is denoted p(y | do(z)) by Pearl [Paar 2009). Causal effects are often defined as interventional contrasts, c.g., the average causal effect (CCE): E| do(z)] - E[Y | do(x')) for values x, x'. Equivalently, one may express causal effects within the formalism of potential outcomes or counterfactual random variables, c.f. (Richardson & Robins 2013).
Given some collection of variables V and observational data on V, one may endcavor to learn the causal structure, i.e., to select causal graph supported by the data. We focus on learning causal relations from purely observational (non-experimental) data here, though in some MI. settings there exists the capacity to "simulate" interventions directly, which may be even more informative. There exists a significant literature on selecting causal graphs from a mix of observational and interventional data, e.g. Triantafillon & Tsamardinos (2015): Wang et al. (2017b), and though we do not make use of such methods here, the approach we propose could be applied in those mixed settings as well.
There are variety of algorithms for causal structure learning, but what most approaches share is that they exploit patterns of statistical constraints implied by distinct causal models to distinguish among candidate graphs. One paradigm is montrriitt-bsse lcarning, which will be our focus. In constraint-basee lcarning, the aim is to select a causal graph or set of causal graphs consistent with observed data by directly lesting a sequence of conditional independence hypotheses. Distinct models will imply distinct patterns of conditional independence, and so by rejecting (or failing to reject) a collection of independence hypotheses, one may narrow down the set of models consistent with the data. (These methods typically rely on some version of the faith hiihefl assumption (Spirtes et al. 2000), which stipulates that all observed independence constraints
3

---

Published in Transactions on Machine Learning Research (05/2025)
74
Z5
11
X5
X6
X7
X8
X9
Figure 1: An example DAG to represent an image data-gencrating process. Y denotes the true label (c.g., disease status), Zs denole high-level interpretable feal ures, and Xs denole (clusters or collections of) pixels. The output of the prediction algorithm trained on X is 8 = f()).
Zi
Zz
Z4
Z5
Figure 2: The PAG reprosentation of the process above, with vertices (Z1, Zz, Za Z).
correspond to missing edges in the graph.) For example, a classic constraint-based method is the PC algorithm (Spirtes et al.) 2000), which aims to learn an equivalence class of DAGs by starting from fully- connected model and removing edges when conditional independence constraints are discovered via statistical tests. Since multiple DAGs may imply the same set of conditional independence constraints, PC estimates a CPDAG (completed partial DAG), a mixed graph with directed and undirected edges that represents a Markov equivalence class DAGs, (Two graphs are called Markov equivalent if they imply the same conditional independence constraints.) Variations on the PC algorithm and related approaches selecting CPDAGs have been thoroughly studied in the literature (Colombo & Maathuis 2014 Cui al. 2016).
In settings with unmeasured (latent) confounding variables, it is typical to study graphs with bidirected edges to represent dependence due to confounding. For example, partial ancestral graph (PAG) (Zhang) 2008a is a graphical representation which includes directed edges (X + Y means X is causal ancestor of Y), bidirected edges (X ++ mcans X and Y are both caused by some unmeasured common factor(s), c.g., X + U Y), and partially directed edges (X o Y or Xoo Y) where the circle marks indicate ambiguity about whether the endpoints are arrowheads or tails. Generally, PAGs may also include additional edge types represent selection bias, but this is irrelevant for our purposes here. PAGs inherit their causal interpretation by encoding the commonalities among set of underlying causal DAGs with latent variables. A bit more formally, PAG represents an equivalence class of maximal ancestral graphs (MAGs), which encode the independence relations among observed variables when some variables are unobserved [Richardson & Spirtes) 2002 Zhang 2008a). The FCI algorithm (Spirtes et al. 2000; Zhang 2008b) is a well-known constraint- based method, which uses sequential tests of conditional independence to select a PAG from data. Similarly the PC algorithm, FCT begins by iteratively deleting edges from fully-connected graph by a sequence of conditional independence tests. However, the space of possible models (mixed graphs with multiple types of edges) is much greater for FCI as compared with PC, so the rules linking patterns of association to possible causal structures more complex. Though only independence relations among observed variables are testable, it can be shown formally that certain patterns of conditional independence among observed variables are incompatible with certain latent structures (assuming faithfulness), so some observed patterns will rule out unmeasured confounding and other observed patterns will on the contrary suggest the existence of unmeasured confounders. Details of the FCI algorithm may be found in the litcrature (Zhang 2008b; Colombo et al. 2012). Variations on the FCI algorithm and alternative PAG-learning procedures have also been studied (Colombo et al. 2012 Claassen & Heskes 2012
Ogarrio et al. 2016 Tsirlis et al. 2018
4

---

Published in Transactions on Machine Learning Research (05/2025)
3.1 Explanatory Units
Consider a supervised learning setting with a high-dimensional set of "low-level" features (e.g., pixels in an image) X= (X1, ... Xg) taking values in an input space 2 and outcome Y taking values in V. A prediction classification algorithm (e.g,, DNN) learns a map f X + y. Predicted values from this function are denoted Y. To explain the predictions Y, we focus on a smaller set of "high-level" features Z (Z1,.., Zp) (p & q) that are "interpretable" in the sense that they correspond to human-mddersanddaie and potentially manipulable elements of the application domain of substantive interest, e.g., "effusion in the lung," " "presence of tumor in the upper lobe," " and so on in lung imaging or "has red colored wing feathers," "has a curved beak," " and so on in bird classification. Though we use mostly examples from image classification in our discussion and hence the language of "pixels," " the raw input data may instead consist in text (words or tokens), measurements taken over time, or other kinds of "low-level" signals.
Though the target prediction algorithm takes low-level features X (XI,..,,,)) as input, our interest is explaining the output Y in termss of Z = (Z....., Zp). One reason is that individual micro-features (e.g, pixels) may make very little causal difference to the output of a prediction model, but have important effects in aggregate. That is, groups of pixels (not necessarily spatially contignous) or higher-level statistical properties of pixels (e-g,, the variance in brightness of some region of pixel space, the existence of shapes, borders, other contrasts) are often truly what make difference to prediction algorithm. A paradigmatic example of this is when the background color or lighting of photograph has a strong effect on the predicted label intervening to change an individual background pixel has no consequence, but setting the photographic subject against a different background or in different lighting conditions may change the output dramatically. This highlights a second reason to focus on macro-level interpretable features: they more often coincide with relevant manipulable elements of the research domain.
3.2 Induced Unmeasured Confounding
When the relevant explanatory units do not coincide with the set of raw features used by the prediction algorithm, unmeasured confounding becomes salient issue. In particular, if the set = (71, ..., 7,p) is selected by a user it is not generally possible to know if "all the relevant features" are included in Z. There is always the possibility that some causally-importa the case if the user did not know priori that "lighting conditions" had an important effect on the prediction output). The data-generating process we assume may be represented by graph like the one in Figure Here Y denotes the true concept or label that we aim to predict (e.g., disease status of a patient). This generates the high-level features Z (e.g,, symptoms or findings on an X-ray), which are rendered via imaging/dale recording technology as pixels X. The output of the prediction algorithm Ý is direct function of only raw inputs X. Importantly, the elements of Z may be causally related to each other: e.g,, in a medical setting, interventions on some symptoms may lead to changes in othe symptoms. Thus there may be directed edges from some Z, + Ly as in Figure [] Note that in the hypothetical model depicted here, some elements of Z have causal pathways to Y and others do not, and some elements may appear associated with Y despite no causal pathway. (Z, has a causal pathway to Y, while Ze does not. Yet, Z1 and Zn are associated due to their common parent Y, so Y and Ze will likely be associated in the data.) In applications, the true underlying data-generating DAG is unknown: we may have substantial uncertainty about both how the macro-level variables are related to each other, which pixels or groups of pixels they affect, and which pixels play an important role in the predicted output Y = f().. Causal discovery algorithms may thus be illuiinatin here, in particular causal discovery algorithms that are consistent the presence of unmeasured confounding.
macro-level features have been excluded (as would be
3.3 Limitations Of Existing Methods
Given this emphasis on macro-level explanatory variables and the attendant issue of unmeasured confound- ing, existing approaches to explainability have important limitations. One popular class of methods are "perturbation-base" based" approaches (Chattopadhyay et al., [2019] [Schwab Karlen [019 [elvaraju et al.,
2017 Simonyan et al. 2013. Shrikumar et al.] 2017 Chen al.] 2024), which perturb each element X;
5

---

Published in Transactions on Machine Learning Research (05/2025)
Z1
Z3
Figure 3: Data generating process for Example 1.
holding the remaining elements X-; fixed and observe the change in Y. Since the relationships among the Xs and Zs are complex and unknown, these perturbation-based quantities cannot. be straightforwardly translated into explanations as a function of macro-level explanatory units ( (Zs). Some cansality-aware fea-
ture importance approaches, such as Partial Dependence Plots (Friedman, 2001 Causal Dependance Plots
[Loftus et al.) 2024), Individual Conditional Expectation plots GGoldseei al 2015), and modified causal variations of classical Shapley values (Janzing et al. 2020 Jung et al.] 2022 Heskes et al.. 2020) may be applied (or adapted) to provide explanatory information at the level of macro variables. However, these ap- proaches assume there is no unmeasured confounding for the chosen set of explanatory features. Ultimately, these approaches will estimate quantities that adjust for (condition on) some subset of the observed features, but in the presence of unmeasured confounding there may be no observed subset that is valid. This can lead to substantially incorrect "explanatory" conclusions.
We illustrate this latter point with a simple example using Shapley values (Lundberg & Lee 2017 Shapley values are feature attribution approach with roots in game theory. Given game with N players and a cost function v(N), Shapley values allocate cost to each player by defining value D, for each player i, with Φ; computed as
|SU(NNN - |S| - 1) (c(SU - e(S)
$
Σ SCN(O)
NN!
The Shapley values satisfy the property - d.. Important criticisms of Shapley values as feature importance measures have been raised elsewhere (Kumar et al. 2020). We examine modified approach proposed in Janzing et al. (2020) that relies on backdoor adjustment (Pearl 2009) to perform cansal at- tribution of the prediction made by & model to its individual features. This amounts to a causally-aware proposal for estimating the contributions v(S U (i)) in the selting where "costs" are model predictions and "players" are features. Though the authors in Janzing et al. (2020) mainly consider a setting wherein the inputs to the target prediction algorithm correspond to the explanatory units, here we consider a version of the approach applied to macro-level features Z = (Zi,.., Zp). (If there is a mismatch between model inputs and explanatory units, it is not straightforward to apply the approach described in Janzing et al. (2020).) Tn the presence of unmeasured confounding, it is possible that the Shapley value calculated following Janzing et al. (2020) for a particular feature is zero even when the feature is in fact causally relevant to the prediction model output. We demonstrate this through a brief example here, with the relevant calculations presented in the Appendix.
Example Consider the cansal graph shown in Figure associated with the following SCM:
Y = cy
Z3 = 7o + Y +
Z₂ = 0o + 0 Y +
7, = up n 72 + u2 vzY + EZ
y = 2 Z3 + 11 + 12262
6

---

Published in Transactions on Machine Learning Research (05/2025)
Bill Shape
Bill Length
Wing Pattern
Belly Color
Wing Shape
Figure 4: An image of a Baltimore Oriole annotated with interpretable features.
The Shapley value for Zi calculated using a valid adjustment set (Zz, Za) may be calculated as:
Φz.
0!2!
{fiz) f/
1!1!
(,2,,2-- -f2.)
1!1!
(S22.20
f23s
2!0!
{ {112122.2))
3!
3!
This leads to the following Shapley value for Z1:
47
M021 E[
Mn Z Za] + - - 1123
2
+ #1
21 - E[Z]
However, if Za is omitted from the chosen set of explanatory features, backdoor adjustment can only be (incorrectly) performed using Zz. In that case, the Shapley value for Zı would be:
oz (2) E[Zi))/uo2s + #1]
Detailed calculations may be found in the Appendix. Now, if E[Zi], then the incorrect Shapley value will be O, while the true Shapley value will be 1 [[ moB| Za] which can be non-zero. This rllustrates key shortcoming of Shapley values in this setting: there unmeasured confounding, even iausally-ifforeed Shapley values may lead misleading conclusions about feature importance. There are promising recent proposals combining Shapley with identification theory for mixed graphs representing confounded variable sets (Jung et al. 2022). but in the setting considered here, the correct graph is not known.
4 Explaining Black- Box Predictions
Recall that our task is to explain the behavior of some black-box prediction model trained on features X (X1,.., X,) by identifying the causal determinants of y f(X) from among set of macro-level features Z (Za,..,,
For now, we assume that the interpretable feature is predetermined and available in the form of an annotated dataset, i.c., that each image is labeled with interpretable features. Later, discuss the possibility of automatically extracting set of possibly interpretable features from data. which lacks annotations. We

---

Published in Transactions on Machine Learning Research (05/2025)
Figure 5: chest X-ray from a pneumonia patient. The image was annotated by a radiologist indicate Effusion, Pneumothorax, and Pneumonia.
allow that elements of Z are statistically and causally interrelated, but assume that contains no variables which are deterministically related (c.g., where Z 2, always implics Zj- 2j for some * j), though we also return to this issue later. Examples from two image datasets that discuss later, on bird classification and lung imaging, are displayed in Figures g and
5
X consists of the raw pixels and Z includes the
interpretable annotations identified in the figures.
Here we describe the assumed data-generating process more formally. We use Pa(V, 9) to denote the set of parents of a vertex V in graph 9, i.e., vertices with directed edges into V. Let 9 be DAG with vertices T, Y such that Pa(Y.g) 0 (the "true label" has no causal parents), Pa 9) C {Z Y} Vi (each macro-level feature may be caused by other macro-level features as well as Y), (a(XX,9)) C Z Vj (micro-level features are determined by macro-level features), and Pa(f, 9) S X (the model predictions are a function of model inputs). Let Z denote the subsct of macro-level features that are recorded in the data. Described as SCM, assune the dala-generating process has the following form:
Y
Z g. (Pa(Z, a.) Vi
X, (PPa(Xj.9).1) Vj
f = S(Pa(P) ,g))
with all errors 2j mutually independent. The marginal distribution over variables ( P) satisfies the Markov condition wrt a MAG M (a projection the DAG g onto this subsct of vertices). The Markov equivalence class of M is represented by PAG P (Zhang 2008a).
Our proposal is to estimate a causal PAG over the variable set V (Z, 9), with the minimal background knowledge imposed that the predicted outcome is a causal non-ancestor of Z (there is no directed path from Ý into any element of Z). This knowledge simply reflects the fact that the output of prediction algorithm trained
image does not cause the content of that image.
We primarily use the FCI algorithm in our experiments, but alternative PAG-learning discovery algorithms may be also be used. FCI selects PAG by executing the following steps: (1) beginning with a complete graph over the vertices, a series of conditional independence hypotheses are tested for every pair of verticies and every possible conditioning set.. When an independence is discovered between two vertices, the edge between them is removed and the relevant conditioning set is recorded (the separating set). (2) The procedure then orients some triples of variables colliders using the so-called "collider rule": triple Z, a-o Zj o-o Zy. with Z; and Zy not adjacent is oriented Z, o-> Z, Zk only if Z, is not in the separating set for Z,, Zu. The result at this stage is a partially-oricnte, graph. (3) Some additional independence tests are carried out based on the graph structure (see detailed discussion in Colombo et. al. (2012)) and the collider rule is repeated. (4) Finally, some additional orientations may follow from the orientations already determined, using provably complete graphical rules that are applicd exhuastively the partially oriented graph (Zhang, 2008b). Assuming faithfulness, the FCI algorithm an asymptotically correct estimator of P (Spirtes et al. 2000 Zhang 2008b Colombo et al. 2012 Simulations previously reported indicate that when the ground
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Types of PAG edges relating some interpretable feature Zi and the predicted outcome Y.
EDGE TYPE
EDGE INTERPRETATION
21 + 8
Z: is cause of y
2 Y
Z, and Y share an unmeasured common cause Z; - Either Zi is a cause of y or there is unmeasured confounding, or both
2 o
truth is sparse and the sample size is sufficiently large, FCI performs well and recovers structure close to the true PAG. However, every conditional independence test performed has some chance of error, and compounding errors may lcad to the incorrect structure especially when sample size is small or the truth is dense. Though it is very difficult lo exactly recover the correct graph with any finile sample size, simulation studies show that graphs with good precision can in many cases be estimated from on the order of a few thousand samples (Colombo et al., 2012 Ogarrio et al. 2016).
Additional background knowledge may also be imposed if it is known, for example, that none of the elements of Z may cause each other (they are only dependent due to latent common causes) there groups of variables which precede others in a causal ordering. If the estimated graph G, there is a directed edge 7: Y, then 74 is cause (definite causal ancestor) of the prediction, if instead there is bidirected edge Z: ++ y then Zi is not cause of the prediction but they are dependent due to common latent factors, and if 7 Y then Z, is a possible cause (possible ancestor) of the prediction but unmeasured confounding cannot be ruled out. These edge types are summarized in Table The reason it is important to search for PAG and not DAG (or equívalence class of DAGs) is that Z will in general not include all possibly relevant variables.
5 A Simulation Study
To illustrate our proposed approach and its ability to distinguish truly causal features from "spurious" (confounded) features, we carry out the following experiment, inspired by and modified from a study reported in Chalupka et al. (2015). Black-and
images are generated containing various geometric shapes (alone in combination): a horizontal bar (H), vertical bar (V), circle (C), or triangle (T), in addition to random pixel noise; see Figure i The true binary label Y is Bernoulli random variable which makes the appearance of certain shapes more or less likely. Data is generated according to the DAG in Figure 7 (a), with the following parameters used in the data generating process:
Y ~ (ornoull((.4))
C ~ Bernoulli(0.25)
V Bernoullf(expit(11111 + 2.3 x Y))
H Bernoullfexppi(( 3.112 V 1.5974 x Y))
T Bernoull((exp((222 C 2.2 H))
The task is to predict the binary label Y on the basis of raw image pixel data. We build black-box model based multiclass neural network architecture called HydraNet (Cohen 2022). In order to control, for the purposes of this illustrative simulation, which variables are truly causes of the prediction output, we combine this neural network with logistic regression. We first pre-train HydraNet on a dataset of 2,500 images to predict the presence of horizontal bars, vertical bars, triangles, and circles in a given image. Then, we train a logistic regression classifier using the predictions from HydraNet as the features and the label Y as target, on a datasct of 20,000 images simulated according to the above DGP, An inspoction of the logistic regression coefficients confirms that Ý is a function of only V and H these are the true causes of the classifier's output, illustrated graphically in Figure Za). We emphasize that the function of the final logistic regression in this prediction pipeline is only to constrain the predictions ÿ depend entirely on V and H. Relevant details and calculations may be found in Appendix.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Labeled Y =
Labeled Y =
Labeled Y =00
Labeled Y = 0
Figure 6: Simulated image examples with horizontal bars, vertical bars, circles, and triangles.
(b)
(a)
Figure 7: (a) A cansal diagram representing the true data generating process. Here Y denotes the true binary label and C,T, H, and indicate the presence of circles, triangles, horizontal bars, and vertical bars in the images. Y denotes the output of the combined neural network and logistic regression classifier. (b) The PAG learned using FCI over (C,T.V,Ý).
The model performs reasonably well: 77.89% accuracy on a set of 20,000 images. Our interpretable features Z are indicators for the presence of the various shapes in the image. Since the true underlying behavior is causally determined by V and H, we expect V and H to be "important" " for the predictions Y, but due to the mutual dependence the other shapes are also highly correlated with Y. Moreover, we mimic setting where Hl is (wrongfully) omitted from the set of candidate interpretable features; in practice, the feature set proposed by domain experts available in the annotated data will typically exclude various relevant determinants of the underlying outcome. In that case, H is unmeasured confounder, Applying the PAG- estimation algorithm FCI to variable set (C, T, V, Y) we Iearn the structure in Figure (b), which indicates correctly that is a (possible) cause, but that T is not: T is associated with the prediction outcomes only due to an unmeasured common cause. (FCT incorporates the user-specified background knowledge that is not a cause of any image features.] | This simple example illustrates how the estimated PAG using incomplete interpretable features can be useful. We disentangle mere statistical associations from (potentially) causal relationships, in this case indicating correctly that interventions on may make a difference to the prediction algorithm's 's behavior but interventions on would not, and moreover that there potentially important cause of the output (H) that has been excluded from our set of candidate features, as evident from the bidirected edge T ++ Y learned by FCI. Existing approaches that do not properly account for unmeasured confounding cannot distinguish between such features and would incorrectly identify T as "important" for explaining Y.
6
Experiments: Bird Classification and Pneumonia Detection from X-rays
We conduct two real data experiments to demonstrate the utility of our approach. First, we study a neural network for bird classification, trained on the Caltech-UCSD 200-2011 image dataset (Wah et al., 2011). It consists of 200 categories of birds, with 11,788 images total. Each image comes annotated with 312 binary attributes describing interpretable bird characteristics like eye color, size, wing shape, etc. We build black- box prediction model based on convolutional neural network (CNN) using pixel features to predict
Throughout, we FCI implemented in the command-line interface
TETRAD freeware:
https://githur coctb..coo/caaa cann/cmu-phiii pasp/mmmaaiiiterrad
10

---

Published in Transactions on Machine Learning Research (05/2025)
the class of the bird and then use FCI to explain the output of the model. Second, we follow essentially the same procedure to explain the behavior of pneumonia detection neural network, trained on a subset of the ChestX-ray8 dataset (Wang et al., 2017a). The dataset consists of 108,948 frontal X-ray images of 32,217 patients, along with corresponding free-text X-ray reports generated by certified radiologists. Both data sources are publicly available online.
6.1 Data Preprocessing & Model Training
Bird Image Data
Since many of the species classifications have few associated images, we first broadly group species into 9 coarser groups. For example, we group the Baird Sparrow and Black Throated Sparrow into one Sparrow class. Details on the grouping scheme can be found in the Appendix. This leads to possible outcome labels and 5514 images (instances that do not fit into one of these categories are excluded from analysis). The number of images across each class is not evenly balanced. So, we subsample overrepresented classes in order to get roughly equal number of images per class. This yields dataset of 3538 images, which is then partitioned into training, validation, and testing datasets of 2489, 520, and 529 images respectively. We train the ResNet 18 architecture (pre-trained on ImageNet) (He et al. 2016) on our dataset and achieve an accuracy of 86.57% on the testing set. For computational convenience and statistical efficiency, we consolidate the 312 available binary attributes into ordinal attributes. (With too many attributes, some combinations of values may rarely or never co-occur, which leads to data fragmentation.) For example, four binary attributes describing "back pattern" (solid, spotted, striped, multi-colored), are consolidated into one attribute Back Pattern taking on five possible values: the 4 designations above and an additional category if the attribute is missing. Even once this consolidation is performed, there st.ill exist many altributes with large number of possible values, so we group together similar values. For example, we group dark colors such as gray, black, and brown into one category and warm colors such as red, yellow, and orange into another. Other altribules are consolidated similarly, as described in the Appendix. After the above preprocessing, for each image we have a predicted label from the CNN and 26 ordinal attributes.
Chest X-ray Data
From the full ChestX-ray8 dataset, we create smaller dataset of chest X-rays labeled with a binary diagnosis outcome: pneumonia or not. The original dataset contained 120 images annotated by a board-certified radiologist to contain pneumonia. To obtain "control" images, we randomly select 120 images with "no findings" reported in the radiology report. To generate interpretable features, we choose the following text- mined radiologist finding labels: Cardiomegaly, Atelectasis, Effusion, Infiltration, Mass, Nodule, and Pneumothorax. If the radiology report contained one of these findings, the corresponding label was given value 1 and 0 otherwise. This produces analysis dataset of 240 images: 120 pneumonia and 120 "control" with each image containing 7 binary interpretable features. Using the same architecture as for the previous experiment and reserving 55 images for testing, ResNet18 achieves an accuracy of 74.55%.
6.2 Structure Learning
Structure learning methods such as FCI produce single estimated PAG output. In applications, it is common to repeat the graph estimation multiple bootstraps subsamples of the original data, in order to control false discovery rates to mitigate the cumulative effect of statistical errors in the independence tests (Stekhoven et al. 2012). We create 50 bootstrapped replicates of the bird image dataset and 100 replicates of the X-ray dataset. We run FCI on each replicate with independence test rejection threshold (a tuning parameter) set to a .05 and a = .01 for the birds and X-ray datasets, respectively, with the knowledge constraint imposed that outcome Y cannot cause any of the interpretable features. Here FCI is used with the x2 independence test, and we limit the maximum conditioning set size to 4 for computational tractability in the birds dataset. (No limit necessary in the smaller X-ray dataset.) As a sensitivity analysis, we repeat both experiments using a different PAG-learning procedure, hybrid algorithm that combines the greedy score-based procedure GRaSP LLam et al. 2022 with FCT. The details are described in Appendix C.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Bidd Clacofication Experiment
Classiication Experiment
Wing
Shaps
Upser cater
Mickellass
Upperports Gatr
Breast Color
Rapa cake
Dell
Coler
Macs
Ivl Buttem
Nodsle
Bill Lengts
tye Calor
Cardeenagal
Back Coler
Creem
Coker
Preumathoras
Bdge Stabiity
Edge Stanilty
Figure 8: Results for two experiments. Potential causal determinants of (a) bird classifier output from bird images and (b) pneumonia classifier output from X-ray images.
6.3 Results
We compute the relative frequency over bootstrap replicates of Z; + and Z, o edges from all attributes. This represents the frequency with which an attribute is determined to be a cause or possible cause of the predicted outcome label and constitutes a rough mcasure of "cdge stability" or confidence that attribute's causal status. The computed edge stabilities are presenled in Figure 8 In the bird classification experiment, we find that the most stable (possible) causes include Wing Shape, Wing Pattern, and Tail Shape, which are intuitively salient features for distinguishing among bird categorics. We have lower confidence that the other features are possible causes of Y. In the X-ray experiment, we find that edges from Atelectasis, Infiltran Infiltration, and Effusion are stable possible causes of the Pneumonia label. This reassuring, since these are clinically relevant conditions for patients with pneumonia. An estimated PAG from one of the bootstrapped subsamples is displayed in Figure I0 The FCI procedure has detected variety of edge types, including bidirected (+) edges to indicate unmeasured confounding among some variables, but edges from Atelectasis, Infiltration, and Effusion consistently either directed (-+ partially directed (o-)) into the predicted outcome.
Our sensitivity analysis using an alternative PAG-learning procedure, GRaSP-FCI, produces similar results. In the X-ray experiment, the final ordering of features by edge stability GRaSP-FCT the same as the ordering produced by FCI. the bird classification experiment, there are some substantive differences: a few features ranked highly by FCI are ranked low by GRaSP-FCI and vice versa, though for most features the conclusions are concordant. Crown Color and Size ranked highest. The full results are presented in Appendix C.
6.4 Comparison Pixel-Level Explainability Approaches
Having discussed some shortcomings of existing approaches Section 3.3 we apply two popular pixel-level explanatory approaches to the data from each experiment. We first examine Locally Interpretable Model Explanations (LIME), proposed in Ribeiro al. (2016). LIME uses an interpretable representation class provided by the user (e.g., linear models or decision trees) to train an interpretable model that is locally faithful to any black-box classifier. The output of LIME in our image classification setting is, for cach input image, a map of super-pixels highlighted according to their positive or negative contribution toward the class prediction. The second approach we investigate is a version of the Shapley Additive Explanation (SHAP) algorithm, proposod by Lundberg Lee (2017). The output of this mothod in our setting list of SHAP values for input features in the image that can be used to understand how the features contribute a prediction.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Underparts Color
Upper Tail Color
Wing Pattern
Throat Color
Wing Shape
Figure 9: A PAG (subgraph) estimated from the bird image data using FCI. Y denotes the output of the bird classification neural network (P takes one of 9 possible outcome labels). For considerations of space and readability, we only display the subgraph over variables adjacent to Y on this run of the algorithm. Directed and edges into Y from several variables indicate that these are likely causes of the prodiction algorithm's 's output, whereas many other variables are mostly not causally relevant. Across subsamples of the data, the graph varies and some edges are more stable than others: Wing Shape and Wing Pattern are picked out the most stable causes of the algorithm's output.
Mass O
Nodule
Cardiomegaly
Effusion
Infiltration
Atelectasis
Pneumothorax
Figure 10: A PAG estimated from the X-ray data using FCI. Ý denotes the output of the pneumonia classification neural network (P = for pneumonia cases, y = O for controls). Directed and partially directed edges into y from Atelectasis, Infiltration, and Effusion indicate that these are likely causes of the prediction algorithm's output, whereas the other variables are mostly not causally relevant. Across subsamples of the data, edges from these three key variables are sometimes directed and sometimes partially directed.
To compare our proposal with LIME and SHAP, we apply LIME and SHAP to the same neural network training pipeline used in our bird classification and X-ray experiments. We use implementations of LIME and SHAP available o For SHAP we apply the Deep Explainer module and for both algorithms we use default settings in the software. Since LIME and SHAP provide local explanations (in contrast with the type- level causal explanations provided by our approach), we present some representative examples from output on both classification tasks and display these in Figures [1 and 12 LIME and SHAP seem to highlight "important" pixels corresponding to the bird or the X-ray, but they do not communicate or distinguish what is important about the highlighted regions. For example, if the body of a bird is (in part or entirely) highlighted, a user cannot distinguish on the basis of LIME whether it is the wing pattern, wing shape, size, some other relevant ornithological feature that is underlying the algorithm's 's classification. All one can conclude is that the "bird-bodly-rgionn of the image is important for classification, which is a reasonable bascline requirement for any bird classification algorithm but provides limited insight. Similarly, SHAP highlights pixels or clusters of pixels, but the pattern underlying these pixels and their relationship to bird characteristics are not transparent or particularly informative, In the X-ray images, we see the same issues: the diffuse nature of potential lung discase symptoms (varictics of tissue damage, changes lung density, presence of fluid) is not clearly distinguished by either LIME or SHAP. Both algorithms simply highlight substantial sections of the lung. There is a mismatch between what is scientifically interpretable in these tasks and the "interpretable features" selected by LIME and SHAP, which are super-pixels or regions of pixel space. Finally, the local image-specific output of these algorithms makes it difficult to generalize about what in fact is "driving" the behavior of the neural network across the population of images. We conclude that LIME and SHAP, though they may provide valuable information in other settings, are largely uninformative relative to our goals these image classification tasks in which distinguishing between spatially co-located
2 Available https: httgggihhh.ccmmaacooeeeee
and https: /ttpt:://gthbb..... com/ slunderg//sap.p...
13

---

Published in Transactions on Machine Learning Research (05/2025)
Figure 11: Results for LIME selected images from the birds and X-ray datasets. Green highlights super- pixels with positive contribution to the predicted class and red indicates negative contribution.
-20
15
SH-P
Figure 12: Results for SHAP on selected images from the birds and X-ray datasets. Columns correspond to output class labels (i..,, 9 bird categories and the binary pneumonia label). Original input images are in the leftmost column. Large SHAP values indicate strong positive (red) or negative (blue) contributions to the predicted class label.
image characteristics ("wing shape" and "wing pattern" are both co-located on the wing-region of the image, pneumothorax and effusion are distinct problems that may affect. the same region of the chest) is important to support meaningful explanatory conclusions.
In contrast, the graphical representations learned by FCI do distinguish between meaningfully distinct causal factors that may occupy roughly the same region of pixel space. Namely, in the bird classification task results suggest that Wing Shape and Wing Pattern are important determinants of the output but Wing Color is not, Tail Shape may be important but not Tail Pattern, elc. In the X-ray experiment, FCI identifies clinical symptoms such as Infiltration, Atelectasis, and Effusion as important, but not the presence of Mass or lung Nodule. Without such interpretable features, a highlighted region of lung tissue may leave it ambiguous which kind of abnormality affecting that region causally relevant the classifier. The ability to distinguish between "similar" and spatially-overlapping causal factors is consequence of
14

---

Published in Transactions on Machine Learning Research (05/2025)
using clinically distinguishable interpretable features as imput to the structure learning algorithm for this reason, we emphasize that input and validation from domain experts is important for applying the proposed approach in practice (and we return to this issue below). Whether algorithm end-users will find the distinctions relevant will depend on the domain and the degree of collaboration.
7 Discussion
We have presented an analytical approach (based on existing tools) to support explaining the behavior of black-box prediction algorithms. Below we discuss some potential uses and limitations.
7.1 Algorithm Auditing and Evaluation
One important goal related to building explainable AT systems is the auditing and evaluation of algorithms post-hoc. a prediction algorithm appears to perform well, it is important to understand why performs well before deploying the algorithm. Users will want to know that the algorithm is "paying atlention to" the right aspects of the input and not tracking spurious artifacts (Bhatt et al.) 2020). This important both from the perspective of generalization to new domains as well as from the perspective of fairness. To illustrate the former, consider instances of "dalaset bias" or "data leakage" wherein an irrelevant artifact of the data collection process proves very important to the performance of the prediction method. This may impede generalization to other datasets where the artifact is absent or somehow the data collection process is different. For example, WWinkler et al. (2019) study the role of violet image highlighting on dermoscopic images in a skin cancer detection task. "Thyy find that this image highlighting siguificantly affects the likelihood that a skin lesion is classified as cancerous by commercial CNN tool. (The researchers are able to diagnose the problem because they have access to the same images pre- and post-highlighting: effectively, they are able to simulate intervention on the highlighting.)
To illustrate the fairness perspective, consider a recent episode of alleged racial bias in Google's Vision Cloud API, a tool which aulomatically labels images into various categories (Kayser-Bril 2020). Users found an image of a dark-skinned hand holding a non-contact digital thermometer that was labeled "gun" by the API, while a similar image with light-skinned individual was labeled "clectronic devico." More tellingly, when the image with dark skin was crudely altered to contain light beige-colored skin (an intervention on "skin color"), the same object was labeled "monocular:" This simple experiment was suggestive of the idea. that skin color was inappropriately cause of the object label and tracking biased stereotyped associations. Google apologized and revised their algorithm, though denied any "evidence of systemic bias related to skin tone." "
Auditing algorithms to determine whether inappropriate features have a causal impact on the output can be an important part of the bias-checking pipeline. Moreover, a benefit of our proposed approach is that the black-box box model may be audited without access to the model itself, only the predicted values. This may be desirable in some settings where the model itself is proprictary.
7.2 Informativeness and Background Knowledge
It is important to emphasize that a PAG-based causal discovery analysis is informative to a degree that depends on the data (the patterns of association) and the strength of imposed background knowledge. Here we only imposed the minimal knowledge that V is not a cause of any of the image features and we allowed for arbitrary causal relationships and latent structure otherwise. Being entirely agnostic about the possibility of unmeasured confounding may lead, depending the observed patterns of dependence and independence in the data, to only weakly informative results if the patterns of association cannot rule out possible confounding anywhere. If the data fails to rulc out confounders and fails to identify definite causes of Y, this does not indicate that the analysis has failed but just that only negative conclusions are supported e.g., the chosen set of interpretable features and background assumptions are not sufficiently rich to identify the causes of the output. It is standard in the causal discovery litcrature to acknowledge that the strength of supported causal conclusions depends on the strength of input cansal background assumptions (Spirtes et al., (000). In some cases, domain knowledge may support restricting the set of possible causal structures, e.g., when it
15

---

Published in Transactions on Machine Learning Research (05/2025)
is believed the some relationships must be unconfounded or some correlations among Z may only be due to latent variables (because some elements of Z cannot cause each other).
7.3 Selecting or Constructing Interpretable Features
In our experiments we use hand-crafted interpretable features that are available in the form of annotations with the data. Annotations are not always available in applications. In such scttings, one approach would be to manually annotate the raw data with expert evaluations, e.g., when clinical experts annotate medical images with labels of important features (e-g. "tumor in the upper lobe," " "effusion in the lung," " etc). In settings where annotations are available only on a small subset of examples, semi-supervised learning tochniques developed in recent years may be useful for generating "pseudo-labels," and this may make the approach described here feasible even in seitings with only a few annolated examples. Pipelines such as FixMatcch (Sohn et al. 2020) use the predictions from a supervised model trained on annotated data combined with multiple "augmented" versions of unlabeled images that are expected to having matching labels to generate pseudo-labels at scale with excellent predictive performance.
Alternatively, one may endeavor to extract interpretable features automatically from the raw unlabeled data. Unsupervised learning techniques may be used in some contexts to construct features, though in general there is no guarantee that these will be substantively (e.g., clinically) meaningful or correspond to manipulable elements of the domain. We explored applying some recent proposals for automatic feature extraction methods to our problem, but unfortunately, none of the available techniques proved appropriate, for reasons we discuss.
2017 has introduced techniques for extracting causal features from "low-level" data, i.c., features that have causal effect on the target outcome. In Chalupka et al. (2015), the authors introduce an important distinction underlying these methods: observational classes versus causal classes of images, each being defined as an equivalence class over conditional or interventional distributions respectively. Specifically, let Y be a binary random variable and denote two images by X x=== =z. Then x, x' are in the same observational class if p(Y | X z) = p(Y X = ='). x, are in the same causal class if p(Y do(X =)) = p(Y do(X x²))- Under certain assumptions, the causal class is always coarsening of the observational class and so-called "manipulator functions" can be learned that minimally alter an image to change the image's causal class. The idea is that relevant features in the image are first "learned" and then manipulated to map the image between causal classes. However, there are two important roadblocks to applying this mothodology in our setting.
A series of related papers (Chalupka et al. 2015 2016
First, since our target behavior is the prediction y (which is some deterministic function of the image pixcls), there is no obsorved or unobsorved confounding between Y and X. This implics our observational and causal classes are identical. Hence any manipulator function we learn would simply amount to making the minimum number of pixel edits to the image in order to alter its observational class, similar to so-called "minimun-edit" counterfactual methods (Wachter et al. 2017 Sharma et al. 2019 Goyal et al. 2019).
Second, even if we learn this manipulator function, the output is not readily useful for our purposes here. The function takes an image as input and produces as output another ("close") edited image with the desired observational class. It does not correspond to any label which we may apply to other images. (For example, taking a photo of an Oriole as input, the output would be a modified bird image with some different pixels, but these differences will not generally map on to anything we can readily identify with bird physiology and use to construct an annotated dataset.) This does not actually give us access to the "interpretable features" that were manipulated to achieve the desired observational class.
Alternative approaches to automatic feature selection also proved problematic for our purposes. In Lopez- Paz et al. (017)), the authors train an object detection network that serves as a feature extractor for future images, and then they use these extracted features to infer directions of causal elfects. However, this approach relies on the availability of training data with a priori associated object categories: the Pascal Visual Object Classes, which are labels for types of objects (e.g., acroplanc, bicycle, bird, boat, etc.) (Everingham et al.) 2015). This approach is thus not truly data-driven and relies on auxiliary information that is not generally available. In another line of work, various unsupervised learning methods (based autoencoders
16

---

Published in Transactions on Machine Learning Research (05/2025)
variants of independent component analysis) are applied to extract features from medical images to improve classification performance (Arevalo et al. 2015 Sari & Gunduz-Demir 2018 Jiang et al.) 2018). These approaches are more data-driven but do not produce interpretable features they produce a compact lower- dimensional representation of the pixels in an abstract vector space, which does not nocessarily map onto something a user may understand or recognize. Recently, there has been interest in using large language models (LLMs) as annotators for unlabeled data, either alone or collaboration with humans (Wang et al. 2024 Tang al. 2024 Beck et al., 2025). This technology and its use for annotation is still in its development, but there is growing evidence that with careful engineering and human input, LLM-angmented annotations may cut costs for generating annotations at scale but also face limitations that are important to better understand.
In general, we observe a tradeoff between the interpretability of the feature set versus the extent to which the features are learned in a data-driven manner. Specifically, "high-level" features associated with datasets (whether they are extracted by human judgmont or automatically) may not always be interpretable in the sense intended here: they may not correspond to manipulable elements of the domain. Moreover, some features may be deterministically related (which would pose problem for most structure learning algorithms) and so some feature pre-sclection may be necessary. Thus, human judgment may be indispensable at the feature-selection stage of the process and indeed this may be desirable if the goal is to audit or evaluate potential biases in algorithms as discussed in Section 7.1
Finally, we note that macro-level feature annotations, whether they are derived from human judgment or statistical models, may contain errors. When labels are "noisy" or are corrupted by measurement error, this degrade the performance of causal discovery algorithms. Some approaches have been proposed for mitigating the effect of measurement error constraint-basel algorithms (Blom al. 2018), but in general this is a challenging and mostly open problem.
8 Conclusion
Causal structure learning algorithms specifically PAG learning algorithms such as FCI and its variants may be valuable tools for explaining black-box prediction methods. We have demonstrated the utility of using FCI in both simulated and real data experiments, where are able to distinguish between possible causes of the prediction outcome and features that are associated due to unmeasured confounding. We hope the analysis presented here stimulates further cross-pollination between research communities focusing on causal discovery and explainable AI.
References
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural net- works. In Advances in Neural Information Processing Systems, 2018.
John Arevalo, Angel Cruz-Roa, Viviana Arias, Eduardo Romero, and Fabio A González. An unsupervised feature learning framework for basal cell carcinoma image analysis. Arlificial Intelligence in Medicinee 64 (2):131-145, 2015.
Aristotle. Posterior Analytics.
Mohammad Taha Bahadori and David E Heckerman. Debiasing concept-based explanations with causal analysis. In International Conference Learning Representations, 2021.
Jacob Beck, Lukas Malte Kemeter, Konrad Dürrbeck, Mohamed Hesham Ibrahim Abdalla, and Franke Kreuter. Towards integrating chatgpt into satellite image annotation workflows. a comparison of label quality and costs of human and automated annotators. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2025.
Sander Beckers. Causal explanations and XAI. In Proceedings of the First Conference on Causal Learning and Reasoning, pp. 90-109. PML.R, 2022.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Fokersley. Explainable machine learning in deployment. Tn Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 648 657, 2020.
Tineke Blom, Anna Klimovskaia, Sara Magliacane; and Joris M Mooij. An upper bound for random mea- surement error in causal discovery. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, 2018.
Nancy Cartwright. Causal laws and ellective strategies. Nous, pp. 419 419-437, 1979.
Krzysztof Chalupka, Pietro Perona, and Frederick Eberhardt. Visual causal feature learning. In Proceedings of the 31st Conference Uncertainly in Artificial Intelligence, pp. 181-190, 2015.
Krzysztof Chalupka, Tobias Bischoff, Pietro Perona, and Frederick Eberhardt. Unsupervised discovery of el nino using causal feature learning on microlevel climate data. In Proceedings of the 32nd Conference on Uncertainly in Artificial Intelligence, pp. 72 81, 2016.
Krzysztof Chalupka, Frederick Eberhardt, and Pictro Perona. Causal feature learning: overview. Behav- iormetrika, ((1):137-1164 2017.
Aditya Chattopadhyay, Piyushi Mamupriya, Anirban Sarkar, and Vineeth N Balasubramanian. Neural net- work attributions: A causal perspective. In Proceedings of the 36th International Conference on Machine Learning, pp. 981 990, 2019.
Xucxin Chen, Ruichu Cai, Zhengting Huang, Yuxuan Zhu, Julien Horwood, Zhifeng Hao, Zijian Li, and José Miguel Hernández-Lobato. Feature attribution with necessity and sufficieccy via dual-stage pertur- bation test for causal explanation. In International Conference on Machine Learning, pp. PMLR, 2024.
David M. Chickering. Optimal structure identification with greedy scarch. Journal of Machine Learning Research, 3(N7 554, 2002.
Tom Claassen and Tom Heskes. A Bayesian approach to constraint based causal inference. In Proceedings of the 28th Conference on Uncerlainty in Arlificial Intelligence, pp. 207-21222000 2012.
Jeremy Cohen. Multi-task learning and hydranets with pytorch. In Puneet Chugh, Aritra Roy Gosthipaty, Susan Huot, and Ritwik Raha (eds.), PyJmageSearch. 2022. https://pying..c/// ro/hd177.
Diego Colombo and Marloes H. Maathuis. Order- Bidde-indeeendeei constraint- causal structure learning. Journal of Machine Learning Research, 15(1):3741-337 2014.
Diego Colombo, Marloes Н. Maathuis, Markus Kalisch, and Thomas S. Richardson. Learning high- dimensional directed acyelic graphs with latent and selection variables. Annals of Statistics, pp. 294-321, 2012.
Ruifei Cui, Perry Groot, and Tom Heskes. Copula PC algorithm for causal discovery from mixed data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 377-392, 2016.
Andre Esteva, Alexandre Robiequet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebast ian Thrun, and Jeff Dean. A guide to deep learning in healthcare. Nature Medicine, 25(1):21 ):24-29, 2019.
Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98 136, 2015.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of Statistics, pp. 1189-1232. 2001.
18

---

Published in Transactions on Machine Learning Research (05/2025)
Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin. Peeking inside the black box: Visualiz- ing statistical learning with plots of individual conditional expectation. Journal of Compulational and Graphical Statistics, 24(1):44 65, 2015.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lec. Counterfactual visual explanations. In International Conference on Machine Learning, pp. 2376 2384. PMLR, 2019.
Rita Georgina Guimaraes, Renata L. Rosa, Denise De Gaetano, Demostenes Rodriguez, and Graca Bressan. Age groups classification in social network using deep learning. TEFE Access, 5:10805-10816, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Carl G. Hempel. Aspects of scientific explanation. Free Press, 1965.
Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models. 2020.
Dominik Janzing, Lenon Minorics, and Patrick Blöbaum. Feature relevance quantification in explainable ai: A causal problem. In International Conference on Artificial Intelligence and Statistics, pp. 2907-2916. PMLR, 2020.
Junjun Jiang, Jiayi Ma, Chen Chen, Zhongyuan Wang, Zhihua Cai, and Lizhe Wang. SuperPCA: A super- pixelwise PCA approach for unsupervised feature extraction of hyperspectral imagery. IEEE Transactions on Geoscience and Remote Sensing, 56(8):4581-435) -5999, 2018.
Yonghan Jung, Shiva Kasiviswanathan, Jin Tian, Dominik Janzing, Patrick Blöbaum, and Elias Bareinboim. On measuring causal contributions via do interventions. In International Conference on Machine Leorning, pp. 10476-10501. PMLR, 2022.
Nicolas Kryser-Bril. Google apologizes after its Vision Al produced racist results. Algorithm Watch, 2020.
raciis//
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). International Conference on Machine Learning, pp. 2668 2677. PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. 5338 5348. PMLR, 2020.
Elizabeth Kumar, Suresh Venkatasubramannia, Carlos Scheidegger, and Sorelle Friedler. Problems with shapley-value-basee explanations as feature importance measures. In International Conference on Machine Learning, pp. 5491-5500, 2020.
Wai-Yin Lam, Bryan Andrews, and Joseph Ramsey. Greedy relaxations of the sparsest permutation al- gorithm. In Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence, pp. 1052 1062. PMLR, 2022.
Joshua R Loftus, Lucius EJ Bynum, and Sakina Hansen. Causal dependence plots for interpretable machine learning. In Advances in Newral Information Processing Systems, 2024.
Tania Lombrozo and Nadya Vasilyeva. Causal explanation. Orford Handbook of Causal Reasoning, pp- 415-432. Oxford University Press, 2017.
David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Discover- ing causal signals in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6979 6987, 2017.
19

---

Published in Transactions on Machine Learning Research (05/2025)
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advonces in Newral Information. Processing Systems, pp. -778, 2017.
Raha Moraffah, Mansooreh Karami, Ruocheng Guo, Adrienne Raglin, and Huan Liu. Causal interpretability for machine learning-problent methods and evaluation. ACM SIGKDD Explorations Newslelter, 22(1): 18-33, 2020.
Juan Miguel Ogarrio, Peter Spirtes, and Joseph Ramsey. A hybrid causal search algorithm for latent variable models. In Proceedings of the Eighth International Conference on Probabilistic Graphical Models, pp. 368- 379, 2016.
Judea Pearl. Causality. Cambridge University Press, 2009.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Flements of cousal inference: foundations and learning algorithms. MIT Press, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'Why should trust you?' Explaining the predic- tions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 1135-1144, 2016.
Thomas S. Richardson and James M. Robins. Single world intervention graphs (SWIGs): A unification of the counterfactual and graphical approaches to causality. Center for the Statistics and the Social Sciences, University of Washington Working Paper 128, pp. 1-146, 2013.
Thomas S. Richardson and Peter Spirtes. Ancestral graph Markov models. Annals of Stalistics, 30(4): 962-1030, 2002.
Wesley C. Salmon. Scientific explonation and the causal structure of the world. Princeton University Press, 1984.
Can Taylan Sari and Cigdem Gunduz-Demir. Unsupervised feature extraction via deep learning for histopathological classification of colon tissue images. TEEE Transactions on Medical Imaging, 38(5): 1139-1149, 2018.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation lcarning. Proceedings of the IEEE, 109(5): 612-634, 2021.
Patrick Schwab and Walter Karlen. CXPlain: Causal explanations for model interpretation under uncer- tainty. In Advances in Neural Information Processing Systems, pp. 10020-10220) 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618 626, 2017.
Shubham Sharma, Jette Henderson, and Joydeep Ghosh. Certifai: Counterfactual explanations for ro- bustness, transparency, interpretability, and fairness of artificial intelligence models. ar preprint arXiv:1905.07885 2019.
Donghee Shin. The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable ai. International Journal of Human- CommunCr Studies, 146:102551, 2021.
Avanti Shrikumar, Peyton Greenside, and Aushul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pp. 3145-3153 PMLR, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.603, 2013.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with con- sistency and confidence. Advances in Neural Information Processing Systems, 33:596 608, 2020.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Peter L. Spirtes, Clark N. Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2000.
Daniel J. Stekhoven, Izabel Moraes, Gardar Sveinbjörnsson, Lars Hennig, Marloes H. Maathuis, and Peter Büihlman. Causal stability ranking. Bioinformatics, 28(21):2819-2823) 2012.
Yi Tang, Chia-Ming Chang, and Xi Yang. Pdfchatanmotator: A human-llm Il collaborative multi-modal data annotation tool for pdf-format catalogs. In Proceedings of the 29th International Conference on. Intelligent User Interfaces, pp. 419 430, 2024.
Sofia Triantafillou and Ioannis Tsamardinos. Constraint-based causal discovery from multiple interventions over overlapping variable sets. Journal of Machine Learning Research, 16:2147 2012147-220,, 2015.
Konstantinos Tsirlis, Vincenzo Lagani, Sofia Triantafillou, and loannis Tsamardinos. On scoring maximal an- cestral graphs with the max-min hill climbing algorithm. International Journal of Approzimate Reasoning, 102:74- 05, 2018.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & Technology, 31:841, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. ChestX- ray8: hospital-scale chest X-ray database and benchmarks on weakly-superviee classification ion and localiza- tion of common thorax diseases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2097-2106, 2017a.
Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjic Miao. Human-llm collaborative annotation through effective verification of 1lm labels. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pp. 1 21, 2024.
Yuhao Wang, Liam Solus, Karren Yang, and Caroline Uhler. Permutation-based causal inference algorithums with interventions. In Advances in Neural Information Processing Systems, pp. 5822 5822-5831, 2017b.
Julia K. Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk, Teresa Deinlein, Rainer Hofmann- Wellenhof, Lue Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, and Holger A. Hacnssle. Associa- tion between surgical skin markings in dermoscopic images and diagnostic performance of deep learning convolutional neural network for melanoma recognition. JAMA Dermatology, 11551011333-11 2019.
James Woodward. Making things hoppen: A theory of causal explanation. Oxford University Press, 2005.
Chih-Kuan Ych, Boen Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. In Advances in Neural Informa- tion Processing Systems, 2020.
Jiji Zhang. Causal reasoning with ancestral graphs. Journal of Machine Learning Research, 9:1437 1474, 2008a.
Jiji Zhang. On the completeness of orientation rules for causal discovery in the presence of latent confounders and selection bias. Artificial Intelligence, (7(11-17))1873 2008b.
A Appendix: Calculations For Example Using Shapley Values
For the graph in Example 1, with the following SCM,
21

---

Published in Transactions on Machine Learning Research (05/2025)
Z1
Z3
Figure 13: Data generating process for Example 1.
= Cy
Z3 = To + 71 + +
Ze 9o + Y +
Z1 M0 Z2 + V223 23Y + EZi
ÿ = no 23 + Z1 + 12Z6
Example 1 assumes Z3 is omitted, and backdoor adjustment is done using Zz.
A.1 Shapley Value With Valid Adjustment Set
The Shapley value for Z1 given
Øz.
0.2!
(f(21) -66 f6}+
111!
111! 3! (12.22 - f2aa 3!
(12.22 - 8820}
3
2!0!
[fiz2z120) - f(zeizs)
3!
Computing each of the individual pieces
f/ = E[ 22, = - + + [2
f(z) = Ef Zz, = [o + 11 + [22
f (22 B[J'(z, 23) = B[7] + 1 1222
f(z)) = E[J'(Z,, 22, Za]] = /.
+ E Z1| + 8222
f(2,7) E[f Zz, 23) = Mb M1 12 Zz]
f(z)) = (Z) Z2, = /403 E[ / / E22
(22122.22) = E[ 2)) /30 =1 + 12
f(zz,7)) E[f'(Z,. 22, 23) = - + /4 1P| 71| + 82
And this gives the following Shapley value for Zı
Φ7. 2 [[a] - 22 40 Mo22B|Z1| + - E[21]
22

---

Published in Transactions on Machine Learning Research (05/2025)
H
(a)
Figure 14: (a) A causal diagram representing the true data generating process utilized in the simulation study.
A.2 Shapley Value With Invalid Adjustment Set
In this case, the Shapley value for Z1
0!1!
1101 f
((2 -f(z)) f(zz)
oz.
(1,)
2!
21
Computing these with Zy missing
f/2 / 22,, Zz)p(z Au + + p2E|Z2]
f(z)) / M22 21,1, Zz)p( (2) = Pb Zz + 11 + +32
f(2,7)) f212,, Zz) Mu Zs + 11 + 1222
f(2) = / 12,2,,2,
22, 7z)p( = nn Z 21 + nn [[ 12222
And the incorrect Shapley value will be
oz, (2) - E/22)) + [1]
B Appendix: Simulation Details And Implementation
HydraNet preloaded with ResNet-18 weights fine tuned on a training dataset of 2500 images, with each image containing circles, triangles, horizontal bars and vertical bars all generated from an unbiasod coin flip. The final layer of the network is replaced with heads, each used to predict the presence of circle, triangle, horizontal bar or vertical bar in the image. Each of the heads is fully connected layer of size (512, 2). All the parameters of this network except for the heads are frozen, and this network is optimized using joint loss function as
loss = Le + Lu + 2.5 * LT + Ly
Where Le is the cross-cntropy loss for the circle head, LH is the cross-entropy loss for the horizontal bar head, Lt is the cross-entropy loss for the triangle head, and Ly is the cross-entropy loss for the vertical bar head. The cross-entropy loss for the triangle head is up- weighted to encourage good triangle detection accuracy with HydraNet carlier training runs where the weight was 1 did not have acceptable accuracy. The model is trained for 15 epochs with a batch size of 64 and using the SGD optimizer with a learning rate of 0.01 and momentum of 0.09. Additionally, we schedule a learning rate decay with a step size of 7 and = 0.1.
We measure model performance on a validation dataset of 500 images, and our final network achieves the following accuracy the validation dataset. A 100% accuracy in predicting circles, 99.60% for predicting
23

---

Published in Transactions on Machine Learning Research (05/2025)
Hid Liacsiicttionn Exparimett
Kny Classificaiion Expedisent
Crenn Ceter
Initration
Bit Length
Lep Color
Thous Dote
Under pares
Porcheed Coter
EMulln
ming thape
Wing Color
Nase
Betly Cow
shape
Upparparts
Cotor
Candooeggtly
Raye Caler
Back Color
Pncumethrnex
Edge Staniityy
Edge Stabeiity
Figure 15: Results for two experiments using GRaSP-FCL. Potential causal determinants of (a) bird classifier output from bird images and (b) pneumonia classifier output from X-ray images.
horizontal bars, 97.80% accuracy for predicting triangles and 100% accuracy in predicting vertical bars. Our logistic regression model is using the sklearn library, with no penalty. The learned intercept is -1882530524, and the learned coefficients are 0.09042825, 1.56043942, 0.07410451, and 1.27937188 for circle, horizontal bar, triangle and vertical bar respectively. This full black-box model achieves 77.89% accuracy on the dataset generated according to the DGP outlined the simulation section.
For the FCI algorithm, we use the implementation in the TETRAD package, with the Chi-square inde- pendence test. Additionally, impose the background knowledge that Y must be descendant of the interpretable features.
C Appendix: sensitivity analysis with hybrid causal discovery algorithm
As alternative to FCI, also apply hybrid procedure called GRaSP-FCI to the data chest X-ray and bird images. There is no fully greedy score-based procedure for learning PAGs analogous to the GES (Greedy Equivalence Search) algorithm for learning CPDAGs (Chickering 2002). However, two-step hybrid procedure has been proposed: first learn CPDAG by using a greedy score based procedure, and then ignore edge directions and FCT-orientation rules to produce a PAG (Ogarrio et al., 2016). Details underlying this hybrid approach and proof of its correctness are described in Ogarrio et al. (2016). Though the authors use GES as the greedy score-based algorithm in that paper, more recently it has become possible substitute the GRaSP (Greedy Relaxations of Sparsest Permutation) algorithm for learning CPDAG (Lam et al. 2022 in the first stage of this two-step procedure; the result is called GRaSP-FCI. This algorithm is implemented in the TETRAD package and thus forward to run on the same data with the same pipeline (including the background knowledge described above). We use the BDeu score and Chi-square test of independence, with all other settings the same as for FCI. Results are summarized in Figure 15 In the X-ray image analysis, the ordering of fcatures by edge stability is the same for GRaSP-FCI as FCI. The ordering for the bird image analysis has substantive differences: the edge stabilities are overall lower, and the top variables in the ordering are Crown Color and Size.
the resulting undirected graph with some additional independence tests and
D Appendix: Preprocessing for Bird Dataset
For reproducibility, describe preprocessing of the bird image data.
24

---

Published in Transactions on Machine Learning Research (05/2025)
D.1 Bird Classification Grouping
The Caltech-UCSD 200-2011 Birds dataset comes with 200 categories of birds. We coarsen them using the group scheme below:
Flycatcher
041.Scissor Tailed Flycatcher
040.Olive _ided Flycatcher
043. Yellow Beeloweellllleeeeeeeeee Bellied Flycatcher
038. 1388.reat__i Cherat__rested____ FFlyctcche
042. Vermilion Flycatcher
037.Aca Flycathhe
Flycatcher
Gull
059. Califorioornia__ Gull
063.Ivory_ Guul
060. Gla.colucous Winged Gull
066. Western Gull
062.Herring Gull
064. Ring Billed Gull
061.Heermann Gull
Kingfisher
081.Pied Kingfisher
Ringed Kingfisher
080. Green Kingfisher
079. Belted Kingfisher
083. White Breasted Kingfisher
Sparrow
127. Savaaaan Sparrow
Nelsoneson__Satt Sharp Tailed Sparrow
116. J16.CChiping Sparrow
Chipping_Sparrow
114.Black Throated Sparrow
SesssrrsshppperSSpaniee Sparrow
119.Field Sparrow
1ar.Ha _Sarriss_Sprroww
Tree Sprar_wSprroow
SeasSees Sparrdoww_prrwww
118. L.s.House_Sprrroos
1hite. ThroateddSSpaaee
115.Brewer SSarroo
117.Clay Colored Sparrow
131. Vesper Vesper_Sparrow
123. Henslow HpprroowSpparrow
Fox Sparrow
25

---

Published in Transactions on Machine Learning Research (05/2025)
129..ong Sparrow
Lincoln Lincoln_SSparrow
132. W12te__e__oo Crote_dCddweede Sparrow
124.Le Conte Sparrow
113.Baird___ _Sprrrow
Tern
141.Artic Tern
144.Commni Common Tern
116...ossee Forsters Tern
143. Caspian Tern
11e5.lle Tern
147 Least _ern
142.Black Tern
Vireo
156. 156. Eyed
Vireo
154.Red Eyod Virco
152. Headed Vireo
157. Yellow Tarooo___TToaeeWWWW Vireo
155. www.Pahaddlppha_iiiiiii Vireo
155. Warbling Vireo
151.Black Capped Vireo
Warbler
168. _
Warbler
163e Baape_Maa_Warbble Warbler
Bay Breasted Warbler
162.Canada Warbler
182. Yellow Warbler
161.Blue Winged Warbler
_ Warbler
180.Wilson Warbler
171.Myrtle Warbler
_rabler
166. 166..ollen Winged Warbler
173. Barngee_rrowee_____
177. Pr7.hrotthnotary Warbler
178. Swainson Warbler
1699.Magnlia Warbler
159. aaa_aanddehe_aaaaaaaaaaaaaaaaaae
166.Chestmut_.55 Siattmut_Siddd__aaalteee Warbler
164.Cerulest Warbler
Warbler
174.Palm Warbler
Praiire Warbler
160. B Throated Siue_Warbler
179. Tennessee Warbler
770.MMurning__arrre Warbler
Nashvile Warbler
26

---

Published in Transactions on Machine Learning Research (05/2025)
1orm Batti___tti_____ee Warbler
W15..Pine_Wartieeee Warbler
Woodpecker
188. Pileated Woodpecker
189.Red Bellied__ Woodpecker
190. Red Cockaded _Woodpccerrr
187.American Three Toed Woodpecker
191.Red _ae___hadd________ Woodpecker,
192.Downy__aa poodeeeeerr
Wren
Carolina Wren
197. 197.Marsh__ Wren
196. Wren
193.Bewick Wren
198. Rock Wren
199. Winter Wren
Cactus Wren
Any other bird labels that did fit into these categories were excluded from our analysis.
D.2 Attribute Groupings
The attributes were grouped as follows:
bill shape:
0 Missing
curved (up or down), hooked, hooked seabird
dagger, needle, cone
specialized, all-purpose
- spatulate
wing color:
0 - Missing
blue, yellow, red
2
green, purple, orange, pink, buff, iridescent
rufous, grey, black, brown
4 -
white
upperparts color:
Missing
blue, yellow, red
green, purple, orange, pink, buff, iridescent
- rufous, grey, black, brown
4 white
underparts_com color
0 Missing
27

---

Published in Transactions on Machine Learning Research (05/2025)
1 - blue, yellow, red
-
green, purple, orange, pink, buff, iridescent
3 rufous, grey, black, brown
- white
breast pattern
Missing
solid
spotted
- striped
multi-colored
back color
back_colorr
-
Missing
blue, yellow, red
green, purple, orange, pink, buff, iridescent
rufous, grey, black, brown
4 - white
tail shape
Missing
- forked tail
rounded tail
3 notched tail
fan-shaped tail
5 pointed_ _til
squared tail
upper tail color
Missing
blue, yellow, red
2
green, purple, orange, pink, buff, iridescent
rufous, grey, black, brown
4 white
breast color
Missing
blue, yellow, red
green, purple, orange, pink, buff, iridescent
3 rufous, grey, black, brown
white
throat Colort
Missing
blue, yellow, red
green, purple, orange, pink, buff, iridescent
- rufous, grey, black, brown
4 white
28

---

Published in Transactions on Machine Learning Research (05/2025)
eye color
Missing
blue, yellow, red
2 green, purple, orange, pink, buff, iridescent
rufous, grey, black, brown
4 white
bill length
Missing
about_ ttt__tthe______________________ same as head
longer than head
3 - shorter than head
forehead color
Missing
blue, yellow, red
green, purple, orange, pink, buff, iridescent
- rufous, grey, black, brown
white
under _ail color
Missing
blue, yellow, red
green, purple, orange, pink, buff, iridescent
3 rufous, grey, black, brown
white
nape color
Missing
blue, yellow, red
2
green, purple, orange, pink, buff, iridescent
rufous, grey, black, brown
4 white
beloy_color
Missing
blue, yellow, red
green, purple, orange, pink, buff, iridescent
3 - rufous, grey, black, brown
white
wing shape
Missing
rounded-wings
pointed-wings
3 broad-wing!
- tapered-wings
long-wing onngwinggs
29

---

Published in Transactions on Machine Learning Research (05/2025)
size
Missing
large (arge((16-3iin)
2 - small (5-9in)
very large_ (32-72in))
medium mediim_(9991 (9-16in)
very small (3-5in)
back pattern
Missing
solid
- spotted
striped
4 multi-colored
tail pattern
Missing
solid
spotted
- striped
4 multi-colored
belly pattern
Missing
solid
- spotted
3 striped
multi-colored
primary
ccooory_color color
0- Missing
blue, yellow, red
2
green, purple, orange, pink, busf, iridescent
rufous, grey, black, brown
4 - white
leg color
Missing
- blue, yellow, red
green, purple, orange, pink, buff, iridescent
3 rufons, grey, black, brown
white
bill color
-
Missing
blue, yellow, red
-
green, purple, orange, pink, buff, iridescent
rufous, grey, black, brown
30

---

Published in Transactions on Machine Learning Research (05/2025)
4 white
crown color
Missing
blue, yellow, red
green, purple, orange, pink, buff, iridescent
3 - rufous, grey, black, brown
white
wing pattern
Missing
solid
spotted
3 striped
multi-colored
31


Reviews:
[{"review_id": "rU1M5EwJr2", "reviewer": "Reviewer_GYZw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The motivation for the proposed method is clearly described and seems reasonable.\n- The proposed method is robust to confounders even when the practitioner has not included these confounders in the set of candidate explanatory features.\n\nWeaknesses:\n- At 16 pages, the submission is quite long, with extensive discussion of the philosophy of causal inference and how the proposed method might be applied, but includes less than a paragraph of detail of the the FCI algorithm itself, which is core to the proposed method.\n- There is no obvious ground truth for the causal features of the bird and pneumonia classifiers, so we do not know whether the inferred causal features used by the bird and pneumonia classifiers are actually the correct causal features. The finding that features associated with pneumonia seem to be causal determinants of pneumonia is promising, but not conclusive.\n- The experiments do not provide a quantitative measure of the effectiveness of the proposed method versus others. The only comparison with other methods is a demonstration that, when applied to pixels, explainability methods LIME and SHAP fail to provide interpretable explanation of the bird and chest x-ray classifiers. In Section 3.3, the submission mentions that there are other causality-aware feature importance approaches that can be applied to macro-level features (though it states that they cannot account for unmeasured confounding) but none of these methods are compared with FCI.\n- Part of the introduction is a verbatim restatement of the abstract, which seems unnecessary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission proposes to use the FCI algorithm to estimate a PAG over a set of macro-level variables that is correct even if there exist confounders that are not in the set of macro-level variables. It applies this technique to a classifier trained on synthetic data, where it is shown to correctly infer the PAG when one of the true causal factors is omitted from the set of explanatory features, and to bird and chest x-ray classifiers.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The submission needs experiments that show that FCI is capable of correctly inferring the PAG in nontrivial settings. The current experiments show that FCI can correctly infer the PAG in a somewhat trivial setting and that it infers a PAG in real settings, but it is unclear if the inferred PAG in these real settings is correct. To demonstrate correctness, the authors might consider perturbing some of the inferred causal determinants to verify the inferred PAG, e.g., showing that the bird classifier is sensitive to throat color but invariant to bill color. Alternatively, it might be possible to train classifiers on intentionally confounded real-world data, determine the features the classifier uses by testing it on unconfounded data, and then analyze the ability of causal discovery methods to discover those features.\n- It would be nice to have a more detailed description of FCI and the situations under which it is and is not effective. In the current submission, it feels like it's a magical algorithm that can infer PAGs in the presence of unmeasured confounding, but given the challenges of causal inference, I suspect that it has limitations.\n- It would be nice to see a comparison of FCI with PC and perhaps other causal discovery methods, to see if the conclusions are substantially different.\n- (Non-essential) Although the submission is a pleasure to read, I feel that it could be shortened without compromising essential detail.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8julI5vzni", "reviewer": "Reviewer_c3wj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "However, there are some problems to solve.\n1.\tThe motivation of this paper is unclear. For example, why is it necessary to distinguish between variables with direct causal effects and those influenced by confounders?\n2.\tHow are micro-level and macro-level features defined?\n3.\tWhile post-hoc explainability is valuable, if the macro-level latent variables in the model are not distinguishable (e.g., features with semantic confusion), is the proposed method still applicable?\n4.\tOne aspect is particularly confusing: z is essentially already a latent variable, yet the authors introduce unmeasured confounding on top of z. Does this imply that the latent variables have not been fully recovered? Furthermore, not all black-box models align with the true data generation process, meaning there is no guarantee that z in a black-box model corresponds to the latent variables in the real data process. As a result, the model's explanations could be misleading.\n5.\tMore causal explainability work needs to be considered and discussed, such as [1][2][3].\n\n[1] Chen, Xuexin, et al. \"Feature Attribution with Necessity and Sufficiency via Dual-stage Perturbation Test for Causal Explanation.\" arXiv preprint arXiv:2402.08845 (2024).\n[2] Beckers, Sander. \"Causal explanations and XAI.\" Conference on causal learning and reasoning. PMLR, 2022. \n[3] Venkateswaran, Aparajithan, and Emilija Perković. \"Towards complete causal explanation with expert knowledge.\" arXiv preprint arXiv:2407.07338 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a causal approach to post-hoc explainability for black-box models, addressing two key issues: the reliance on micro-level features (e.g., pixels) instead of interpretable macro-level features and the unrealistic assumption of no unmeasured confounding. The authors introduce a causal graphical method that differentiates between features that causally influence model predictions and those merely associated with confounding. The authors evaluate the proposed method on the simulations and real-world datasets, enabling more meaningful explanations for black-box models without requiring access to their internal workings.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N.A.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sov0a7hRg8", "reviewer": "Reviewer_mZit", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe paper presents an approach to explain the behavior of black-box prediction algorithms by identifying the features that are most relevant for the prediction.\nThe approach is based on causal learning and uses expert annotations to identify the features.\nThe authors show that the approach can be used to identify the most relevant features on two real-world datasets, a bird classification task and a medical diagnosis task.\nThe identified features are indeed relevant for the prediction, which shows that the approach can be used to explain the behavior of black-box prediction algorithms.\nOverall, the approach can be a valuable tool for certain usecases, where expert annotations are available and the goal is to understand the behavior of the model.\n\nThe paper is well-written and easy to follow.\nThe explanations and introduction to topics are quite thorough, such that also readers not familiar with causal learning can follow the paper.\nThe experimental setup is detailed and the results are presented in a clear and understandable way.\n\n### Weaknesses\n\nThe approach has the clear disadvantage of needing manual annotations of the features. \nThis requires an expert annotator for the task, which especially in the medical domain can be quite expensive.\nFurther, novel complex features, that may not be known to the annotator, are not considered in the analysis.\nThis is in contrast to existing methods, that give pixel-level explanations and not require manual annotations.\n\nAdditionally, it is unclear how many samples are needed for really stable results.\nFor example, for the bird classification task, a dataset of 3500 images was used.\nBut already there, a considerable amount of pre-processing was done on the expert-feature annotations to reduce the space of possible features.\nThis might not be feasible for smaller datasets, where the number of samples is limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the explainability of black-box machine learning models. \nThe authors propose a novel algorithm based on learning a causal graph between the predicted label and annotated high-level features. \nThe causal graph is learned with standard causal discovery methods like FCI.\nApplied on two real-world datasets, a bird classification task and a medical diagnosis task, the authors show that the approach can identify the most relevant features for the prediction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical Changes**\n\nI encourage the authors to include a more detailed discussion on the sample size needed for the approach, and its sensitivity to noisy expert annotations.\nFurther, a discussion on how to scale this approach to larger datasets, where manual annotations are not feasible, would be valuable.\n\n**Minor Changes**\n\nThe highlighting in Figure 11 is done with a very low alpha, such that the actual content behind it is almost invisible. It would be better to use more transparent overlaps, such that the content is still visible.\n\nThe text in Figure 8 is very small. It would be better to increase the font size, such that the text is more readable.", "broader_impact_concerns": "No broader impact concerns were identified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main criticism is on the empirical evaluation, which contains real data sets but lacks causal ground truth. I took a closer look, and the results are reasonable, exceeding the standard of the community. The synthetic data, albeit simple, validates the claim, and the real data shows applicability to realistic scenarios.
Decision: Accept (poster)