Causal Dependence Plots
Authors: Joshua R. Loftus
Lucius E.J. Bynum
Sakina Hansen
Venue: NeurIPS 2024 poster (2024)
Keywords: Interpretable machine learning
interpretability
explainable AI
explainability
causality
partial dependence plots
total dependence plots
model agnostic explanations
Primary area: interpretability_and_explainability
TL;DR: We introduce a framework for creating model explanation plots based explicitly on causal relationships and illustrate several types including the popular existing method of partial dependence plots as a special case

Abstract: To use artificial intelligence and machine learning models wisely we must understand how they interact with the world, including how they depend causally on data inputs. In this work we develop Causal Dependence Plots (CDPs) to visualize how a model's predicted outcome depends on changes in a given predictor *along with consequent causal changes in other predictor variables*. Crucially, this differs from standard methods based on independence or holding other predictors constant, such as regression coefficients or Partial Dependence Plots (PDPs). Our explanatory framework generalizes PDPs, including them as a special case, as well as a variety of other interpretive plots that show, for example, the total, direct, and indirect effects of causal mediation. We demonstrate with simulations and real data experiments how CDPs can be combined in a modular way with methods for causal learning or sensitivity analysis. Since people often think causally about input-output dependence, CDPs can be powerful tools in the xAI or interpretable machine learning toolkit and contribute to applications like scientific machine learning and algorithmic fairness.

Causal Dependence Plots
Joshua R. Loftuu Department Statistics London School of Economics London, England, UK r 1rr.lffus@lsee.e ac
Lucius E. J. Bynum Center for Data Science New York University New York, NY, USA lucius@nyu. edu
Sakina Hansen
Department of Statistics London School of Economics London, England, UK aalaeenn@lse.. ac. uk
Abstract
To use artificial intelligence and machine learning models wisely we must under- stand how they interact with the world, including how they depend causally data inputs. In this work we develop Causal Dependence Plots (CDPs) to visu- alize how model's predicted outcome depends changes in given predictor along with consequeni causal changes in other predictor variables. Crucially, this differs from standard methods based on independence or holding other predictors constant, such as regression coefficients or Partial Dependence Plots (PDPs). Our explanatory framework generalizes PDPs, including them as a special case, as well as variety of other interpretive plots that show, for example, the total, direct, and indirect effects of causal mediation. We demonstrate with simulations and real data experiments how CDPs can be combined in modular way with methods for causal learning or sensitivity analysis. Since people often think causally about input-output dependence, CDPs can be powerful tools in the xAI or interpretable machine learning toolkit and contribute to applications like scientific machine learning and algorithmic fairness.
Introduction
This paper develops Causal Dependence Plots (CDPs) to visualize relationships between input variables and a predicted outcome. Motivated by explaining or interpreting AI or machine learning models [8] [6 [7] 36), for simplicity we consider supervised learning, i.e. regression or classification. We also focus on the model-agnostic or "black-box" setting, where the interpreter can query the model but not access its internal structure. Model-agnostee interpretation methods are functionally limited observing how the model responds to variation in the inputs. While this initial application forms our practical motivation, we emphasize that CDPs are more general.
Simple explanations that focus one input variable at a time understanding. However, just as with the interpretation of linear regression model coefficients, these simple relationships can be misleading. When varying one input variable, we must make some choice about what values to use for the other imputs. CDPs make this choice using an explicit causal model, and to our knowledge this is the first work that does so. We compare CDPs to other state-of- satte-of-the-are art, non-causal explanation methods like the Partial Dependence Plot (PDP) [12], Individual Conditional Expectation (ICE) [14, Accumulated Local Effect (ALE) [3] and Shapley Additive Explanation (SHAP) feature plot B3 Explanation methods may respect existing causal dependencies between predictors break them.
be powerful tools for human
Problem statement. there are causal relationships between predictors but our visualization, interpretation, explanation method does not respect them the resulting model explanation may be irrelevant or misleading [37 (54.. Such explanations could lead to incorrect decisions for regulating or aligning algorithmic systems, sub-optimal allocations of resources based on model predictions. breakdown between human feedback and reinforcement learning systems, or other forms of error and
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

harm. In scientific machine learning- -wheee explanations can be used to generate hypotheses for follow-up investigation a flawed interpretation may support spurious hypotheses. For these reasons, the causal validity of model explanations should be a top priority.
High level proposal. We wish to interpret or explain a given supervised machine learning model f(x) with p input features x (Z1...., (p)). Specifically, we want to understand how the predictions 9 = fx) of this model depend on feature zj for a given j. S j S p. PDPs and ICE plots do this by varying 2j and holding the other features x constant, where j is the (p 1) -ttupl containing all features except for zz- This implicitly assumes independence between Ij and the other inputs. Our method replaces this independence assumption with Explanatory Causal Model (ECM) for the input features. This auxiliary ECM is a tool we use to help explain f, it determines how other inputs x vary when 2j is changed.
CDP pseudo-algorithm. To construct CDP showing how f(x) depends on z a user specifies an ECM M containing the predictors x, and an intervention (Tx ) in M. The intervention changes zj, and may change other features if they are caused by xy in M. The type of intervention is chosen based on the type of causal explanation desired, with several example options demonstrated later. An explanatory dataset D (x; i 1,...,,7 .n can be given or, if unavailable, generated by M. Note that we use the notational convention where indexes observations or examples while j indexes features. The horizontal axis of the plot specified by a grid {T... k I,.... K} of possible values for zy, with & indexing grid points. For each observation X; in D, and at each grid point Jz.4
1. Use the ECM to simulate counterfactual values X for all features of observation i under the intervention I ((.,.))
2. Input counterfactual features to the prediction function f, and store the resulting counterfac- tual prediction x = f(x;,² ) in an array indexed by (i, k).
For each observation i D, construct the individual counterfactual prediction curve (āj.k: H(,) by connecting points that are adjacent on the plot grid. Plot the empirical average of these curves, which is the main output of the CDP. The individual curves can be shown or suppressed as desired. The resulting CDP shows how the model's 's predictions ý causally depend on 2; when this predictor is varied by the intervention [2;) in ECM M.
PDP and ICE algorithm. Start with the notation and setup as above but without any ECM or intervention. For each observation x; in D, and at each grid point ãz.::
1. Define XLx by setting feature Fij to the grid point āy and keeping other features x, fixed at their original values in x, from D, that is
XL,x = (Ma1) Tij + Tjkk- Tip)-
2. Compute prediction Uh, = f (x/,x) and store in an array indexed by ()).
Plotting (āx,k: (k,)) generates an ICE curve for each i, and the empirical average of these is the PDP.
Motivating example. Consider model for parental income P. school funding F, and graduates' average starting salary S, with HCM shown in the bottom row of Figure [ In the top row, the ECM functions plotted in the left panel, and the remaining panels show visual explanations of supervised models that predict S i(P, F). In this example the training data for black-box models was generated by the ECM, but later we will see real data examples where this is not the case. Blue curves show how S depends on P when P is causally manipulated without holding F constant, i.e. under the intervention do = p). Orange curves show the dependence of Š on P when F is held constant at its observed value, and coincide exactly with standard PDPs. Full definitions of these are given in Section 2 Several key takeaways:
Comparing the direct (or partial) dependence curves and total dependence curves we see there can be qualitative differences depending on the type of explanation desired, for example one can increasing while the other is decreasing. This is a consequential fact when considering how interventions may change (predicted) outcomes. Increasing P causes larger values of Š, but if the increase in P is done while holding F constant then it could cause decrease in S (or smaller increase). An intervention which does not hold other
2

---

True model
Linear model
fit
RF PCP ICE
2
-1
1.5
1.5
0.0
10
1.5
P
[b]
(c)
~ 11 1.5),
P
F
S
F = 2P +NN00, 0.22).
5 5 F - p2 +..(0.0.2))
$= F)
$ F(P.F)
Figure 1: Motivating example. Causal Dependence Plots (top row) and the Explanatory Causal Model (bottom) for the motivating example. Points show the explanatory dataset, which this example is also the training data for the predictive models. Counterfactual curves for individual points shown as thin, light lines, with averages displayed as thick, dark lines. Total Dependence (TDP) is represented in blue and Natural Direct Dependence in orange. Panel (a) shows the relationships the ECM. Panels (b-c) show CDPs for a linear model and random forest (RF) model, respectively. Panel (d) shows PDP and ICE curves for the RF model from standard software library. This identical to our NDDP in panel (c). We show this holds true in general: PDP/ICE are a special case of CDPs.
predictors constant-arguabll the canonical causal operation-cam be shown by our TDP.
Our framework includes some existing model explanation plots like ICE and PDPs as special cases. panels (c-d), and later in Theorem 2.10 we see that PDP + ICE = NDDP. A practitioner sceing only the PDP in panel (d) may conclude that "dependence" of Š P is weak, especially if P 1. The TDP in panel (c) shows stronger increasing relationship closer to the true total dependence and a more holistic view of how $ depends on P. Our work clarifies that the weaker form of dependence shown by PDPs is natural direct dependence. We also see the weak dependence empirically for SHAP and ALE plots in Figure
Explanations of models can be qualitatively different from the underlying causal relation- ships. For example, the random forest in panel (c) shows a direct dependence of Š on P that is increasing when the true direct dependence of S on P is decreasing, Predictive machine learning models may fail to capture causal dependence, and in this case studying the black-box would not necessarily help us learn about the real world. As another example, panel (b) shows that the total dependence linear model on a predictor can be non-linear, in this case because the mediator F depends non-linearly on P.
1.1 Applications
Different combinations of the predictive setting and choice of ECM generate many uses for CDPs. In general, ECMs can designed based on particular desired explanation, make use of prior domain knowledge, or be learned and estimated from data using causal learning methods in modular fashion. Importantly, the ECM does not need to contain the outcome variable y except in one special case residual plots- be discussed later.
Causal bridge. In one special case we may wish to understand how f depends on a variable which is not one of f'ss inpul features but is causally related to them. Other methods cannot do this, but CDPs can provided the ECM also contains z. For simplicity we choose notation in the rest of the paper to reflect the case where the explanatory variable is an input, but this is not a loss of generality
3

---

since we can simply define when 2 changed, can see how f depends on 2. This could be useful to probe a predictive model for fairness with respect to sensitive attribute that the model does not use directly.
== f(x) and apply CDPs to faop- Since the ECM may vary
Incomplete causal knowledge. There are various applications where an ECM does not need to be a fully specified or "correct" model for all features. First, CDPs only use the predicted- not actual- outcome. This is useful for semi-supervised anticausal learning: given causal structure among predictors only and a supervised learning model, attempt causal inference for the outcome Second, we may only require explanations or plots for one a small number of features. In such cases we only need information from the ECM about interventions on the features of interest and their causal descendants, and not other predictors. Finally, predictive models often use features that are known transformations or representations construct an FCM. For example, if the features are (x., 2 2). an ECM can simply encode the fact that x² depends causally on æ. Even if we do not know the dependency between æ and z, we can make use of our partial knowledge about the features.
12 63
inputs, and these transformations can be used to
Multiparty auditing, e.g. for fairness. An owner of predictive model may not have causal knowledge or incentives to use such knowledge. Predictive accuracy is their only concern. But separate party, like a regulator, may audit that model. This party may have more causal knowledge may be legally obliged make certain causal assumptions for the purpose of the audit, e.g. to allow disparities only along certain causal pathways and not others. Previous work applied causality faimess [0] 27 29 32 (34
due specializing in auditing,
38]
48 60 62], recourse
7 26 444 (57, and other desiderata. Existing methods like PDPs are limited to only showing direct dependence, and this may hide the full extent of unfairness or discrimination [18]. CDPs can be used to probe a black-box for unfairness in the form of total dependence or partially controlled dependence.
Explanations under covariate shift. Often a pre-trained model is used for predictions data from a different data generating process than the training DGP. We can use ECMs and CDPs to visualize how the model will behave out-of-distributin ECMs could even be chosen adversarially.
Scientific theory development. Large and complex models may be fit data where underlying structure is largely unknown. In such settings, relatively simple ECMs can be used to formulate simple hypotheses relating some predictors and plot causal dependencies check these hypotheses or generate new ones. This can also be done hypothetically, assuming an ECM for exploration.
1.2 Contributions
After defining the CDP framework demonstrate CDPs synthetic and real datasets in Section and Appendix [ including in conjunction with structural causal learning in [8.2 We compare CDPs with other state of the art competitor visualization methods, for example in Figure Theorem 2.10
establishes the first universally valid causal interpretation of PDP and ICE plots. Finally, in Section 2.7 illustrate how visualize uncertainty about the choice of ECM.
2 Methodology
2.1
Supervised lcarning models
We are given a predictive model f. possibly estimated or learned using empirical risk minimization (ERM) f = arg minkEN Σ;n 8 (xxx;, y1) with some loss function E, pre-specified function class H. and an independent and identically distributed training sample { x;) i 1...., n) with feature vectors x!_ E RP. In Section 2.6 we focus on simple mediation analysis and partition the predictor variables into subsets so that X and M both notate predictors, M being mediator.
2.2 Fundamental problem of univariate explanations
To create an explanation of model dependence on a single feature, like a plot with Ty on the horizontal axis and f on the vertical axis, we must decide what to do with the other features when varying z; along the plot axis. Most explanation methods use the same approach as the PDP and ICE plots: they hold other features fixed at values in a (auxiliary, explanatory) dataset. This may be unrealistic

---

if other features depend on æ causally, or even mathematically undefined if features are mutually constitutive, e.g. (x, x²) ) or the set {GDP, GDP per capita, population).
2.3 Structural Causal Models
Our notational conventions and definitions are influenced by 41 43 Let U be set of exogeneous noise variables, set of p = -|V observable variables, and G a set of functions such that for each j E .....: P have V, 9n (PA,, Uy), where PA, S V and Uj C U are the observable and exogencous parents, respectively, of variable Vj. Let the directed acyclic graph (DAG) g have vertices given by variables and, for each V2 V and each of the parent variables in PA, and Uz. directed edge oriented from the parents to Vj.
Definition 2.1 (Structural Causal Model (SCM)). (probabilistic) SCM M is tuple (U, V, G, Pu) where PU is the joint distribution of the exogeneous variables. This distribution and the functions G determine the joint distribution P. over all the variables in M. Finally, causality in this model is represented by additional assumptions that M admits the modeling of interventions and/or counterfactuals as defined below.
Definition 2.2 (Interventions). For the SCM M, an intervention / produces modified SCM denoted Mdd(1) which may have different structural equations G Correspondingly, some variables may have different parent sets, so the DAG representation gdo(1) may also change. We denote the new, interventional distribution as P.M,o(1), A simple class of interventions involves intervening on one variable, e.g.
= (v, ;()))
which changes how V, and all variables on directed paths from V, G are generated. An even simpler sub-class of these are the atomic interventions setting one variable Vi to one constant value which denote L, := do(1) v). Note that in this case Vj has no parents in the graph gil(!); the source of the intervention itself is outside the world of the model.
Interventions are useful for modeling changes to data generating process (DGP), for example, experiments that control a particular variable to see how its value changes other variables, or policy changes aimed at altering or removing existing causal relationships. In addition to generating new observations as a DGP, an SCM can also be used to model counterfactual values for observations that have already been determined. A counterfactual distribution is interventional distribution defined over a specific dataset with information or constraints given by some of the observed values in that data, we now describe.
Definition 2.3 (Counterfactuals). Let V be the observed variables for observations in a given dataset, PA, = v and Uy the observed and exogeneous parents of variable Vj. and and intervention that modifies any of Vj's parents. The intervention / may hold some or all of v fixed and vary Uj- + u, passing these through 9y (v, u), or through ğ (v, u) if the intervention also changes any of v + V. The counterfactuals (V u) are values V, would have taken if any of its observed and/or exogeneous parents had taken the different values (V.u). u). To define the counterfactual distribution Vvv we use the posterior or conditional (depending on our probability model approach) distribution Pulv=y to model uncertainty about U while computing counterfactual values of variables for an observation in the modified SCM Mdo(T)
Remark 2.4. Note that if the desired causal explanation uses counterfactuals, then we likely obtain the observed values from an auxiliary explanatory dataset. But since an SCM can generate data, we may also use it to generate the initial observed values and then re-use these when computing counterfactuals for the explanation.
2.4 Causal explanations
Our proposed solution to the fundamental problem highlighted for univariate explanations is to use an auxiliary ECM My and let this causal model determine how other features vary functions of
We denote these explanations as Mf; or E (f) if the context clear. In the deterministic or noiseless case, suppose know continuous functions gkj such that Iy Iky(,,)) with Pjs the identity. In this case the model M, tells us g(x,) == (o.)(4))..... Ij.... Ipj (zz)) is a curve in Rº parameterized by Zj. and generate the explanation Ef( by plotting (9(2,)) against zz.
MM))
.
5

---

G
X
E(f)
Y
M
E(f)
Y
M
E(F)
Figure 2: An ECM for predictors is used to produce an explanation E(f) of the predictive model Solid arrows represent possible causal relationships in the ECM, and dotted arrows show dependence of the model explanation on predictors. In (a) Gx denotes the subgraph of the SCM for predictors. In the mediation example (b), predictor X causes Y directly and also through mediator M. creating an important distinction between direct dependence (orange) and total dependence (bluc). The reverse causality example (c) shows variables useful in predicting Y may be caused by Y, and also be causes of prediction Y f(X, M) and the explanation of that prediction.
To extend this strategy to non-deterministic causal models we use ECM .Mx for the predictor variables with Gx its associated DAG. We represent this graphically in Figure The expressive power of SCMs allows to pose various interpretive questions and compute various kinds of explanations by performing operations in Mx.
2.5 Causal Dependence Plots
For the following definitions, we assume predictor variables X Ωx. outcome of interest Y E fly and black-box function f(x): Qix fy with outputs that may also denote Y. A structural causal model M, either assumed or learned from data, specifies causal relationships for the predictors X, i.e. it need not involve the outcome Y. Note that predictors may only be subset of the variables in M as in the "causal bridge" application discussed Section I.1
Definition 2.5 (Explanatory Causal Model (ECM)). An ECM M' augments the SCM containing predictors by including the predicted outcome as an additional variable with as its structural equation.
Generating causal explanations for involves performing abduction, action, and prediction with this ECM. large ECM graph we may suppress all arrows into Y except those from the explanatory feature and its descendants. This is to simplify the display, as in Figure 5
Additional notation and conventions. We use the shorthand f(PM), where f takes a distribution PM its argument, denote using data from that distribution the input the black-box function f. For each type of causal explanation with given Named Effect based on intervention I, we define the Individual Counterfactual Named Effect curves as the set of counterfactual curves ((PMIV=vid(() for each individual, the Named Effect Function their (empirical) expectation [(PMIV=Vd6(1)) and the Named Dependence Plot as a plot displaying all of these curves.
Definition 2.6 (Causal Dependence Plot (CDP)). Given a function f. explanatory dataset D, ECM M, and family of interventions 1s parameterized by 0, we construct plot with 0 as the horizontal axis and display Individual Counterfactual (IC) curves
IC(0) ((PM
These show the effect of intervention Io on black-box output for each individual observation in the explanatory dataset as 0 varies. The (empirical) average these (over the explanatory data) is (an estimate of) the Causal Effect Function (CEF), and plot showing the IC and CEF is a Causal Dependence Plot.
We typically apply this create plots for one explanatory feature X, at time using interventions like = (X, = 0). Horizontal axes for plots use grid over the possible values X, given by its range in datasct D. Bar graphs can be used when the explanatory feature is categorical.

---

Next is perhaps the most straightforward and important named effect.
Definition 2.7 (Total Dependence Plot (TDP)), For an intervention 1, the Individual Counterfactual Total Effect (ICTE) curves
TE(T)- f ((PM)X=xx)b())
show the total effect of intervention 1 on black-box output for each individual observation in the explanatory dataset. The (empirical) average of these (over the explanatory data) (an estimate of) the Total Effect Function (TEF), and plot showing the ICTE and TEF is Total Dependence Plot (TDP). We compute the TDP following Algorithm
Algorithm Total Dependence Plot (TDP)
Inputs: M (ECM), / (black-box predictor), DD (explanatory dataset), X, (covariate of interest)
Let X be a grid of possible values of X, Set N to the number of observations in DD
Initialize N x |X] matrix of predictions y for in X do
Define intervention
do(X, = x)
Sample counterfactual dataset Dx,+= entailed by p.MiD:do(I)
Set [t] to f(Dx.+=)
end for
Plot N lines (X, [t, (Anddividual Counterfactuals))
Plot average (X, E,EY,, (i,:)/N) ((Causal Dependence)]
Remark 2.8. In the remaining definitions, we give notation only for the individual counterfactual curves and leave the other objects implicitly defined.
We often wish to decompose how much of the total effect of X on Y is attributable to different pathways between the variables. This can be explored via direct dependence below, well with other named CDPs described in Appendix A
Definition 2.9 (Natural Direct Dependence Plot (NDDP)). Given intervention define a correspond- ing intervention J that intervenes all descendants of any variables that are changed by I, except for Y, and resets them to their observed values in dataset D. We then define the Individual Counterfactual Natural Direct Effect curves
NDE(T) ((
This quantity represents the effect of intervention I on black-bor output Y while all variables not directly intervened upon by are fixed their 'natural,' i.e., pre-intervention values in D. Algorithm demonstrates how to compute the NDDP.
From this construction of NDDP, see by comparing it to the PDP and ICE algorithm that it is equivalent to these, confirming what we observed in Figure dd).
Theorem 2.10 (PDP + ICE NDDP). When generating plots for the predictive model f using the dataset 'D and feature X,, the ICE plot curves and Individual Counterfactual Natural Direct Dependence curves identical. Hence, the NDDP identical PDP that includes ICE curves.
Proof. We have implicitly assumed both plots will use the same range for their horizontal axes. This natural as implementations use the range of the feature in the dataset, and both plots are constructed from the same feature X, in the same dataset D. Since the PDP and NDDP both contain empirical averages of their respective Individual curves suffices show these are equal at each point in the plot grid.
Consider individual i in dataset D with features x; (In .Fa...... Tip) The value of the ICE curve à for this individual is, from ( f(x'. where x == (I.1. Vis + Tip): i.e. the original
features x, but with entry set to We must show this is the same the value of the Individual Counterfactual Natural Direct Dependence curve at 2 for individual i. Applying Definition 2.9 we use interventions
I = do z) and J do(X, xj if X, is ancestor of X)).
7

---

Depestacc
Natural Depaniee
Netuaall
Figure 3: TDP, NDDP, and NIDP uncertainty bands for the salary example using the random forest model in Figure T The range of curves is induced by two candidate ECMs described in Section 2.7
The NDDP applies these in the order followed by J. First, 1 sets the value feature X, to E for all individuals, and may modify other features if they are descendants of X,, Then J intervenes on each descendant Xj of X, and resets it to its observed values in D, and in particular for individual these each reset to Tay- Hence, the value of the Individual Counterfactual Natural Direct Dependence curve for individual i also given by (x))))
Remark 2.11. Note that there is some subtlety in the assumption of using the same dataset: Bayesian probability modeling approach to SCMs may add more randomness when sampling counterfactuals. In this case, rather than the ICE and ICNDD curves being identical, the ICE will equal the expectation of the ICNDD curves over this additional source of randomness. The additional randomness is specified by priors exogenous variables, and the expectation can be estimated by more sampling at a computational cost of a constant factor. Since the usage of these plots is visual and somewhat qualitative this subtlety is not important limitation of the theorem, and it would only apply under particular modeling assumptions.
Remark 2.12. To our knowledge this is the first result establishing a universally valid causal interpre- tation of PDPs. Its most important limitation is that it applies to the model output and not necessarily the original outcome Y.
Several other types of named CDPs are described in Appendix A
2.6 Mediation analysis
Many applications involve a causal structure we refer to as a mediation triangle, with examples shown in Figure [ and Figure 26 In mediation analysis, often wish to decompose how much of the total effect of X on Y is attributable to the pathway through M and how much is direct. CDPs allow us visualize frequently studied quantities of interest in this setting including other special cases defined in Appendix I. Although mediation analysis motivates CDPs and helps build intuition, we emphasize that definiiions can be used with any structural causal model. See Section for for other, more complex examples.
2.7 Uncertainty and sensitivity analysis
There are various ways to incorporate uncertainty about the ECM into CDPs. We explore a natural first extension of the CDP that shows range between possible effect functions induced by a of auxiliary ECMs. The set of ECMs could be pre-specified or, for example, given by Markov equivalence class output by causal structure learning algorithm. Returning our motivating example from Section [] we might question whether parental income P' impacts school funding F, considering instead an SCM without mediation; P' 1 S +- F. Figure 3 shows range of possible effect functions interpolating between this ECM without the indirect effect and the original ECM in Section [ for each of the TDP, NDDP, and NIDP. In this we have assumed the same structural equations for the edges that are common to both models. These plots show range for how Š may depend on P when we are unsure how F depends on P. Figure D in Appendix shows an example with real data where we use candidate ECMs discovered by the PC algorithm. These examples are not confidence regions, but any method for producing confidence sets SCMs could also be used with CDPs to display uncertainty regions. Future work can develop additional methods for
8

---

TOP NODP
+ ICF
10 Plot Fertimanos
SHAP Pict
66 contdetion servv)
-2
cin
5.5
0.51
1.00
1:5
1.50
0.2 025
111
0.55
1.00
196
1.90
601 056
350
6.75
1:22
0.0
0.5
1.0
101
Figure 4: Comparison of CDP (a) with PDP (b). ALE (c), and SHAP plots (d) for the salary example Our TDP stands out, and all other plots are qualitatively similar.
Figure I
PKC
pjnk
TDP
NDDP
SHAP
ALE
400
400
95% confidence interva
300
300
P38
praf
pmek
200
200
100-
100
plcg
PKA
p44/42
0-
pakts473
2000
4000
6003
8000
0
2000
4000
6000
8000
PCA
PIP2
PIP3
p44/42
Figure 5: ECM for the Sachs al. 49 dataset (left), CDPs for an MI.P predictive model (center), ALE (line) and SHAP (points) plots (right). All plots visualize the effect of PKA predicted p44/42. PKA and its descendants are bolded. The NDDP (i.c. PDP + ICE), ALE, and SHAP all show an overall decrease, while the TDP shows increase. Conclusions depend strongly, qualitatively, on the specific interpretive question we ask, and causal modeling allows us to formulate questions precisely.
visualizing uncertainty, for example by leveraging sensitivity analysis based on conformal prediction 2121 [1]6]] 61).
3 Experiments
We demonstrate CDPs a series of experiments with simulated and real datasets.
Comparison with other explanatory plots. Figure 4 shows accumulated local effect (ALE) [ and Shapley Additive Explanation (SHAP) 33 feature plots for the salary example. These appear similar to the PDP, with a more weakly increasing relationship than that seen in the TDP. TDPs represent significant and novel contribution to the existing model visualizations.
Real data with domain knowledge, An ECM may be constructed using domain expertise. Figure shows ECM and CDPs for the Sachs et al. [49] dataset of expression levels of proteins and phospholipds in human cells, for which data and a ground-truus -tuth DAG are publicly available in the Causal Discovery Toolbox (24). While the actual biology of the problem is not focus here, there are meaningful takeaways from the figure. For this model, the TDP shows an increasing relationship, while the NDDP/PDP shows decrease. The overall direction of the trend in predictions based on PKA is reversed if we hold other predictors fixed. This is an important lesson for using model explanations in scientific machine learning.
Following the discussion in [45 and follow-up up ground truth DAG for the Sachs et al. 44 dataset in Figure 5 of 45 choose the edge PIP3 PIP2 in order eliminate would-be cycle.

---

Additional experiments can be found in Appendix B Notably, in Appendix B.2 we learn an ECM from data with a causal structural learning algorithm and then use it to produce CDPs. The main takeaway of the real data experiments is that CDPs can be useful in practice.
In simulation experiments we know the true DGP, so we can compare its functional form to various black-box models and their explanatory plots. Results in Appendix [B. show that CDPs are sensitive to whether the functional form assumptions of the black-box model fit the DGP. In other words, if a black-box model is a poor fit to the DGP, then CDPs can accurately explain the black-box but will not reffect the true DGP. This limitation not specific to CDPs but applies to all explanation methods. Figure also shows that different ECMs can produce different CDPs for the same black-box model, and a misspecified ECM can produce misleading CDPs.
4 Discussion
Related work. Recent work in recourse 25 uses contrastive or counterfactual explanations [56].
Some of this focuses on causal dependence 50 Blöbaum and Shimizu 44] identify the predictor
with the largest total effect, which is most applicable when assuming linearity. Zhao and Hastie [63] investigated causal interpretations of PDP, aiming for causal inference about the underlying DGP, and showed that when the DGP satisfies the backdoor criterion 39 then a PDP visualizes the total effect (TE) of predictor. Cox Jr [1] observed an association between partial dependence plots and NDE, an equivalence we formally establish in Theorem 2.10 to our knowledge the first such result. Lazzari et al. [30] weight observations when computing PDPs. There has been some recent work creating causal variants of SHAP [ 19 23 58), and in future work we will explore comparisons of appropriate special cases of CDPs with these. We are not aware of any previous causal explanation work with the generality of CDPs.
Limitations. Causal modeling always involves some limitations 15
47
For CDPs, full specification
of an ECM can be a strong assumption. However, in Sections 1.I and 2.7 we discussed some ways this can be relaxed. In general. if a causal explanation is desired or necessary, then we cannot avoid making causal assumptions. Model-agnostie explanation methods also always have certain limitations 36). For example, if the predictive model fails to fit the DGP, then any model explanation will also fail if our interpretive goal to learn about the DGP 63 CDPs may be misleading if the true DGP differs in important ways from the ECM, as shown in Figure 6 However, standard PDPs and similar explanation methods also require auxiliary explanatory data, and that data may also differ from the target DGP. So this is not an additional limitation specific to method.
Conclusion. Causal Dependence Plots use an explanatory causal model to create plots with causal interpretations. This allows us to use the powerful language of structural causal models to pose and answer a variety of meaningful questions. Our framework generalizes Partial Dependence Plots, which Theorem 2.10 shows it includes as a special case, and allows other kinds of causal interpretations we have not seen previously explored in the literature. Future work in this direction could expand on some canonical causal structures for useful applications, or interface with other kinds of models, for example extending to non-tabular data by applying causal representation learning. Relating explanation methods to Pearl's 'ss ladder of causation [40], most previous interpretable machine learning and explainable Al methods like HaPs-concert associations and hence are confined to the first rung of the ladder. With CDPs we ascend the ladder, creating model interpretations intended to change the world. Interpretability provided the initial motivation for CDPs, but since plots are qualitative CDPs also open the door for future work on causal methodology that relaxes assumptions while maintaining visual validity.
Broader Impacts. There are many potential societal consequences of our work: essentially those shared by all tools for model interpretability and explainability. Model explanations can be misleading. either due to error or intentional deception. When a user is convinced by a flawed model explanation to reach misguided conclusions about a model, they may make harmful or sub-optimal decisions about how or whether to use that model. For example, if an explanation tool is used to assess the fairesss of a model, a flawed explanation could lead to the conclusion that a discriminatory model is fair or that a fair model is discriminatory. In applications related to science, a flawed model explanation can lead to wasting resources pursuing a dead-end hypothesis or to missing out on an important discovery. Similarly, poor business decisions can be made the basis of flawed explanations.
10

---

References
[1] Emanuele Albini, Jason Long, Danial Dervovic, and Daniele Magazzeni. Counterfactual shapley additive explanations. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages
2022.
[2] Altmann, J Bodensteiner, Dankers, T Dassen, N Fritz, Gruber, et al. Limitations of interpretable machine learning methods, chapter 3-4. 2020. URL https://sldss 1mu github. io/ mettommmethhhsss..mmmmeeemmme limi tatiins/
[3] Daniel W Apley and Jingyu Zhu. Visualizing the effects of predictor variables in black box supervised learning models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(4):1059-1086 2020.
[4] Patrick Blöbaum and Shohei Shimizu. Estimation of interventional effects of features prediction. 20/7 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6. IEEE, 2017.
[5] Lucius Bynum, Joshua Loftus, and Julia Stoyanovich. Disaggregated Interventions to Reduce Inequality. In Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1-13. Association for Computing Machinery, New York, NY, USA, October 2021. ISBN 978-1-4503- 8553-4. https://:///di org/ 10. 1145/3465416 3483286
[6] Lucius Bynum, Joshua Loftus, and Julia Stoyanovich. Counterfactuals for the future. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[7] Lucius E.J. Bynum, Joshua Loftus, and Julia Stoyanovich. A new paradigm for counterfactual reasoning in fairness and recourse. In Kate Larson, editor, Proceedings of the Thirty-Third Inter- national Joint Conference on Artificial Intelligence, IJCAI-24, pages 7092-7100, International Joint Conferences Artificial Intelligence Organization, 2024. doi: 2244963/jjaai.2224778. Main Track.
URL https:://oi..oo org/ 10. 24963/4433222 ijjcaicccaiiiii 2024/784
[8] Diogo Carvalho, Eduardo M. Pereira, and Jaime Cardoso. Machine Learning Interpretabil- ity: A Survey on Methods and Metrics. Electronics, 8(8):832, August 2019. ISSN 2079-9292. doi: (0.339//eectronncc88008333.. URL https://wwwt mdpi madpi.com///e//eo 2ttp/2079--222 /21/2229999999////8/2 Number: 8 Publisher: Multidisciplinary Digital Publishing Institute.
[9] Victor Chernozhukov, Kaspar Wüthrichh and Yinchu Zhu. An exact and robust conformal inference method for counterfactual and synthetic controls. Journal of the American Statistical Association, 1 1866536):184911888 2021.
[10] Silvia Chiappa. Path-specific counterfactual fairness. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7801-7808, 2019.
[11] Louis Anthony Cox Jr. Modernizing the bradford hill criteria for assessing causal relationships observational data. Critical reviews in toxicology, 4888):682-712, 2018.
[12] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[13] Christopher Frye, Colin Rowat, and Ilya Feige. Asymmetric shapley values: incorporating causal knowledge into model -agnostic explainability. Advances in neural information processing systems, 33:1229-1222 1239, 2020.
[14] Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin. Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. journal of Computational and Graphical Statistics, 24(1):44 2015.
[15] Sander Greenland and Mohammad Ali Mansournia. Limitations of individual causal models, causal graphs, and ignorability assumptions, as illustrated by random confounding and design unfaithfulness. European journal of epidemiology, 10:1101-1110, 2015.
11

---

[16]
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A Survey of Methods for Explaining Black Box Models. ACM Computing Surveys, 51(5): 13:422, August 2018. ISSN 0360-0300. doi: 10.1145/323600) URL
hattps://dol http:///o..orr///////
10 /ttp://....rrr/11111113/32336000 111...1311133330999
[17] David Gunning, Eric Vorm, Jennifer Yunyan Wang, and Matt Turek. DARPA's 's explainable AI (XAI) program: A retrospectivee Applidd Al Letters, 2(4):e61, 2021. ISSN 2689-5595. doi: 10.
1002/ail2.61. URL httpe:///onlimelibraay wiley wttp://o/aab//111100///////////0 _eprint: http:///ouiiiluuaaaa/o/////o/ooa
61
[18] Sakina Hansen and Joshua Loftus. Model-agnostic auditing: lost cause? In CEUR Workshop Proceedings, volume 3442, 2023.
[19] Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models. Advances in neural information processing systems, 13:4778-4788 2020.
[20] J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, (3):90-95, 2007. doi: 10. N0.100 URL hattps://natplotliib org/ Licensed under the PSF license.
[21] Ying Jin, Zhimei Ren, and Emmanuel J Candès. Sensitivity analysis of individual treatment effects: A robust conformal inference approach. Proceedings of the National Academy of Sciences, 120(6):e2214))) 148891 20, 2023.
[22] D. Jomar, Galli, and S. Kiran. PyALE: A python implementation of Accumulated Local Effects. 2024. URL attps://gitte com cow/DanaJomarrPyyaaaoooaao Licensed under the MIT License.
[23] Yonghan Jung, Shiva Kasiviswanathan, Jin Tian, Dominik Janzing, Patrick Blöbaum, and Elias Bareinboim. On measuring causal contributions via do-interventions. In International Conference on Machine Learning, pages 10476-10501. PMLR, 2022.
[24] Diviyan Kalainathan, Olivier Goudet, and Ritik Dutta. Causal discovery toolbox: Uncovering causal relationships in python. Journal of Machine Learning Research, 21(37):1-5, 2020. URL potts::/giiee
Licensed under
the MIT License.
[25] Amir-Hossein Karimi, Gilles Barthe, Bernhard Schölkopf, and Isabel Valera. A survey of algorithmic recourse: definitions, formulations, solutions, and prospects. arXiv:2010.04055 [cs,
stat], March 2021. URL httt:://arrriiiii hwwp:/..rrabbbabbb aaaaa///10.. 04050 arXiv: 2010.04050.
[26]
Amir-Hossein Karimi, Julius von Kügelgen, Bernhard Schölkopf, and Isabel Valera. Towards Causal Algorithmic Recourse. In Andreas Holzinger, Randy Goebel, Ruth Fong, Taesup Moon, Klaus-Robert Müller, and Wojciech Samek, editors, xxAI Beyond Explainable Al: International Workshop, Held in Conjunction with ICMI. 2020, July 18, 2020, Vienna, Austria, Revised and Extended Papers, Lecture Notes in Computer Science pages 139-166. Springer International Publishing, Cham, 2022. ISBN 978-3-031 -0408-3-- doi: lhttps://det orgo 1000000//90 /978-378-33 00- 000-04000---00000000000000000000 2__
8. URL
[27] Niki Kilbertus, Mateo Rojas Carulla, Giambatista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. Avoiding discrimination through causal reasoning. Advances in neural information processing systems, 30, 2017.
[28] Matt Kusner, Chris Russell, Joshua Loftus, and Ricardo Silva. Making Decisions that Reduce Discriminatory Impacts. In Proceedings of the 36th International Conference on Machine Learning, pages 3591-3600. PMLR. May 2019. v97/kusner 19a. html
mlr. press/
ISSN: 2640-3498.
[29] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual Fair- ness. In Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URI. lnttps://proceedingg http:///rrooeendneeeeeereeeeeeep wttp:/sew....pppppeeeeeeeeeeeeee hetprris..c/aappa////////aaaal 4456cc07eccc3227777166fffff66663 http:///..222..........ooaa..a.. Attrratt.tttaaattttt.mm.
12

---

[30] Matilde Lazzari, Jose M Alvarez, and Salvatore Ruggieri. Predicting and explaining employee turnover intention. International Journal of Data Science and Analytics, 14(3):279-255 2022.
[31] Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Suciety Series B, 83(5):911-933 2021.
[32] Joshua R. Loftus, Chris Russell, Matt J. Kusner, and Ricardo Silva. Causal Reasoning for Algorithmic Fairness. OrXiv::8055 805.05859 [cs], May 2018. URL http://arxii....//p//// www..orgggago abs/ 1805. 05859
arXiv: 1805.05859.
[33] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017. URL httpps:::///it com/ Licensed under the MIT License.
shap/shar shap
[34] Karima Makhlouf, Sami Zhioua, and Catuseia Palamidessi. Survey on causal-based machine learning faimess notions. arXiv preprint arXiv:2010.09553, 2020.
[35] Olvi L Mangasarian and William H Wolberg. Cancer diagnosis via linear programming. Technical Report 5, 1990.
[36] Christoph Molnar. Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. 2022.
book/
[37] Raha Moraffah, Mansooreh Karami, Ruocheng Guo, Adrienne Raglin, and Huan Liu. Causal interpretability for machine rations Newsletter, 22(1):18 33, 2020.
methods and evaluation. ACM SIGKDD Explo-
[38] Razich Nabi and nya Shpitser. Fair inference on outcomes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[39] Judea Pearl. [bayesian analysis in expert systems]: Comment: Graphical models, causality and intervention. Statistical Science, 8(3):266-269, 1993. ISSN 08834237. URL http:
// jsstor org/ htt/:/taaleee22599
[40] Judea Pearl and Dana Mackenzie. The book why: the new science of cause and effect. Basic books, 2018.
[41] Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CumbriidgeUniiverssyyers 19(2), 2000.
[42] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011. URL htttp://ssckktt -earnn.eeeee...eeee org/ Licensed under the BSD License.
[43] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. MIT Press, 2017.
[44] Rafael Poyiadzi, Kacper Sokol, Raul Radris-Ro Tijl De Bie, and Peter Flach. FACE: Feasible and Actionable Counterfactual Explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 344-350. Association for Computing Machinery, New York, NY, USA, February 2020. ISBN 978-1-4503-7110-0 httpss/s:///oi..oo org/10
1145/3375622 21133377677777755000
[45] Joseph Ramsey and Bryan Andrews. Fask with interventional knowledge recovers edges from the sachs model. ArXiv, abs//805 805.03108, 2018.
[46] Jonathan Richens and Tom Everitt. Robust agents learn causal world models. In The Twelfth International Conference on Learning Representations, 2024. URL httpps://peerreieee net:/forumnid-ppoo0Booww/
[47] Paul Rosenbaum. Observation and experiment: An introduction to causal inference. Harvard University Press, 2017.
13

---

[48] Chris Russell, Matt Kusner, Joshua Loftus, and Ricardo Silva. When Worlds Collide: Integrating Different Counterfactual Assumptions in Faimness. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL
ieeurrrreednnnnnerrrrrrprrppp nwwrinuu...... ctapp.....e.ee.oo pappr/p017//aaas
[49] Karen Sachs, Omar Perez, Dana Pe' Douglas A. Lauffenburger, and Garry P. Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308(5721): 523-529, 2005. doi: 10. 126/science. 105809. URL htttss:/www..r /rt/////oor//////////////
abs/10. 1126/122222
1126/science..e.c science. 1105809
[50] Numair Sani, Daniel Malinsky. and Ilya Shpitser. Explaining the behavior of black-box prediction algorithms with causal learning. arXiv preprint arXiv:2006.02444 2020.
[51] Moritz Schauer. Causalinference..k Causal inference, graphical models and structure learning in julia. 2022. URL httpss///iithbb http://///..aabb.ooo/////s/a//// wwt.:///cchaaeerraassaaaaaennnnn j1 Licensed
under the MIT "Expat" License.
[52] Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 459-466, 2012.
[53] Amit Sharma and Emre Kiciman. Dowhy; An end-to-end library for causal inference. arXiv preprint arXiv:201 1.04216, 2020, URL httptt:///ittttbbbccc under the MIT License.
Implications for explainable ai. International Journal of wwtp:/thhuu.ccooooop--oomt whtp:/yyyww/y//eee dowhy Licensed 2021.
[54] Donghee Shin. The effects of explainability and causability perception, trust, and acceptance: Studies, 146:
[55] Peter Spirtes, Clark Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
[56] Ilia Stepin, Jose M. Alonso, Alejandro Catala, and Martín Fereira-Faritte A Survey of Con- trastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelli- gence. IEEE Access, 9:1 (1197--1200) 2021. ISSN 2169-3536. doi: 10. 10111//CSSS..0022 3051315. Conference Name: IEEE Access.
[57] Berk Ustun, Alexander Spangher, and Yang Liu. Actionable Recourse in Linear Classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT '19, pages 10-19, New York, NY, USA, January 2019. Association for Computing Machinery. ISBN 978-1-4503-6 188-1-1503-625--55 doi: 10.114573285560.3383 32287766.. URL https:://oi. org. 10. 1145/ 6287560.3287566
[58] David Watson. Rational shapley values. In Proceedings of the 2022 ACM Conference Fairness, Accountability, and Transparency, pages 1083-1094, 2022.
[59] William Wolberg. Breast Cancer Wisconsin (Original). UCI Machine Learning Repository, 1992. URL https:///oii org/ 10. 24432/C5HP4Z Licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) license.
[60] Ke Yang, Joshua R. Loftus, and Julia Stoyanovich. Causal Intersectionality and Fair Ranking. In Katrina Ligett and Swati Gupta, editors, 2nd Symposium on Foundations of Responsible Computing (FORC 2021), volume 192 of Leibniz International Proceedings in Informatics (LIPlcs), pages 7:177:00 Dagstuhl, Germany, 2021. Schloss Dagstuhl - Scibniz-Zentrre für Informatik. ISBN 978-3-95977-187700 doi: 10.4230/LIPIss.FORC FORC.2s.. URL https
//drops.dagstuhh http://ppp///ooootteee///0211110 13875
ISSN: 1868-8969.
[61] Mingzhang Yin, Claudia Shi, Yixin Wang, and David M Blei. Conformal sensitivity analysis for individual treatment effects. Journal of the American Statistical Association, pages 1-14, 2022.
[62] Junzhe Zhang and Elias Bareinboim. Fairnsss bacision-making- the causal explanation formula. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
14

---

[63] Qingyuan Zhao and Trevor Hastie. Causal interpretations of black-box models. Journal of Business Economic Statistics, 39(1):272-281, 2021. doi: 0010000077500115201114449993 URL. 10000000000001111 2019. 1624293
15

---

A Methodology supplement
A.1 Additional CDPs
For completeness we include here definitions for several other important, named CDPs.
Definition A.1 (Partially Controlled Dependence Plot (PCDP)). Consider intervention I affecting a subset of variables in Gx and atomic intervention C that holds constant disjoint subset of variables. The Individual Counterfactual Partially Controlled Effect curves
PCE(T, C) = {(PMixxxxol..0))
(5)
represent the effect of intervention on black-box output Y while other variables are set (via intervention) to specific constant values. We compute the PCDP via Algorithm
Definition A.2 (Natural Indirect Dependence Plot (NIDP)). Consider atomic intervention / and corresponding intervention K that removes from 9x all outgoing edges from any of the nodes intervened upon by intervention / and sets those nodes to their observed values in the explanatory dataset. For example, if do( A a, B b), then intervention K will remove all outgoing edges from A and B and set and to their original observed values. We then define Individual Counterfactual Natural Indirect Effect curves
NIE(1) f(PM** [X=xxx((K))
(6)
Notice that intervention is performed before intervention K. This quantity represents the effect of intervention I on black-box output Y that is due only any indirect pathways Y. We compute the NIDP following Algorithm The difference between two values of this function can be used to express the natural indirect effect as special case.
A.2 Algorithms
This section describes additional algorithms including several special cases of named CDPs.
Algorithm 2 Explanatory Causal Model (ECM)
Inputs: M (SCM), (black-box predictor), X (covariates used by black-box)
Output: M' (SCM)
Make copy M' of SCM M and perform all subsequent operations this copy Add node for Y causal graph G' of SCM M'
for x in S do
Add edge in G' from Ý end for
Set structural equation for node y f Set exogenous variable Up to
Algorithm 3 Partially Controlled Dependence Plot (PCDP)
Inputs: M (ECM), ff (black-box predictor). 'D (explanatory dataset), X, (covariate of interest), C (intervention controlling other variables in M))
Let X be a grid of possible values of X,
Set N the number of observations in D Initialize N x XX| matrix of predictions y for X do
Define intervention / = do C) Sample counterfactual dataset DD, D entailed by Did(IT
Set Y 2² f(D+zz,))
end for
Plot N lines (X, f(,, :)) (Indiiviual Counterfactuals)
Plot average (X, E, :]/N) ((Causal Dependence))
16

---

Algorithm 4 Natural Direct Dependence Plot (NDDP)
Inputs: M (ECM), f (black-box predictor), D (explanatory dataset), X, (covariate of interest)
Let X be a grid of possible values of Xx
Set N the number of observations in D
Initialize N x [X] matrix of predictions Y
Get all descendants of X in M, excluding Y. and store in C Get observed values of all variables C and store in c Define intervention J = do(C = c)
for x: in X do
Define intervention / = do x))
Sample counterfactual dataset Dx,+= entailed by (.M(D.do(),))
Set [t, f(xx.tz)
end for
Plot N lines (X, Yi, :)) ((ndividual Counterfactuals))
Plot average (X, Σ, (//N) ((Causal Dependence)
Algorithm 5 Natural Indirect Dependence Plot (NIDP)
Inputs: M (ECM). f (black-box predictor), D (explanatory dataset), (covariate of interest)
Let X be a grid of possible values of X,
Set N to the number of observations in D
Initialize x |X matrix of predictions y
Get all descendants of X, in M, excluding Ý, and store in
Make copy M' of SCM M for x in C do
Remove all incoming edges to from M'
end for Define intervention -= do(X, Xs = x))
for x in X do
for in N do
Sample counterfactual observation de for unit entailed by PM D[i)a((1)
Get counterfactual values of all variables in C from observation and store in c, Define intervention J = do(X, = I, C = c,)
Sample counterfactual observation d' for unit i entailed by ((i()D(),do()))
Set Yii x]] <- d'[[v] for index y corresponding node Y
end for
end for
Plot N lines (X, [i, ((ndiividual Counterfactuals)
Plot average (X, Σ, 1/N) ((Causal Dependence))
B Experiments supplement
B.1 Model misspecification
Consider the non-linear mediation example with the following DGP:
Ux, UM, UM Uy ~ 1)
X Ux
Mi = x33 + UMs
2
Ma = x3 + UNa
4
Y M² M²
x2 + Uv
17

---

Total Dependiece
Partielly Zontrol ed Dependence
Natural Direct Dependence
Natural Indirct Dependence
-25
-20
200
200
0-
332
X00
200
100
-00
140
0
Figure 6: CDPs for the simulation example with data from 7.. shown for 'good' black-box and correct ECM (top row), a 'bad' black-box model and correct ECM (middle row), and a 'good' black-box and misspecified ECM (bottom row).
Effect on prediction (centered) Effect prediction (centerrd)
ALE Plot Continuous
SHAP Plot
40
40
95% confidence interval
for
-20
-20
2
ALE Plot Continuous
SHAP Plot
200
200
confidence
SHAP value for
100
*
100
50-
50
0-
X
Figure 7: ALE and SHAP for the simulation example with data from 77 shown for 'bad' black- box model (top row), and 'good' black-box (bottom row). Both ALE and SHAP show similar relationship as NDDP for both models.
We use this DGP to fit two different black-box models: one model that assumes the correct functional form (i.e., the relationship for Y shown in the DGP above), and an 'incorrect' model that predicts Y via linear regression. We use two different ECMs to construct CDPs, one which is the true DGP and one which incorrectly assumes the structure M, X + My. F shows the CDPs for each of these models using the black-box training data as the explanatory data. Similarly, Figure 7 shows SHAP and ALE plots using the same explanatory data.
We can glean couple insights from Figure First, CDPs are sensitive to whether the functional form assumptions of the black-box model fit the ground truth data generating process. Good explanations for y may be different from good explanations for Y if the black-box model is poorly specified. The second is that a misspecified ECM can produce bad explanations even when the black-box model correctly fits the true causal relationships in the DGP.
In Figure we see that ALE and SHAP produce similar explanations to the PDP/NDDP. We saw this previously for the random forest black-box model in the example from Figure 1
B.2 Real data with structural causal learning
The Breast Cancer Wisconsin (Original) dataset 335 is a publicly available dataset often used to test
algorithms on medical data. The dataset contains 9 ordinal variables, which represent attributes of the cells within breast mass: Clump Thickness, Uniformity of Cell Size, Uniformity of Cell Shape,
18

---

Total dence
Partilly Contectied Depancence
Natural Direct Depundence
Natural al indirect Depandence
3.0
20
3.
24
30
33
Marginal Adhesion
Cell Shape
Cell Size
Normal Nucleoli
Tumor Class
Class
Thickness
Figure 8: Breast cancer data example. CDPs for a random forest classifier and predictors Clump Thickness (first row) and Uniformity of Cell Shape (second row). Structural graph G3 for the ECM learned by the PC algorithm (last row). The outcome Class is binary: for benign, 4 for malignant. For the undirected edge between Cell Size and Cell Shape, we investigate the sensitivity to the different options Figure Note: Our intention is not to make conclusive scientific statements, but only to demonstrate how CDPs could be used in conjunction with causal structure learning.
Marginal Adhesion, Single Epithelial Cell Size, Bare Nuclei, Bland Chromatin, Normal Nucleoli and Mitoses. The outcome variable is the class of the breast tumor, benign or malignant.
We a causal structural learning algorithm, specifically the PC algorithm (55) implemented in Julia Inference [51] to learn a DAG for this dataset, on a smaller subset of predictor variables
for simplicity. Figure & shows the resulting DAG and CDPs for a random forest model to classify the Class variable.
This shows CDPs can be combined with other causal methods like structural learning algorithms. The PC algorithm output had an undirected edge between Cell Size and Cell Shape. We next explore the graph structures consistent with this uncertain edge.
Figure 9 shows the TDP, NDDP, and NIDP for a learned additive noise model with three different structures consistent with Gn: (1) with the edge Cell Shape + Cell Size, (2) with the edge Cell Size
+ Cell Shape, and (3) with no edge between Cell Size and Cell Shape. This figure shows that the takeaway about cell shape impacting tumor class is indeed sensitive to our choice about the uncertain
edge, particularly for the TDP.
B.3 Individual curves showing heterogeneity
One criticism of PDPs is that they may hide heterogeneity or individual variation, and for this reason it may be good practice to include the ICE curves in any PDP. Just like the individual curves in an ICE plot, the individual counterfactual curves in our CDPs can show important effects that are hidden by averaging as illustrated Figure TO In our implementation the default settings for CDPs-an our recommendation-is to show these individual curves.
B.4 Causal dependence for residuals
CDPs are applicable not only to understand model outputs but also to understand model performance. For example, CDPs can be used to probe residuals (or other measures of error) under distribution
19

---

Daper
Varuua Deperdence
Totusa iddlect Deperdence
Undoesty ond
Figure 9: Total Dependence Plots, Natural Direct Dependence Plots and Natural Indirect Dependence
Plots for the Breast Cancer Wisconsin dataset under three possible DAGs consistent with the PC algorithm output: (1) with the edge Cell Shape - Cell Size, (2) with the edge Cell Size + Cell Shape, and (3) with no edge between Cell Size and Cell Shape.
Total Dependence
1.00
0.75
0.25
0.50
-075
-0.
00.
-0.1
0.0
0.1
0.2
0.3
Figure 10: Individual counterfactual curves can show heterogeneous effects. In this example the relationship positive for some individuals and negative for others with average effect of zero.
Residuals (RF)
TDP Residuals (RF)
TDP Residuals (Correct Mode
0.0
0.00
0.25
0.50
0.75
1.00
1.25
1.50 a.0a 0.25
0.50
0.75
3.00
1.25
1.50
0:00
0.25
0.50
0.75
1.00
1.25
1.50
Residuals (RF)
TDP Residuals (RF)
TDP Residuals (Correct Mode!)
2.5
-2.5
Figure 11: Regular versus CDP residuals for the example in Figure 1 plotted against feature the top row and F in the bottom row. Model multiplicity means two models can produce nearly the same predictions, with high accuracy, while using different functional relationships. Accuracy can only show if the model is "observationally correct," (left column) while CDPs can help determine if the model also "causally correct" (middle vs right columns)
20

---

shift. Figure 11 demonstrates this with the salary example in Figure 1 Using the same RF model, residuals show no trend with respect to parental income P nor school funding F. However, TDP over the residuals reveals the random forest model learns causally incorrect functional form for S, even if it fits the training data well. By comparison, only the causally correct model shows no trend residuals with a CDP. This is an empirical verification the correspondence between robustness to distribution shift and causal learning [46].
C Code and reproducibility
Predictive models were fit using scikit-learn
42 CDP implementations make use of causal
modeling functions in dowhy [53). Figures were generated with matplotlib (20]. For causal structural learning we used the PC algorithm [55] implemented in Julia MausalInferent [51]. We
used the Python implementation of ALE plots in 22]. In experiments
used the Breast Cancer
Wisconsin (Original) dataset [35 59] and Sachs et al. 49 dataset. No specialized hardware is required run the experiments as they are not computationally costly and can be reproduced on a personal computer. Our code to implement CDPs, run the experiments, and produce figures is available at this repository: https://g:i: //github. http://ggghhuu...cccca/saaaa//// cotp///uaallyypoeeesss/oooorarre
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction are that we developed a new type of model explanation plot that makes of causal information about predictors. These claims accurately reflect the paper's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed limitations a subsection the Discussion section, as well as throughout the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: Most of the theoretical contributions of this paper consist in novel definitions. Our theoretical result establishing the equivalence of PDP+ICE with NDDP proven by comparing their construction and holds without any additional assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: material.
Results are reproducible from the repository linked in the supplemental
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
Results are reproducible from the repository linked in the supplemental
material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidefines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips /nips.cc/puu cubblccguuide/// www.euumiisiioooollll...oo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The setup for each experiment is described in the paper and full details are given in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The relevant methods for inference, that is, for non-parametric causal effects, are an area for future and ongoing research. There is not currently a consensus on which of the existing methods would be appropriate. Different types of CDPs may require completely different approaches for inference. In the present work we have not made claims that depend strongly statistical significance. Future work using CDPs will have to choose an appropriate method of uncertainty estimation depending on the details of the given application, or, if none available, develop new methods.
Guidelines:

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our experiments are not computationally costly and can be reproduced on personal computer.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpa/rpp....ppppppppccaaaaaaaaa
Answer: [Yes]
Justification: The research in this paper did not use human subjects or sensitive data, We have transparently communicated the limitations and potential impacts.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
25

---

Broader impacts are discussed, and are essentially those common to all model explanation/ntteepreation tools.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Code packages and datasets are cited and citations include licensing informa- tion.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
26

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: Code is documented in the linked repository.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing research with human subjects.
Guidelines:
27

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "BVwAjqpdnz", "reviewer": "Reviewer_6pua", "summary": "The authors tackle the problem of evaluating the dependence between the inputs and output of black box machine learning models.  Generally, analysis of this type of dependence is done in a univariate manner, holding constant all but one variable and visualizing how the outcome changes as we vary that one variable.  However, when the variable of interest is causally related to other input variables, these results may be misleading and only show part of the picture.  To help elucidate the bigger picture in these cases, the authors propose a framework that generalizes partial dependence plots, allowing for the visualization of the dependence between input variables of interest and the outcome in terms of metrics like total, direct, and indirect effects.  The authors demonstrate this approach on simulated and empirical data.", "questions": "I think these are mostly covered by the Weaknesses section.", "limitations": "The biggest weakness of this approach is the need to specify the ECM structure ahead of time.  To the authors' credit, they are upfront about this and provide multiple ways to ease this challenge, such as using causal structure learning methods (such as in the example in the appendix) or by allowing the user to supply multiple candidate models.\n\n\nBased on the authors' responses, I am raising my score from a 4 to a 6.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I really appreciate what this paper is trying to do.  Explainability of black box models is only getting more important, and providing easy-to-use visualization tools can be invaluable when trying to understand the uses and limitations of these models and how they can be used in practice.  The issue of univariate explanations and what to hold constant is described quite well, helping to lay a strong motivational basis for the authors' work.  I also think the writing is generally clear, and I appreciate the authors putting a motivating example very early on.", "weaknesses": "I really want to like this paper.  I love visualization tools and explainability, so work like this appeals to me.  However, I feel like the way this work is presented does it a disservice.  For a paper introducing a new visualization and explainability approach to be adopted, readers need a clear picture of how to apply these types of visualizations to their problems, what sort of results they can expect, how to interpret those results, and then what to do with them.  The authors do provide an example early on in Figure 1 with some key takeaways and some brief discussion of results in the Experiments section.  However, the discussion is generally both too high-level (such as saying \"this plot shows X\" without clarifying how to read X of the plot) and stops short of where actual application would need to go (such as saying \"we can see from the plots that X and Y differ\", without describing why that is significant and how a practitioner should interpret that difference).  This results in a paper that seems to have some interesting visualization ideas and methods but that I feel would struggle to see the methods actually adopted in practice without clearer application guidance.  I struggled a lot with what score to give, and I'm absolutely open to changing it based on other reviewers' comments and the authors' response, but for now, I can't quite vote for accept given these issues.\n\nMore specifics:\n\nFigure 1 is close to being very helpful as an early intuition-builder, and, with a little bit more description, could actually be very useful.  I think my main issues with it as-is are:\n- The left 3 plots have the orange and blue lines as defined in the figure description, but the description doesn't say what the light grey points are in the plot.\n- The caption describes the orange lines as \"Natural Direct Dependence\", a term that is not actually defined until page 7.  The text does, at least, describe the orange lines as the dependence \"when F is held constant at its observed value\", but then describe it as coinciding \"exactly with a standard PDP\", another term which is not defined outside the appendix.\n- I actually really like the three points of takeaways from Figure 1.  However, at least for me, they're missing a step: what exactly are you seeing in the plots that leads to these takeaways?  For example, the first takeaway is that \"there can be qualitative differences between direct (or partial) dependence and total dependence.\"  However, given that we're still in the introduction and these terms haven't been clearly defined yet, you should clarify that you're concluding this by comparing the blue and orange within each individual plot and ideally also what this means semantically for the example. (i.e., If we just looked at total dependence, we may conclude X, but this plot shows that partial dependence actually behaves like Y) The third takeaway is also mostly a great description, but I think I'm still missing something.  Does the takeaway that random forest messes up on direct dependence mean that the random forest model is incorrect?  If all I had done was train a random forest model and saw that result, would I have any way of knowing that it's incorrect?\n\nEven as someone who knows what counterfactuals are, I found your definition of counterfactuals in Definition 2.3 confusing.  It sounds like we're reasoning about interventions on $V_j$, and this intervention could be setting it to a constant or defining a new function for its dependence on its parents. (based on Definition 2.2) However, Definition 2.3 also describes the possibility of \"also do[ing] an intervention that changes any of the values in $PA_j$\".  I'm also not sure what it means that you \"may hold some or all of $v$ fixed vary $U_j := u$\" - if we're varying $U_j$, does that mean the intervention is touching all of the exogenous parents of $V_j$ but maybe only some of the observable parents? But I don't see how the counterfactual definition requires us to vary the parents - if we're intervening at $V_j$, it shouldn't affect its parents, right?  Just how the values of those parents affect $V_j$.  There must be something I'm missing in this definition. (I also don't think it helps that you use capital $V$ for the variable of interest but lowercase $v$ for its parents)\n\nSection 2.4 has some symbology/variables that don't seem well-defined.  I think line 176 is the only place the symbol $\\mapsto$ appears (unless I'm missing something).  What does it mean here? Also, $k$ suddenly appears here as an index to $g$ and $x$, and we see that it ranges from $1$ to $p$.  Given how important this section is, a little more clarity in all of these indices would help.\n\nAlgorithm 1 seems strange and unnecessary.  First, on a bit of a side note, line 195 says that Algorithm 1 describes \"the construction of an ECM\", which seems like an odd title.  Up until now, the ECM has been discussed as the causal model provided by the user, and all algorithm 1 seems to be doing is adding edges from every variable in the ECM to the outcome, so I don't really see how that can be called \"constructing the ECM.\"  More importantly, though, Algorithm 1 seems essentially just like a pre-processing step.  You could just as easily tell the user to provide an ECM describing the causal structure among the input variables that has all of them causing outcome. (which would probably be the default assumption anyway, hence why they're being included as predictors) Given how tight space seems to be, unless there's a nuance I'm missing, Algorithm 1 could just be replaced with the sentence \"First, we add edges from all variables in the ECM to outcome.\"\n\nLine 196 introduces the notation of $\\hat{f}(P^M)$.  However, this notation was used for the first time on line 191 - introduce this notation before you use it.\n\nIn Algorithm 2, the second line is \"Get the possible values of $X_S$ and set it to $X$\".  I think you mean the opposite (i.e., \"Set $X$ to all the possible values of $X_S$), since I'm not otherwise sure what it would mean to set all the possible values of $X_S$ to $X$...\n\nI appreciate the discussion in 2.7 about uncertainty of the ECM structure, but, as with the discussion around Figure 1, I think it stops just short of providing a strong enough example.  If I were uncertain about the structure and produced the plots in Figure 3, what should be my takeaway besides \"the shapes are different for each model\"?  I did look at the example in Appendix B.2, hoping that the added space would allow for this type of discussion, but the only written takeaway for Figure 8 is \"cell shape impacting tumor class is indeed sensitive to our choice about the uncertain edge, particularly for the TDP.\"  The difference does appear very large for TDP, so what does that mean?  Can I use these results in any way to figure out which MEC member I should use?  Is the takeaway \"when reporting these results, I should report values for all three models?\"\n\nIn line 273, the authors conclude that, since ALE and SHAP appear similar to the PDP, \"our TDPs represent a significant and novel contribution to the existing model visualizations.\"  I don't necessarily disagree that the authors' TDPs are useful, but I just don't see how that follows from the previous statement.\n\nThe authors appear to be very pressed for space, relegating a lot of content to the supplementary material.  However, while some of this is reasonable and understandable, there are places where this is either confusing of feels clunky.\n- The authors describe their work as a generalization of PDPs.  However, the authors never actually define PDPs in the body of the paper, leaving that for the appendix.\n- Some of the discussion on page 7 seems overly dependent on the supplementary material, to the point where it's not particularly clear or useful without it.  This especially stood out in lines 224 and 225, where we're asked to compare Definition 2.8 to a definition in A.2, which shows that it's equivalent to the PDP (which is defined in A.1), the proof of which is in A.2.  Rather than having this content awkwardly split between the main paper and the appendix, I'd rather have this stuff referenced at a higher level with a pointer to the appendix, freeing up some space in the main paper for more detail on other parts that need it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your insightful and thorough review. You’ve given us useful feedback that will help clarify and improve the paper. You’re right that we were pressed for space- the CDP framework is quite general and there are many things we would like to demonstrate with it (e.g. other interesting special cases of PCDPs for common DAG structures, more about uncertainty regions, applications to fairness, etc). Accepted papers will have one additional page in the main text, and we can recover some space by removing Algorithm 1 as you suggest. Using that space and this response, we will do our best to answer all of your questions. We hope that you end up liking the paper as much as you wanted to!\n\nOn Figure 1 and the introduction: \n\n- The light gray points show the explanatory dataset (the same dataset used by the comparison methods)\n- **We will add a pseudo-algorithm description of PDPs in the introduction**. This will help serve as a contrast for our proposal, making it clear that the difference is PDPs holding other variables constant. While the NDDP is not defined until later, this should still be enough to drive home the first main point that TDP and PDP are fundamentally different.\n- We will describe how the takeaways are reached from the plot with sentences like the following. On qualitative difference: \"Across multiple black-box models the TDP shows an overall stronger increasing total dependence compared to direct dependence. And the sign can even be flipped, as in the true DGP and the linear model where the direct dependence is negative while the total dependence is mostly positive.\" On the third point: \"The direct dependence of $\\hat S$ on $P$ is increasing while the direct dependence of $S$ on $P$ in the true DGP is decreasing. In this case studying the black-box would not necessarily help us learn about the true DGP.\"\n\nFor Definition 2.3 **we will clarify that we are not intervening on $V_j$, but describing its counterfactual value under some intervention that modifies other variables** (some of which may be observed or exogenous parents of $V_j$). We need this level of abstraction for partially controlled effects, and that may make our definition less similar to cases where the application does not require partial control. \n\nIn Section 2.4 **we will remove the $\\mapsto$ symbol and better explain the indices**, describing the overall point in the text: \"All the other variables $x_k$ are uniquely determined by variable $x_j$ in this model.\"\n\nWe will fix the out-of-order notation usage, thanks for pointing it out.\n\nFor Algorithm 2 you are right and we will change the text (it depends on if we think of \"set\" as operating to the left or right, but leftward is certainly the most common)\n\nFor Section 2.7, the most important takeaway is just that uncertainty about an ECM can be represented visually in CDPs. There are various ways to model uncertainty, and CDPs are agnostic/modular regarding that choice. We demonstrate one method based on having two (or a set of) ECMs. **We will add more text to expand on the specific takeaway**, like: \"If we are not certain which of these two ECMs to use, this plot shows a region interpolating between both of their TDPs (or NIDPs). This example is not a confidence region, but any method for producing confidence sets in SCMs could also be used with CDPs to display uncertainty regions.\"\n\n> Is the takeaway \"when reporting these results, I should report values for all three models?\"\n\nWe think best practices in applications could involve things like pre-registering which results will be computed/reported, though this is somewhat outside of our scope. It will depend on publication norms and requirements that vary between journals and fields.\n\nOn the comparison with ALE and SHAP: Several **popular/SOTA xAI/interpretability tools produce qualitatively similar plots, while our TDP is the only one that stands out in showing a more sharply increasing relationship**. We want to conjecture that this generalizes: most of the xAI tools only show direct dependence. The theorem about PDPs is the only theoretical result we have been able to prove about this so far. Perhaps the general conjecture requires auxiliary hypotheses that vary between different application settings. A follow-up paper focusing on applications to fairness, for example, could add a condition that is likely to hold across many fairness settings, and then possibly prove (or show in experiments) that the other methods only show direct dependence (i.e. \"direct discrimination\").\n\nFinally, we reiterate that with the additional page for accepted papers we can move some supplemental material to the main text. **We will move the result showing PDP + ICE = NDDP to the main text**. Along with defining PDPs in the introduction, these changes will allow future readers to appreciate the main results and takeaways with less difficulty. \n\nThanks again for your time and work. We sincerely appreciate your help improving our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HB9bTIhPBX", "reviewer": "Reviewer_o4Ea", "summary": "The authors propose a new approach to visualize the impact of a change in a variable on the outcome of interest. They argue that if some of the other variables in the model are mediators, and that they should not be held constant as is currently done in variable importance measures, as it may lead to bias (post-treatment bias) on the measure of the effect on the outcome. They define estimands derived from the field of causal mediation analysis, and show how to use them in graphical representations that are easy to interpret and can also incorporate uncertainty.", "questions": "- Can you provide clearer guidance for the user in choosing an adequate model?\n- and for users to correctly verify the assumptions are met, especially if they are not familiar with causal inference and come from the field of explainable AI", "limitations": "the idea is good, but probably hard to use in practice, especially for users that are not familiar with causal inference. It also encourages users to think that they can obtain a valid causal interpretation from an additional step after fitting a machine learning model, instead of insisting that a valid causal conclusion is prepared by all steps of the analysis, including the study design, which is crucial.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- the authors have identified a flaw in the field of explainable ML and propose intuitive ways to remedy to this issue\n- the proposed graphical representations are very informative and useful\n- the authors propose to introduce more causal reasoning into the model explainability, while doing so with easy to interpret graphical representations. I have just updated my rating as I had misjudged the field of the paper, which made it a little weak from a causal inference perspective, but rather original and useful in the explainable AI field", "weaknesses": "- this work is not very theoretical, but it introduces a new way to look at a model results for explainability\n- however, it is a small fix on the problem of causally interpret machine learning models instead of doing a complete causal analysis. The PDP approach proposed by the authors require the same assumptions as a causal analysis: definition of the estimand of interest, verification of the identifiability assumptions, possibly through the construction of a causal graph, and estimation. From Figure 1, we can see that the results are sensitive to the choice of mode, and no guidance is provided on how to chose a suitable model. Overall the authors propose a fix for users that want to interpret a machine learning causally. However, if the model does not include a suitable set of adjustment variables, this approach will not work. Maybe a clearer message would be to to causal inference when one wants a causal interpretation of the model, instead of tinkering a non-working approach. However, the provided plots are interesting, and should be used in mediation analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your assessment. We are happy you found the soundness and contribution of our paper good and the presentation excellent. We agree that most explainable ML has a serious flaw and that an approach based on causal graphs and visualizations is intuitive, useful, and very informative.\n\nWe also agree about the importance of study design, particularly thinking about the choice of estimand, requirements for valid estimation and inference, and, ideally, doing this all before collecting data or fitting models. It is unfortunate that good statistical practices like these are too rarely followed. In the domain of NeurIPS (ML/AI), it is common for people to have datasets with plenty of black-box predictive models already fit to them, and only after achieving SOTA validation accuracy do they start to question why and how the model \"works.\" It is also common for people to use other popular model explanation / feature attribution tools, like SHAP values, and (incorrectly) interpret the results causally. **That is the status quo, that is what our work is trying to improve, and we believe that providing tools like CDPs will point such users in a better direction**.\n\nAfter our points of agreement there are a few issues we must push back on. \n\nFirstly, it is outside the scope of our one paper to provide general guidance on choosing a good causal model. That is the subject of much other work, and it will take a lot of time and education reforms for good practices from this field to make their way into common use. Again, we believe that CDPs will help in these efforts by pointing users who wish to explain/interpret an ML/AI model in the direction of causality. Causality is a large topic, and such users will have to inform themselves by reading other papers and books. **We are very clear in the paper that the limitations of identifying a good causal model apply to using CDPs**. This is responsible, this cannot be left to other work, and we have done it in Figure 1 and in the discussion. In the revision **we will add more references to resources in the causal literature that can provide general guidance to readers**.\n\nSecondly, we disagree that CDPs \"[encourage] users to think that they can obtain a valid causal interpretation from an additional step after fitting a machine learning model.\" On the contrary, **we repeatedly state that interpreting the output of a black-box model is fundamentally different from causal inference about the real world outcome variable**. See the third takeaway point about Figure 1 (starting on line 75), the second paragraph of Section 3 (Experiments), and lines 297-98 in the discussion (Limitations). We also define ECMs (Definition 2.5) with a convention of representing the predicted outcome as a separate node in the ECM graph, distinct from the true outcome variable, and directly caused by each predictor input of the black-box model.\n\nLastly, and perhaps less importantly, the review overemphasizes the importance of mediation. The parts of our paper that focus on simple mediation examples (the introduction and Section 2.6) are included mainly to help understand the basic idea. But with Partially Controlled Dependence Plots (PCDPs) and the direct dependence of model output on all predictors, CDPs are distinct from and more general than typical mediation analysis. **We will emphasize this by including one more definition of a general CDP using an arbitrary family of interventions parameterized by the plot axis** (e.g. it can be an intervention on multiple predictors simultaneously, for example subtracting from one and adding to another, or more generally moving in a certain direction in the input space).\n\nYour review makes us think you are very knowledgeable about causal inference. We are curious to know what you think about the last line of our conclusion:\n\n> Interpretability provided the initial motivation for CDPs, but since plots are qualitative CDPs also open the door for future work on causal methodology that relaxes assumptions while maintaining visual validity.\n\nIn other words, we think the assumptions required for valid quantitative inference (e.g. a confidence interval) are more restrictive than those that will be necessary for some notion of good qualitative inference (e.g. something about the overall shape of the plot being correct). We think this is an exciting direction for more work in causal inference generally once CDPs have been established.\n\nThanks again for the feedback. Your assessment was overall quite positive, including the soundness, presentation, and contribution scores. Please let us know if you have other questions or suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RkPLWwFMY7", "reviewer": "Reviewer_M8fQ", "summary": "The authors present a novel set of attribution methods called Causal Dependence Plots (CDPs) that extend partial dependence plots (PDP) while respecting possible causal dependencies in the method's inputs. The approach aims to obtain truthful and reliable insights for black box model analyses. In detail, a given Explanatory Causal Model (ECM) is utilized to simulate the propagation of possible input intervention effects onto other causally related input variables. The authors claim that, by propagating causal effects for the inputs, a 'natural' configuration (according to the ECM) is presented to the model. This stands in contrast to existing attribution methods, which only vary features independently -- leading to out-of-distribution configurations that do not truthfully reflect the input domain of the model under consideration. The authors differentiate primarily between direct (or 'partial') dependence plots that only evaluates single input variation (corresponding to previous evaluation methods), and the total dependence, which consider changes in causally related inputs accordingly.\n\nEvaluations are performed on multiple setups of synthetic toy examples and a real-world causal protein data set. In both settings, incorporating causal relations yields a better understanding of the system's behavior than non-causal attribution methods.", "questions": "My questions mainly concern the mentioned weaknesses. I would, therefore, like the authors to comment on the following questions:\n\n(1) Regarding (A): Could the authors think of possible applications of their method with causal analysis beyond pure visualization? When computing CDPs, how could a practitioner or automated system learn to infer treatment effects and detect deviations from the expected causal effects?\n\n(2) Could the authors discuss possible relations of their work to [Heskes et al. 2020]?\n\n(3) The authors mention the particular analysis of \"causal descendants\" (l.88). However, inference and predictions can also be performed in the anti-causal direction (consider, for example, [Schölkopf et al., 2012]). How does the presented approach translate to this kind of setting?\n\n\n\n[Heskes et al. 2020], \"Causal Shapley Values: Exploiting Causal Knowledge to Explain Individual Predictions of Complex Models\"\n\n[Schölkopf et al., 2012] \"On Causal and Anticausal Learning\"", "limitations": "By proposing a causal explainability method, various questions regarding the applications and risks of such methods arise for real-world decisions being made based on the resulting visualizations. The authors thoroughly discuss possible societal and ethical impacts due to improper use of their method. Implications of providing incorrect causal knowledge and a sensitivity analyses of the method are provided.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "While the presented approach is seemingly simple, it has not yet been proposed to the best of my knowledge. Considering causal relations between input features when evaluating models is a clearly relevant and appealing idea that allows practitioners to consider either total or direct effects for their particular applications. The authors suggest applications for the important areas of evaluating fairness, distribution shifts, and theory testing.\n\nThe novel concepts are well presented and build upon each other. Additionally, the authors prove in Thm. 2.9, a particular form of CDP, NDDPs, are equal to PDPs with Individual Conditional Expectation (ICE).\n\nSeveral variants of CDPs are presented and compared to each other. Experiments clearly exhibit the qualitative differences to previous PDPs and other attribution methods, such as Shapley values. The authors consider the cases of incomplete causal knowledge and perform a sensitivity analysis of their method.", "weaknesses": "(A) My main concerns are regarding the significance of this work. The presented contributions concerning causal inference seem rather minor, as the paper is mainly about the visualization of already-known results. In my opinion and the context of this conference, possible considerations and/or analysis about learning or causal inference are lacking.\n\n\n(B) While the authors discuss possible important applications to fairness, out-of-distribution testing, or theory testing, none of the presented examples feature any of these topics.\n\n\n(C) The authors primarily compare to associational methods. While not touching on the topic of visualization, causal Shapley values [Heskes et al. 2020] exist and should be compared to.\n\n\nMinor remarks:\n* On page 2, NDDPs are mentioned without being referenced or defined before.\n* Sec 1. l.46 typo'containsing'\n\n\n\n[Heskes et al. 2020], \"Causal Shapley Values: Exploiting Causal Knowledge to Explain Individual Predictions of Complex Models\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. It motivates us to make some small but key additions that should improve the paper. We take the claimed weaknesses and questions seriously and will do our best now to respond to each.\n\nA: Our main goal is not to contribute novel methods for causal learning or inference, but to reformulate (visual) model explanations as a causal problem within a clear conceptual and formal framework. As a comparison, consider the SHAP paper, \"A Unified Approach to Interpreting Model Predictions\" (Lundberg and Lee, NeurIPS 2017). This paper was not trying to contribute novel results or methods for game theory, but to reformulate model explanations as a game theoretic problem. Can such reformulations be significant? Perhaps- the SHAP paper has now been cited over 24k times. We are not conjecturing that CDPs will have a similar number of citations, but **work (like ours) which reformulates an important task (model explanation visualizations) as a causal one (and therefore connects it to much of the existing literature on causality) could end up being significant**.\n\nQ1: Consider the example in Section 2.7 (uncertainty ribbon plots) along with the final sentence of the conclusion: \n\n> Interpretability provided the initial motivation for CDPs, but since plots are qualitative CDPs also open the door for future work on causal methodology that relaxes assumptions while maintaining visual validity.\n\nThese are some initial steps and **we hope to explore this direction more in future work**. Providing a novel estimation or inference procedure along with CDPs would require more theory and experiments, and possibly more discussion and related work. We think that the space constraints of one conference paper make it infeasible for us to include that exploration in the current paper.\n\nB: You are absolutely right. We can and will add examples that elaborate on these ideas, at least for the first two (which are more relevant to machine learning). Some of this may need to appear in the supplemental material, but if the paper is accepted we also have one additional page we can use to showcase an important motivating application like these. **We will write about how the fairness literature contains several causal formulations that may include direct and indirect discrimination, and that our different CDPs can be used to probe a model for such (un)fairness properties, while other methods only show direct discrimination**. Additionally, we will include an example where a model does not take a sensitive attribute as a direct input, but the ECM allows testing for that sensitive attribute’s influence on the prediction through its effect on the other inputs.\n\nC and Q2: We have found 3 papers which define some type of causal Shapley values, including the one you mention.\n\n- Causal Shapley (Heskes et al, 2020)\n- Asymmetric Shapley values (Frye et al, 2020)\n- do-Shapley (Jung et al, 2022)\n\nWe have work in progress on comparisons with these. Among them, the do-Shapley method is closest in spirit to our approach and would likely make the most natural comparison. Unfortunately, so far we have had difficulty using the code from that paper’s supplement, and the same is true for the Causal Shapley paper. The ASV (asymmetric) paper is the only one with code that we’ve gotten to work by now, so we can most likely include an experiment to compare with that. We will continue working on the others and hopefully be able to include them all, but cannot promise that. At minimum, **we will include more discussion about the Heskes et al paper as requested**, and the other two, as related work. While we are working hard to include more comparison with these works, it is also worth mentioning that their goals differ somewhat from ours. They do relatively simple and automatic feature attribution without any specific guiding or motivating question. A strength of CDPs is that they can be constructed with an arbitrary intervention, and that could be motivated by a specific model auditing or scientific question.\n\nQ3: There are two near connections between our work and anti-causal learning. The first is that, as in anti-causal learning, the case where a cause is predicted from its effects has special implications for CDPs. CDPs show the causal dependency of the model’s predictions on a given variable, and in this case the direction of causality appears reversed from the underlying true outcome. This is another example about why explaining a model and explaining the DGP can be fundamentally different questions. The second connection is that a type of semi-supervised learning could be useful in both settings. For CDPs, unlabeled data could possibly be used when constructing an ECM. **We will expand our discussion of these points in the paper**, either with the additional page after acceptance or in the supplement.\n\nYou pointed out that our proposal is simple, novel, appealing, relevant, well-presented, and clearly shown to be qualitatively different from the existing comparison methods. We thank you for this assessment, as well as your more critical feedback. Please let us know if you have any other questions, suggestions, or concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I33Q8XRMqK", "reviewer": "Reviewer_GtVe", "summary": "The paper introduces causal dependence plots to visualize how (black-box) model predictions are affected by changes in input variable distributions. The explanations are generated based on an underlying explanatory causal model and can be generated for different types of causal quantities such as direct or indirect effects.", "questions": "1. The noisy case is not explained very well. Are the exogenous parents estimated based on an estimated model for the functional relationship? \nThe demo.ipynb seems to rely on additive form assumptions, which should be highlighted and discussed in the paper.", "limitations": "See Weaknesses", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a new explanatory plotting feature, which leverages knowledge about the underlying causal model to identify different types of causal quantities. The method builds upon known results in the causal inference literature and nicely embeds these quantities in a general explanatory framework.\n\n2. All claims are well supported and the paper is mostly well structured. The visualizations make it easy to follow the ideas and results\n\n3. Leveraging causal structures for explanatory plots is a relevant topic and enables detailed insights and model explanations.", "weaknesses": "1. The authors mention the limitation of a fully specified ECM. As this not only requires complete knowledge of the DAG but also the functional form of the corresponding functions G, this is a strong assumption. I think the paper would benefit, if the authors discuss the case if these functions are estimated instead. This severely reduces applicability and clarity of the approach. \n\nMinor Comments:\n- L. 45: Define $\\hat{f}$ before \n- L. 46: typo in “containsing”\n- Please define abbreviations such as TDP (l. 68) and NDDP (l. 71) before mentioning them\n- Please clarify the legend of Figure 3. Mention which model corresponds to which ECM", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad you agree our method embeds the causal inference literature in a general framework for model explanations and that you found our paper relevant and well-supported. \n\nThank you for pointing out the minor comments and typos. We will be careful to address each of these in the revision.\n\nWe now respond to your question and the stated weakness together because they are linked. You are correct from your reading of the demo notebook that we use additive noise modeling assumptions. Our implementation also estimates the functions in the ECM using the explanatory dataset. In some examples we assume the (graph) structure of the ECM is known, but in the breast cancer data example we also learn the structure using the PC algorithm. So, **while you are correct that complete knowledge of the DAG and functions in an ECM is a strong assumption, we have actually already included work showing the cases where the functions are estimated and the structure learned from data**.\n\nThe reason we define CDPs to take the ECM as an input is to make them modular. This way we are not wedded to any specific structure learning algorithm or function estimation method. A user can choose from the literature an estimation method that best fits with their application and then apply that method while constructing their ECM. **You correctly pointed out that we are building on other known results in the causal inference literature, so this actually makes CDPs more broadly applicable**.\n\nWhile our repository with code and notebooks shows implementation details, for clarity we will add more text description in the paper (e.g. how we constructed the ECMs), and also emphasize this point that other implementations could make use of different, context-specific learning algorithms and estimation methods, and then construct CDPs using their differently-estimated ECMs.\n\nThanks again for your feedback and we hope this has answered your question. Let us know if you have any other concerns or suggestions on how we can improve our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are really grateful for these high quality reviews. We condense reviews/rebuttals below and invite reviewers to correct us if we changed the meaning or missed important points (full reviews and responses are of course available separately).

# Summary

## Reviewer 6pua

**Positives**: "I really appreciate what this paper is trying to do. [The problem] is only getting more important, and providing easy-to-use visualization tools can be invaluable [...]" and "The issue of univariate explanations and what to hold constant is described quite well, helping to lay a strong motivational basis for the authors' work. I also think the writing is generally clear." The paper is upfront about its most important limitation (requiring a causal model as an input) and also provides multiple ways to deal with it.

**Criticisms**: Some of the presentation is unclear. Several things are close to being helpful but need additional clarification. Some notation or definitions are confusing and/or presented out of order. Algorithm 1 is unnecessary. The paper is pressed for space and some important results have been put in the supplement but should be in the main text.

**Response**: With the additional page allowed for accepted papers we could move key material from the supplement to the main text. We will clarify the identified definitions/notation and expand some descriptions/explanations.

## Reviewer o4Ea

**Positives**: Identifies a flaw in ML explanations and proposes an intuitive solution. "[V]ery informative and useful" with high scores on soundness (3), presentation (4), and contribution (3).

**Criticisms**: The work is not very theoretical. It provides only a small fix that will encourage users to interpret a machine learning model with a possibly wrong causal conclusion instead of properly verifying the assumptions required for correct causal conclusions. There should be more guidance for users on how to choose a good causal model. 

**Response**: These criticisms are most appropriate when the motivating question is to make causal conclusions about the underlying data generating process. But CDPs are, firstly, a tool for explaining a predictive model. It is true that additional assumptions are required for an explanation of a predictive model to have any validity for the underlying true outcome, but that is only one specific application of model explanations. The CDP framework helps distinguish between these applications and makes it more clear that causal conclusions depend on assumptions that must be checked. We will add more references to other important causal literature that readers should be familiar with in order to use CDPs.

## Reviewer M8fQ

**Positives**: The appoach is simple, novel, appealing, relevant, well-presented, and clearly shown to be qualitatively different from the existing comparison methods. Evaluation experiments in multiple synthetic and real datasets show "incorporating causal relations yields a better understanding of the system's behavior [...]." Limitations are thoroughly discussed. Included sensitivity analysis for the case of incomplete causal knowledge.

**Criticisms**: The contribution for causal inference is minor, lacking analysis or guidance. Possible applications (e.g. to fairness) are mentioned but not demonstrated in any examples. There should be some discussion about an existing method for causal Shapley values and an existing paper about anti-causal learning.

**Response**: Our contribution is not targeted to causal inference but rather to explainable/interpretable ML/AI. As an example for comparison, the SHAP paper was not a novel contribution on game theory, but reformulated feature attribution as a game theoretic problem. We aim to do a similar thing with feature visualization. We will add more detailed explanation and/or examples for applications like fairness and distribution shift. We will include more discussion about recent work on causal Shapley values and the connections between our work and anti-causal learning. 

## Reviewer GtVe

**Positives**: New general framework for model explanation plots that leverages/builds on existing literature on causal inference. Claims are well-supported, ideas and results easy to follow, topic is relevant.

**Criticisms**: CDPs use a fully specified ECM which is a strong assumption. Paper should discuss the case if ECM functions are estimated.

**Response**: We have already shown examples where the functions are estimated and even when the structure is learned from data. In Section 2.7/Figure 3 and Section B.2/Figure 8 we have shown a way to visualize uncertainty coming from that step. And as the Reviewer correctly points out, since we provide a general connection/bridge to the literature on causal inference, users can adapt any application-specific method for causal estimation and inference to construct their ECM and visualize uncertainty regions.

# Our action items

Some specific concerns were raised which we're confident can be addressed with the minor revisions below. (Accepted papers get 1 additional page)

- Add defn/pseudo-code for PDP in intro
- Move PDP + ICE = NDDP to main text
- Condense Algorithm 1
- Small change of notation and definitions to emphasize that predictors could be a subset of ECM variables (an intervention could target something which is not a direct input to the model)
- Add most general CDP: any family of interventions parametrized by the plot axis
- More about fairness, distribution shift, causal SHAPs, connections with anti-causal learning
- More refs to other causal literature with guidance
- More detail about experiments (how we fit ECMs)
- Fix some typos

# Discussion

The reviews are of very high quality and fairly positive overall. Even those which were more critical or gave lower ratings still said strong positive things about the paper. Hopefully reviewers find we've engaged their writing substantively and answered their concerns. We look forward to a productive discussion

Meta-review: The consensus is full among the four reviewers, they awarded exactly the same rating at the first review.

The authors-reviewers engagement was high with 33 posts published on this manuscript.

The strenghts of the paper overwhelm the weaknesses.

I greatly appreciated the opinion from 6pua who raised several issues on how the paper is presented. In particular, a paper introducing a new visualization and explainability approach should give a clear picture of how the new technique must be applied, which kind of results can be expected and how these results can re interpreted and leveraged on. Another weakness is the limitation implied by the need to have a fully specified ECM. However, the rebuttal and subsequent discussion clarified on this aspect that was finally considered not that a strong limitation.

Amongst the strengths, I feel that the following are of some interest; i) the authors revealed a flaw in explainable ML while proposing a solution, ii)  the manuscript introduces more causal reasoning into the model explainability, while doing so they try to improve the interpretation of graphical representations. I share the same view as two reviewers that this manuscript is a little weak from a causal inference perspective, but quite original and useful in the explainable AI field. An additional strength of the paper is that it presents the novel concepts very well and also develops theoretical arguments (Theorem. 2.9),that are useful.

In conclusion, I think the paper deserves attention even if it could be significantly improved whether the authors can address and solve the issues raised by some of the reviewers. In particular, I think that to improve the quality of the paper the authors can refer to the excellent review from 6pua.
Decision: Accept (poster)