Question: Is it necessary to require the ML model and the causal model agree on each other?
Evidence: Assume that we have an ML-model which implements the overall function h : R(I) → R(Y ), where I consists of all the input variables and Y is the single output variable. As mentioned, we also assume knowledge of a causal model M of the target domain (i.e., that part of the world which h is a model of). A first requirement is that such a causal model is consistent with the ML-model, meaning that they at least agree on how to classify all observations. A second requirement that is helpful given the conceptual nature of our analysis is that the endogenous variables I suffice to obtain deterministic causal knowledge of the target domain. Technically this means that there are only two kinds of endogenous variables: those which are determined directly by the unobserved exogenous variables, and those which are determined entirely by members of I. In future work both requirements can be loosened by adding probabilities to each. For the first, we can demand that both models are likely to agree on observations. For the second, we can add new exogenous variables and a probability distribution over them, which give us probabilistic causal models. These allow for probabilistic generalizations of all the definitions here presented.
either has a unique single exogenous parent U i ∈ U or no exogenous parents, and for all i ∈ R(I), y ∈ R(Y ):
As knowledge of the causal model is hard to come by many approaches to action-guiding explanations ignore it entirely and assume that the input variables are causally independent.
Definition 4 A causal model M that agrees with h satisfies Independence if for each I a , I b ∈ I, I a is not a parent of I b .
Obviously Independence rarely holds, and thus any account of action-guiding explanation that depends on it is limited. Although this limitation is recognized, this paper offers several results that formally characterize how extreme this limitation is. Simply put, under Independence a variety of very different causal notions become indistinguishable from each other. This holds for notions of sufficient explanation, counterfactual explanations, and various notions of actual causation.
Because of this limitation, work on action-guiding explanations in XAI has failed to take up the most relevant lesson that the literature on causation has to offer, namely that to give a causal explanation of an outcome is to give actual causes of that outcome. An additional reason for this oversight is that the precise relation between explanation and prediction has been the subject of much debate in the history of philosophy of science.
Prediction in its most natural application is a forward-looking notion, meaning one predicts an event before it takes place. Explanation on the other hand is a backward-looking notion, meaning that one explains an event after it has happened. Yet as many papers on XAI clearly illustrate, explanations about past events are often required precisely to inform predictions about future events. Therefore a suitable notion of causal explanation, and thus also of actual causation, needs to specify how it relates to predictions. Given the tumultuous history that this relation has in the philosophy literature, it has been duly neglected in the philosophical work on causation, thereby obscuring the importance of causation for the practical goals that XAI is concerned with. forms a notable exception!) 2 This paper corrects this by developing an account of causal explanation that shows both how it is connected to actual causation and how it can lead to action-guiding predictions.
Simply put, the goal of this paper is to upgrade the formalization of Woodward's influential philosophical account of causal explanation described below with the most recent insights from the causation literature, whilst also keeping track of the action-guiding demands that are prevalent in XAI.
Put differently, my idea is that one ought to be able to associate with any successful explanation a hypothetical or counterfactual experiment that shows us that and how manipulation of the factors mentioned in the explanation (the explanans, as philosophers call it) would be a way of manipulating or altering the phenomenon explained (the explanandum). Put in still another way, an explanation ought to be such that it can be used to answer what I call a what-if-things-had-been-different question: the explanation must enable us to see what sort of difference it would have made for the explanandum if the factors cited in the explanans had been different in various possible ways.
A crucial novel element of my account is the addition that a successful explanation must also be explicit about those factors that may not be manipulated for the explanation to hold, i.e., it must state which variables are to be safeguarded from interventions. Importantly, this is distinct from stating which variables must be held fixed at their actual values, for to hold variables fixed in fact means to intervene on them.
The following example (modified from) is helpful for illustrating the various purposes that explanations can serve.
Example 1 Consider a system for loan applications that is captured by a causal model such that
where Y is a binary variable representing whether the loan is granted, X 1 is the applicant's income, and X 2 is the applicant's savings. Further, assume that the applicant's savings are determined by their initial deposit X 3 and their income in the following manner: X 2 = 3/10 • X 1 + X 3 . It is also the case that people with high savings take out a safety deposit box (X 4 ) at the bank: X 4 = X 2 > 1, 000, 000.
Standard predictions in ML take place on the first rung of Pearl's causal ladder, namely that of observations. For example, an ML-model might pick up on the fact that all observed loan applicants who have a safety deposit box (X 4 = 1) also get a loan (Y = 1), and thus could predict that an applicant who has a safety deposit box will get a loan, i.e., it might learn a function so that h(x 1 , x 2 , x 3 , 1) = 1 for all values x 1 , x 2 , x 3 . In terms of the causal model, it is indeed the case that refer to as an anchor, which they interpret as a sufficient explanation of the outcome. Yet clearly such observational explanations are not a good guide towards action, for it would be absurd to recommend to someone to take out a safety deposit box so that their loan application is approved. This point can be brought out by making use of the second rung of the ladder, namely that of interventions: an applicant who takes out a safety deposit box as the result of being advised to do so will not necessarily get a loan:
In the same manner, interventions can be used to offer advice that is a good guide towards action, for example by telling an applicant that if they manage to obtain savings of 45, 001 then they are guaranteed to get a loan:
In future work both requirements can be loosened by adding probabilities to each. For the first, we can demand that both models are likely to agree on observations. For the second, we can add new exogenous variables and a probability distribution over them, which give us probabilistic causal models. These allow for probabilistic generalizations of all the definitions here presented
Answer: Regarding whether it is necessary that the ML-model and the causal model agree on all observations, note that the paper already addresses this to some extent, for it says: “In future work both requirements can be loosened by adding probabilities to each. For the first, we can demand that both models are likely to agree on observations.”

Of course this short comment leaves much to be determined. For example, if the causal model disagrees with the ML-model for some input-output pair, do we want to assume that at least one of them is correct? And if so, should it always be the same model? Or should we generalize such that both models come with probabilities which represent their uncertainty? The latter scenario would be most natural if both models were learnt entirely independently of each other, but that seems unrealistic (and inefficient). In fact, given that ML-methods are influential precisely because they are so good at learning associations, it seems likely that we would use the function h as a constraint when learning the solutions to the causal model in the absence of interventions, and then — per construction — both models would agree on all observations. I believe addressing these issues (and how to incorporate uncertainty more generally) merits a paper of its own. The reason is that otherwise one runs the risk of conflating conceptual and definitional issues with practical empirical issues.