Counterfactual Explanations in Sequential Decision Making Under Uncertainty

1 Introduction . In recent years , there has been a rising interest on the potential of machine learning models to assist and enhance decision making in high-stakes applications such as justice , education and healthcare [ 1– 3 ] . In this context , machine learning models and algorithms have been used not only to predict the outcome of a decision making process from a set of observable features but also to find what would have to change in ( some of ) the features for the outcome of a specific process realization to change . For example , in loan decisions , a bank may use machine learning both to estimate the probability that a customer repays a loan and to find by how much a customer may need to reduce her credit card debt to increase the probability of repayment over a given threshold . Here , our focus is on the latter , which has been often referred to as counterfactual explanations . The rapidly growing literature on counterfactual explanations has predominantly focused on one-step decision making processes as the one described above [ 4–6 ] . In such settings , the probability that an outcome occurs is typically estimated using a supervised learning model and finding counterfactual explanations reduces to a search problem across the space of features and model predictions [ 7–11 ] . Moreover , it has been argued that , to obtain counterfactual explanations that are actionable and have the predicted effect on the outcome , one should favor causal models [ 12–14 ] . In this work , our goal is instead to find counterfactual explanations for decision making processes in which multiple , dependent actions are taken sequentially over time . In this setting , the ( final ) outcome of the process depends on the overall sequence of actions and the counterfactual explanation specifies an alternative sequence of actions differing in at most k actions from the observed sequence that could have led the process realization to a better outcome . For example , in medical treatment , assume a physician takes a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . sequence of actions to treat a patient but the patient ’ s prognosis does not improve , then a counterfactual explanation would help the physician understand how a small number of actions taken differently could have improved the patient ’ s prognosis . However , since there is ( typically ) uncertainty on the counterfactual dynamics of the environment , there may be a different counterfactual explanation that is optimal under each possible realization of the counterfactual dynamics . Moreover , since , in many realistic scenarios , a decision maker needs to decide among a small number of actions on the basis of a few observable covariates , which are often discretized into ( percentile ) ranges , in this work , we focus on decision making processes where the state and action spaces are discrete and low-dimensional . The work most closely related to ours , which lies within the area of machine learning for healthcare , has achieved early success at using machine learning to enhance sequential ( clinical ) decisions [ 15– 18 ] . However , rather than finding counterfactual explanations , this line of work has predominantly focused on using reinforcement learning to design better treatment policies . A notable exception is by Oberst and Sontag [ 18 ] , which introduces an off-policy evaluation procedure for highlighting specific realizations of a sequential decision making process where a policy trained using reinforcement learning would have led to a substantially different outcome . While our work builds upon their modeling framework , we do not focus on off-policy evaluation but , instead , on the generation of ( better ) alternative action sequences as a means of explanation for the process ’ s outcome . Therefore , we see their contributions as complementary to ours . More broadly , our work is not the first to use counterfactual reasoning in the context of sequential decision making in the machine learning literature [ 19 , 20 ] . However , previous work has used counterfactual reasoning either to learn optimal policies using limited observational data [ 19 ] or to explain the action choices of a reinforcement learning agent [ 20 ] . In contrast , our work uses a causal model of the environment to compute alternative action sequences , close to the observed one , which , in retrospect , could have improved the outcome of the decision making process . Refer to Appendix A for a discussion of further related work . Our contributions . We start by formally characterizing a sequence of ( discrete ) actions and ( discrete ) states using finite horizon Markov decision processes ( MDPs ) . Here , we model the transition probabilities between a pair of states , given an action , using the Gumbel-Max structural causal model [ 18 ] . This model has been shown to have a desirable counterfactual stability property and , given a sequence of actions and states , it allows us to reliably estimate the counterfactual outcome under an alternative sequence of actions . Building upon this characterization , we formally state the problem of finding the counterfactual explanation for an observed realization of a sequential decision making process as a constrained search problem over the set of alternative sequences of actions differing in at most k actions from the observed sequence . Then , we present a polynomial time algorithm based on dynamic programming that finds a counterfactual policy that is guaranteed to always provide the optimal counterfactual explanation on every possible realization of the counterfactual transition probability induced by the observed process realization . Finally , we validate our algorithm using both synthetic and real data from cognitive behavioral therapy and show that the counterfactual explanations can provide valuable insights to enhance sequential decision making under uncertainty1 . 2 Characterizing Sequential Decision Making using Causal Markov Decision Processes . Our starting point is the following stylized setting , which resembles a variety of real-world sequential decision making processes . At each time step t ∈ { 0 , . . . , T − 1 } , the decision making process is characterized by a state st ∈ S , where S is a space of n states , an action at ∈ A , where A is a space of m actions , and a reward R ( st , at ) ∈ R. Moreover , given a realization of a decision making process τ = { ( st , at ) } T−1t=0 , the outcome of the decision making process o ( τ ) = ∑ tR ( st , at ) is given by the sum of the rewards . Given the above setting , we characterize the relationship between actions , states and outcomes using finite horizon Markov decision processes ( MDPs ) . More specifically , we consider an MDP M = ( S , A , P , R , T ) , where S is the state space , A is the set of actions , P denotes the transition probability P ( St+1 = st+1 |St = st , At = at ) , R denotes the immediate reward R ( st , at ) , and T is the time horizon . While this characterization is helpful to make predictions about future states and design action policies [ 21 ] , it is not sufficient to make counterfactual predictions , e.g. , given a realization of a decision making process τ = { ( st , at ) } T−1t=0 , we can not know what would have 1Our code is accessible at https : //github.com/Networks-Learning/counterfactual-explanations-mdp . happened if , instead of taking action at at time t , we had taken action a′ 6= at . To be able to overcome this limitation , we will now augment the above characterization using a particular class of structural causal model ( SCM ) [ 22 , 23 ] with desirable properties . Let C be a structural causal model defined by the assignments St+1 = gS ( St , At , Ut ) and At = gA ( St , Vt ) , ( 1 ) where Ut and Vt are n- and m-dimensional independent noise variables , respectively , and gS and gA are two given functions , and let P C be the distribution entailed by C. Then , as argued by Buesing et al . [ 19 ] , we can always find a distribution for the noise variables and a function gS so that the transition probability of the MDP of interest is given by the following interventional distribution over the SCM C : P ( St+1 = st+1 |St = st , At = at ) = P C ; do ( At=at ) ( St+1 = st+1 |St = st ) ( 2 ) where do ( At = at ) denotes a ( hard ) intervention in which the second assignment in Eq . 1 is replaced by the value at . Under this view , given an observed realization of a decision making process τ = { ( st , at ) } T−1t=0 , we can compute the posterior distribution P C |St=st , St+1=st+1 , At=at ( Ut ) of each noise variable Ut . Building on the conditional density function of this posterior distribution , which we denote as f C |St=st , St+1=st+1 , At=at Ut ( u ) , we can define a ( non-stationary ) counterfactual transition probability Pτ , t ( St+1 = s ′ |St = s , At = a ) = P C |St=st , St+1=st+1 , At=at ; do ( At=a ) ( St+1 = s′ |St = s ) = ∫ Rn P C |St=st , St+1=st+1 , At=at ; do ( At=a ) ( St+1 = s ′ |St = s , Ut = u ) × fC |St=st , St+1=st+1 , At=at ; do ( At=a ) Ut ( u ) du ( a ) = ∫ Rn 1 [ s′ = gS ( s , a , u ) ] · fC |St=st , St+1=st+1 , At=atUt ( u ) du = EUt |St=st , St+1=st+1 , At=at [ 1 [ s ′ = gS ( s , a , Ut ) ] ] , ( 3 ) where , in ( a ) , we drop the do ( · ) because Ut andAt are independent in the modified SCM . Importantly , the above counterfactual transition probability allows us to make counterfactual predictions , e.g. , given that , at time t the state was st and , at time t + 1 , the process transitioned to state st+1 after taking action at , what would have been the probability of transitioning to state s′ after taking action a 6= at if the state had been s at time t. Refer to Figure 1 for a visual description of our model and the notion of counterfactual predictions . However , for state variables taking discrete values , the posterior distribution of the noise may be non-identifiable without further assumptions , as argued by Oberst and Sontag [ 18 ] . This is because there may be several noise distributions and functions gS which give interventional distributions consistent with the MDP ’ s transition probabilities but result in different counterfactual transition distributions . To avoid these non-identifiability issues , we follow Oberst and Sontag and restrict our attention to the class of Gumbel-Max SCMs , i.e. , St+1 = gS ( St , At , Ut ) : = argmax s∈S { logP ( St+1 = s |St , At ) + Ut , s } , ( 4 ) where Ut , s ∼ Gumbel ( 0 , 1 ) and P ( · |St , At ) is the transition probability of the MDP . More specifically , this class of SCMs has been shown to satisfy a desirable counterfactual stability property , which goes intuitively as follows . Assume that , at time t , the process transitioned from state st to state st+1 after taking action at . Then , in a counterfactual scenario , it is unlikely that , at time t , the process would transition from a state s to a state s′ other than st+1—the factual one—unless choosing an action a that decreases the relative chances of st+1 compared to the other states . More formally , for a model satisfying counterfactual stability , given τ = { ( st , at ) } T−1t=0 , for all s , s′ with s′ 6= st+1 such that P ( St+1 = st+1 |St = s , At = a ) P ( St+1 = st+1 |St = st , At = at ) ≥ P ( St+1 = s ′ |St = s , At = a ) P ( St+1 = s′ |St = st , At = at ) , it holds that Pτ , t ( St+1 = s′ |St = s , At = a ) = 0 . In practice , in addition to solving the non-identifiability issues , the use of Gumbel-Max SCMs allows for an efficient procedure to sample from the corresponding noise posterior distribution P C |St=st , St+1=st+1 , At=at ( Ut ) , described elsewhere [ 18 , 24 ] , and given a set of d samples from the noise posterior distribution , we can compute an unbiased finite sample Monte-Carlo estimator for the counterfactual transition probability , as defined in Eq . 3 , as follows : Pτ , t ( St+1 = s ′ |St = s , At = a ) ≈ 1 d ∑ j∈ [ d ] 1 [ s′ = gS ( s , a , uj ) ] ( 5 )

Summary: This paper is about counterfactual explanations for Markov decision processes. Instead of the usual counterfactual explanations as "alternative traces which can be used to explain that the chosen (optimal) choices are in fact optimal" as in explainable AI and reinforcement learning (RL), this paper uses them in offline settings, where data is available of the chosen paths, and where potentially better alternatives (counterfactual) can be computed. The main idea is to employ structural causal models to capture the model of the MDP such that better alternatives (here: action sequences that differ at most in k actions) can be computed in that model. The paper features experiments in two domains; one artifical and one in a small (in terms of the model) medical treatment domain. The experiments clearly show the benefits and characteristics of the approach.