Counterfactual Explanations in Sequential Decision Making Under Uncertainty

1 Introduction . In recent years , there has been a rising interest on the potential of machine learning models to assist and enhance decision making in high-stakes applications such as justice , education and healthcare [ 1– 3 ] . In this context , machine learning models and algorithms have been used not only to predict the outcome of a decision making process from a set of observable features but also to find what would have to change in ( some of ) the features for the outcome of a specific process realization to change . For example , in loan decisions , a bank may use machine learning both to estimate the probability that a customer repays a loan and to find by how much a customer may need to reduce her credit card debt to increase the probability of repayment over a given threshold . Here , our focus is on the latter , which has been often referred to as counterfactual explanations . The rapidly growing literature on counterfactual explanations has predominantly focused on one-step decision making processes as the one described above [ 4–6 ] . In such settings , the probability that an outcome occurs is typically estimated using a supervised learning model and finding counterfactual explanations reduces to a search problem across the space of features and model predictions [ 7–11 ] . Moreover , it has been argued that , to obtain counterfactual explanations that are actionable and have the predicted effect on the outcome , one should favor causal models [ 12–14 ] . In this work , our goal is instead to find counterfactual explanations for decision making processes in which multiple , dependent actions are taken sequentially over time . In this setting , the ( final ) outcome of the process depends on the overall sequence of actions and the counterfactual explanation specifies an alternative sequence of actions differing in at most k actions from the observed sequence that could have led the process realization to a better outcome . For example , in medical treatment , assume a physician takes a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . sequence of actions to treat a patient but the patient ’ s prognosis does not improve , then a counterfactual explanation would help the physician understand how a small number of actions taken differently could have improved the patient ’ s prognosis . However , since there is ( typically ) uncertainty on the counterfactual dynamics of the environment , there may be a different counterfactual explanation that is optimal under each possible realization of the counterfactual dynamics . Moreover , since , in many realistic scenarios , a decision maker needs to decide among a small number of actions on the basis of a few observable covariates , which are often discretized into ( percentile ) ranges , in this work , we focus on decision making processes where the state and action spaces are discrete and low-dimensional . The work most closely related to ours , which lies within the area of machine learning for healthcare , has achieved early success at using machine learning to enhance sequential ( clinical ) decisions [ 15– 18 ] . However , rather than finding counterfactual explanations , this line of work has predominantly focused on using reinforcement learning to design better treatment policies . A notable exception is by Oberst and Sontag [ 18 ] , which introduces an off-policy evaluation procedure for highlighting specific realizations of a sequential decision making process where a policy trained using reinforcement learning would have led to a substantially different outcome . While our work builds upon their modeling framework , we do not focus on off-policy evaluation but , instead , on the generation of ( better ) alternative action sequences as a means of explanation for the process ’ s outcome . Therefore , we see their contributions as complementary to ours . More broadly , our work is not the first to use counterfactual reasoning in the context of sequential decision making in the machine learning literature [ 19 , 20 ] . However , previous work has used counterfactual reasoning either to learn optimal policies using limited observational data [ 19 ] or to explain the action choices of a reinforcement learning agent [ 20 ] . In contrast , our work uses a causal model of the environment to compute alternative action sequences , close to the observed one , which , in retrospect , could have improved the outcome of the decision making process . Refer to Appendix A for a discussion of further related work . Our contributions . We start by formally characterizing a sequence of ( discrete ) actions and ( discrete ) states using finite horizon Markov decision processes ( MDPs ) . Here , we model the transition probabilities between a pair of states , given an action , using the Gumbel-Max structural causal model [ 18 ] . This model has been shown to have a desirable counterfactual stability property and , given a sequence of actions and states , it allows us to reliably estimate the counterfactual outcome under an alternative sequence of actions . Building upon this characterization , we formally state the problem of finding the counterfactual explanation for an observed realization of a sequential decision making process as a constrained search problem over the set of alternative sequences of actions differing in at most k actions from the observed sequence . Then , we present a polynomial time algorithm based on dynamic programming that finds a counterfactual policy that is guaranteed to always provide the optimal counterfactual explanation on every possible realization of the counterfactual transition probability induced by the observed process realization . Finally , we validate our algorithm using both synthetic and real data from cognitive behavioral therapy and show that the counterfactual explanations can provide valuable insights to enhance sequential decision making under uncertainty1 . 2 Characterizing Sequential Decision Making using Causal Markov Decision Processes . Our starting point is the following stylized setting , which resembles a variety of real-world sequential decision making processes . At each time step t ∈ { 0 , . . . , T − 1 } , the decision making process is characterized by a state st ∈ S , where S is a space of n states , an action at ∈ A , where A is a space of m actions , and a reward R ( st , at ) ∈ R. Moreover , given a realization of a decision making process τ = { ( st , at ) } T−1t=0 , the outcome of the decision making process o ( τ ) = ∑ tR ( st , at ) is given by the sum of the rewards . Given the above setting , we characterize the relationship between actions , states and outcomes using finite horizon Markov decision processes ( MDPs ) . More specifically , we consider an MDP M = ( S , A , P , R , T ) , where S is the state space , A is the set of actions , P denotes the transition probability P ( St+1 = st+1 |St = st , At = at ) , R denotes the immediate reward R ( st , at ) , and T is the time horizon . While this characterization is helpful to make predictions about future states and design action policies [ 21 ] , it is not sufficient to make counterfactual predictions , e.g. , given a realization of a decision making process τ = { ( st , at ) } T−1t=0 , we can not know what would have 1Our code is accessible at https : //github.com/Networks-Learning/counterfactual-explanations-mdp . happened if , instead of taking action at at time t , we had taken action a′ 6= at . To be able to overcome this limitation , we will now augment the above characterization using a particular class of structural causal model ( SCM ) [ 22 , 23 ] with desirable properties . Let C be a structural causal model defined by the assignments St+1 = gS ( St , At , Ut ) and At = gA ( St , Vt ) , ( 1 ) where Ut and Vt are n- and m-dimensional independent noise variables , respectively , and gS and gA are two given functions , and let P C be the distribution entailed by C. Then , as argued by Buesing et al . [ 19 ] , we can always find a distribution for the noise variables and a function gS so that the transition probability of the MDP of interest is given by the following interventional distribution over the SCM C : P ( St+1 = st+1 |St = st , At = at ) = P C ; do ( At=at ) ( St+1 = st+1 |St = st ) ( 2 ) where do ( At = at ) denotes a ( hard ) intervention in which the second assignment in Eq . 1 is replaced by the value at . Under this view , given an observed realization of a decision making process τ = { ( st , at ) } T−1t=0 , we can compute the posterior distribution P C |St=st , St+1=st+1 , At=at ( Ut ) of each noise variable Ut . Building on the conditional density function of this posterior distribution , which we denote as f C |St=st , St+1=st+1 , At=at Ut ( u ) , we can define a ( non-stationary ) counterfactual transition probability Pτ , t ( St+1 = s ′ |St = s , At = a ) = P C |St=st , St+1=st+1 , At=at ; do ( At=a ) ( St+1 = s′ |St = s ) = ∫ Rn P C |St=st , St+1=st+1 , At=at ; do ( At=a ) ( St+1 = s ′ |St = s , Ut = u ) × fC |St=st , St+1=st+1 , At=at ; do ( At=a ) Ut ( u ) du ( a ) = ∫ Rn 1 [ s′ = gS ( s , a , u ) ] · fC |St=st , St+1=st+1 , At=atUt ( u ) du = EUt |St=st , St+1=st+1 , At=at [ 1 [ s ′ = gS ( s , a , Ut ) ] ] , ( 3 ) where , in ( a ) , we drop the do ( · ) because Ut andAt are independent in the modified SCM . Importantly , the above counterfactual transition probability allows us to make counterfactual predictions , e.g. , given that , at time t the state was st and , at time t + 1 , the process transitioned to state st+1 after taking action at , what would have been the probability of transitioning to state s′ after taking action a 6= at if the state had been s at time t. Refer to Figure 1 for a visual description of our model and the notion of counterfactual predictions . However , for state variables taking discrete values , the posterior distribution of the noise may be non-identifiable without further assumptions , as argued by Oberst and Sontag [ 18 ] . This is because there may be several noise distributions and functions gS which give interventional distributions consistent with the MDP ’ s transition probabilities but result in different counterfactual transition distributions . To avoid these non-identifiability issues , we follow Oberst and Sontag and restrict our attention to the class of Gumbel-Max SCMs , i.e. , St+1 = gS ( St , At , Ut ) : = argmax s∈S { logP ( St+1 = s |St , At ) + Ut , s } , ( 4 ) where Ut , s ∼ Gumbel ( 0 , 1 ) and P ( · |St , At ) is the transition probability of the MDP . More specifically , this class of SCMs has been shown to satisfy a desirable counterfactual stability property , which goes intuitively as follows . Assume that , at time t , the process transitioned from state st to state st+1 after taking action at . Then , in a counterfactual scenario , it is unlikely that , at time t , the process would transition from a state s to a state s′ other than st+1—the factual one—unless choosing an action a that decreases the relative chances of st+1 compared to the other states . More formally , for a model satisfying counterfactual stability , given τ = { ( st , at ) } T−1t=0 , for all s , s′ with s′ 6= st+1 such that P ( St+1 = st+1 |St = s , At = a ) P ( St+1 = st+1 |St = st , At = at ) ≥ P ( St+1 = s ′ |St = s , At = a ) P ( St+1 = s′ |St = st , At = at ) , it holds that Pτ , t ( St+1 = s′ |St = s , At = a ) = 0 . In practice , in addition to solving the non-identifiability issues , the use of Gumbel-Max SCMs allows for an efficient procedure to sample from the corresponding noise posterior distribution P C |St=st , St+1=st+1 , At=at ( Ut ) , described elsewhere [ 18 , 24 ] , and given a set of d samples from the noise posterior distribution , we can compute an unbiased finite sample Monte-Carlo estimator for the counterfactual transition probability , as defined in Eq . 3 , as follows : Pτ , t ( St+1 = s ′ |St = s , At = a ) ≈ 1 d ∑ j∈ [ d ] 1 [ s′ = gS ( s , a , uj ) ] ( 5 )

Summary: This paper proposes a novel technique to generate counterfactual explanations for Markov decision processes (MDPs). Technically, it first remodels the state transition of MDPs with a Gumbel-Max structural causal model. Then, it formalizes the counterfactual explanation generation as an optimization problem and proposes an algorithm to generate the explanations. With the experiments on both a synthetic and a real-world dataset, the paper demonstrates the effectiveness of the proposed technique in finetuning a given policy to collect more rewards, especially when the corresponding environments have uncertainties.