SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Stratis Tsirtsis
Max Planck Institute for Software Systems
Kaiserslautern, Germany
stsirtsis@mpi-sws.org
Manuel Gomez-Rodriguez
manuelgr@mpi-sws.org
Whenever a clinician reflects on the efficacy of a sequence of treatment decisions for a patient, they may try to identify critical time steps where, had they made different decisions, the patient's health would have improved. While recent methods at the intersection of causal inference and reinforcement learning promise to aid human experts, as the clinician above, to retrospectively analyze sequential decision making processes, they have focused on environments with finitely many discrete states. However, in many practical applications, the state of the environment is inherently continuous in nature. In this paper, we aim to fill this gap. We start by formally characterizing a sequence of discrete actions and continuous states using finite horizon Markov decision processes and a broad class of bijective structural causal models. Building upon this characterization, we formalize the problem of finding counterfactually optimal action sequences and show that, in general, we cannot expect to solve it in polynomial time. Then, we develop a search method based on the  $\mathbf{A}^*$  algorithm that, under a natural form of Lipschitz continuity of the environment's dynamics, is guaranteed to return the optimal solution to the problem. Experiments on real clinical data show that our method is very efficient in practice, and it has the potential to offer interesting insights for sequential decision making tasks.
Had the chess player moved the king one round later, would they have avoided losing the game? Had the physician administered antibiotics one day earlier, would the patient have recovered? The process of mentally simulating alternative worlds where events of the past play out differently than they did in reality is known as counterfactual reasoning [1]. Thoughts of this type are a common by-product of human decisions and they are tightly connected to the way we attribute causality and responsibility to events and others' actions [2]. The last decade has seen a rapid development of reinforcement learning agents, presenting (close to) human-level performance in a variety of sequential decision making tasks, such as gaming [3, 4], autonomous driving [5] and clinical decision support [6, 7]. In conjunction with the substantial progress made in the field of causal inference [8, 9], this has led to a growing interest in machine learning methods that employ elements of counterfactual reasoning to improve or to retrospectively analyze decisions in sequential settings [10-15].
In the context of reinforcement learning, sequential decision making is typically modeled using Markov Decision Processes (MDPs) [16]. Here, we consider MDPs with a finite horizon where each episode (i.e., each sequence of decisions) consists of a finite number of time steps. As an example, consider a clinician treating a patient in an intensive care unit (ICU). At each time step, the clinician observes the current state of the environment (e.g., the patient's vital signs) and they choose among a set of potential actions (e.g., standardized dosages of a drug). Consequently, the chosen action causes the environment to transition (stochastically) into a new state, and the clinician earns a reward (e.g.,
satisfaction inversely proportional to the patient's severity). The process repeats until the horizon is met and the goal of the clinician is to maximize the total reward.
In this work, our goal is to aid the retrospective analysis of individual episodes as the example above. For each episode, we aim to find an action sequence that differs slightly from the one taken in reality but, under the circumstances of that particular episode, would have led to a higher counterfactual reward. In our example above, assume that the patient's condition does not improve after a certain period of time. A counterfactually optimal action sequence could highlight to the clinician a small set of time steps in the treatment process where, had they administered different drug dosages, the patient's severity would have been lower. In turn, a manual inspection of those time steps could provide insights to the clinician about potential ways to improve their treatment policy.
To infer how a particular episode would have evolved under a different action sequence than the one taken in reality, one needs to represent the stochastic state transitions of the environment using a structural causal model (SCM) [8, 17]. This has been a key aspect of a line of work at the intersection of counterfactual reasoning and reinforcement learning, which has focused on methods to either design better policies using offline data [10, 12] or to retrospectively analyze individual episodes [11, 13]. Therein, the work most closely related to ours is by Tsirtsis et al. [13], which introduces a method to compute counterfactually optimal action sequences in MDPs with discrete states and actions using a Gumbel-Max SCM to model the environment dynamics [11]. However, in many practical applications, such as in critical care, the state of the environment is inherently continuous in nature [18]. In our work, we aim to fill this gap by designing a method to compute counterfactually optimal action sequences in MDPs with continuous states and discrete actions. Refer to Appendix A for a discussion of further related work and to Pearl [8] for an overview of the broad field of causality.
Our contributions. We start by formally characterizing sequential decision making processes with continuous states and discrete actions using finite horizon MDPs and a general class of bijective SCMs [19]. Notably, this class of SCMs includes multiple models introduced in the causal discovery literature [20-25]. Building on this characterization, we make the following contributions:
(i) We formalize the problem of finding a counterfactually optimal action sequence for a particular episode in environments with continuous states under the constraint that it differs from the observed action sequence in at most  $k$  actions.
(ii) We show that the above problem is NP-hard using a novel reduction from the classic partition problem [26]. This is in contrast with the computational complexity of the problem in environments with discrete states, which allows for polynomial time algorithms [13].
(iii) We develop a search method based on the  $A^{*}$  algorithm that, under a natural form of Lipschitz continuity of the environment's dynamics, is guaranteed to return the optimal solution to the problem upon termination.
Finally, we evaluate the performance and the qualitative insights of our method by performing a series of experiments using real patient data from critical care.
2 A causal model of sequential decision making processes
At each time step  $t \in [T - 1] \coloneqq \{0, 1, \dots, T - 1\}$ , where  $T$  is a time horizon, the decision making process is characterized by a  $D$ -dimensional vector state  $s_t \in S = \mathbb{R}^D$ , an action  $a_t \in \mathcal{A}$ , where  $\mathcal{A}$  is a finite set of  $N$  actions, and a reward  $R(\boldsymbol{s}_t, a_t) \in \mathbb{R}$  associated with each pair of states and actions. Moreover, given an episode of the decision making process,  $\tau = \{(s_t, a_t)\}_{t=0}^{T-1}$ , the process's outcome  $o(\tau) = \sum_{t} R(s_t, a_t)$  is given by the sum of the rewards. In the remainder, we will denote the elements of a vector  $s_t$  as  $s_{t,1}, \ldots, s_{t,D}$ .<sup>2</sup>
Further, we characterize the dynamics of the decision making process using the framework of structural causal models (SCMs). In general, an SCM is consisted of four parts: (i) a set of endogenous variables (ii) a set of exogenous (noise) variables (iii) a set of structural equations assigning values to the endogenous variables, and (iv) a set of prior distributions characterizing the exogenous variables [8]. In our setting, the endogenous variables of the SCM  $\mathcal{C}$  are the random variables representing the states  $S_0,\ldots ,S_{T - 1}$  and the actions  $A_0,\dots ,A_{T - 1}$ . The action  $A_{t}$  at time
$^{1}$ Our code is accessible at https://github.com/Networks-Learning/counterfactual-continuous-mdp.
Table 1 in Appendix B summarizes the notation used throughout the paper.
step  $t$  is chosen based on the observed state  $S_{t}$  and is given by a structural (policy) equation
$$
A _ {t} := g _ {A} \left(\boldsymbol {S} _ {t}, \boldsymbol {Z} _ {t}\right), \tag {1}
$$
where  $Z_{t} \in \mathcal{Z}$  is a vector-valued noise variable, to allow some level of stochasticity in the choice of the action, and its prior distribution  $P^{\mathcal{C}}(\mathbf{Z}_t)$  is characterized by a density function  $f_{\mathbf{Z}_t}^{\mathcal{C}}$ . Similarly, the state  $S_{t + 1}$  in the next time step is given by a structural (transition) equation
$$
\boldsymbol {S} _ {t + 1} := g _ {S} \left(\boldsymbol {S} _ {t}, A _ {t}, \boldsymbol {U} _ {t}\right), \tag {2}
$$
where  $\mathbf{U}_t \in \mathcal{U}$  is a vector-valued noise variable with its prior distribution  $P^{\mathcal{C}}(\mathbf{U}_t)$  having a density function  $f_{\mathbf{U}_t}^{\mathcal{C}}$ , and we refer to the function  $g_S$  as the transition mechanism. Note that, in Eq. 2, the noise variables  $\{\mathbf{U}_t\}_{t=0}^{T-1}$  are mutually independent and, keeping the sequence of actions fixed, they are the only source of stochasticity in the dynamics of the environment. In other words, a sampled sequence of noise values  $\{\mathbf{u}_t\}_{t=0}^{T-1}$  and a fixed sequence of actions  $\{a_t\}_{t=0}^{T-1}$  result into a single (deterministic) sequence of states  $\{\mathbf{s}_t\}_{t=0}^{T-1}$ . This implicitly assumes that the state transitions are stationary and there are no unobserved confounders. Figure 4 in Appendix B depicts the causal graph  $G$  corresponding to the SCM  $\mathcal{C}$  defined above.
The above representation of sequential decision making using an SCM  $\mathcal{C}$  is a more general reformulation of a Markov decision process, where a (stochastic) policy  $\pi(a \mid s)$  is entailed by Eq. 1, and the transition distribution (i.e., the conditional distribution of  $S_{t+1} \mid S_t, A_t$ ) is entailed by Eq. 2. Specifically, the conditional density function of  $S_{t+1} \mid S_t, A_t$  is given by
$$
\begin{array}{l} p ^ {\mathcal {C}} \left(\boldsymbol {S} _ {t + 1} = \boldsymbol {s} \mid \boldsymbol {S} _ {t} = \boldsymbol {s} _ {t}, A _ {t} = a _ {t}\right) = p ^ {\mathcal {C}; d o \left(A _ {t} = a _ {t}\right)} \left(\boldsymbol {S} _ {t + 1} = \boldsymbol {s} \mid \boldsymbol {S} _ {t} = \boldsymbol {s} _ {t}\right) \\ = \int_ {\boldsymbol {u} \in \mathcal {U}} \mathbb {1} [ \boldsymbol {s} = g _ {S} (\boldsymbol {s} _ {t}, a _ {t}, \boldsymbol {u}) ] \cdot f _ {\boldsymbol {U} _ {t}} ^ {\mathcal {C}} (\boldsymbol {u}) d \boldsymbol {u}, \tag {3} \\ \end{array}
$$
where  $do(A_{t} = a_{t})$  denotes a (hard) intervention on the variable  $A_{t}$ , whose value is set to  $a_{t}$ .<sup>3</sup> Here, the first equality holds because  $S_{t+1}$  and  $A_{t}$  are d-separated by  $S_{t}$  in the sub-graph obtained from  $G$  after removing all outgoing edges of  $A_{t}$ <sup>4</sup> and the second equality follows from Eq. 2.
Moreover, as argued elsewhere [11, 13], by using an SCM to represent sequential decision making, instead of a classic MDP, we can answer counterfactual questions. More specifically, assume that, at time step  $t$ , we observed the state  $S_{t} = s_{t}$ , we took action  $A_{t} = a_{t}$  and the next state was  $S_{t + 1} = s_{t + 1}$ . Retrospectively, we would like to know the probability that the state  $S_{t + 1}$  would have been  $s'$  if, at time step  $t$ , we had been in a state  $s$ , and we had taken an action  $a$ , (generally) different from  $s_{t}, a_{t}$ . Using the SCM  $C$ , we can characterize this by a counterfactual transition density function
$$
\begin{array}{l} p ^ {\mathcal {C}} \mid S _ {t + 1} = s _ {t + 1}, S _ {t} = s _ {t}, A _ {t} = a _ {t}; d o (A _ {t} = a) \left(S _ {t + 1} = s ^ {\prime} \mid S _ {t} = s\right) = \\ \int_ {\boldsymbol {u} \in \mathcal {U}} \mathbb {1} [ \boldsymbol {s} ^ {\prime} = g _ {S} (\boldsymbol {s}, a, \boldsymbol {u}) ] \cdot f _ {\boldsymbol {U} _ {t}} ^ {\mathcal {C} \mid \boldsymbol {S} _ {t + 1} = \boldsymbol {s} _ {t + 1}, \boldsymbol {S} _ {t} = \boldsymbol {s} _ {t}, A _ {t} = a _ {t}} (\boldsymbol {u}) d \boldsymbol {u}, \tag {4} \\ \end{array}
$$
where  $f_{\boldsymbol{U}_t}^{\mathcal{C}}|_{S_{t+1}} = \boldsymbol{s}_{t+1}, \boldsymbol{S}_t = \boldsymbol{s}_t, A_t = a_t$  is the posterior distribution of the noise variable  $\boldsymbol{U}_t$  with support such that  $\boldsymbol{s}_{t+1} = g_S(\boldsymbol{s}_t, a_t, \boldsymbol{u})$ .
In what follows, we will assume that the transition mechanism  $g_{S}$  is continuous with respect to its last argument and the SCM  $\mathcal{C}$  satisfies the following form of Lipschitz-continuity:
Definition 1. An SCM  $\mathcal{C}$  is Lipschitz-continuous iff the transition mechanism  $g_{S}$  and the reward  $R$  are Lipschitz-continuous with respect to their first argument, i.e., for each  $a\in \mathcal{A}$ ,  $\pmb {u}\in \mathcal{U}$ , there exists a Lipschitz constant  $K_{a,\pmb{u}}\in \mathbb{R}_{+}$  such that, for any  $\pmb {s},\pmb{s}^{\prime}\in \mathcal{S}$ ,  $\| g_S(\pmb {s},a,\pmb {u}) - g_S(\pmb {s}^\prime ,a,\pmb {u})\| \leq$ $K_{a,\pmb{u}}\| \pmb {s} - \pmb{s}^{\prime}\|$ , and, for each  $a\in \mathcal{A}$ , there exists a Lipschitz constant  $C_a\in \mathbb{R}_+$  such that, for any  $\pmb {s},\pmb{s}^{\prime}\in \mathcal{S}$ ,  $|R(s,a) - R(s^{\prime},a)|\leq C_{a}\| \pmb {s} - \pmb{s}^{\prime}\|$ . In both cases,  $\| \cdot \|$  denotes the Euclidean distance.
Note that, although they are not phrased in causal terms, similar Lipschitz continuity assumptions for the environment dynamics are common in prior work analyzing the theoretical guarantees of reinforcement learning algorithms [27-35]. Moreover, for practical applications (e.g., in healthcare), this is a relatively mild assumption to make. Consider two patients whose vitals  $s$  and  $s'$  are similar
In general, the  $do$  operator also allows for soft interventions (i.e., setting a probability distribution for  $A_{t}$ ).
This follows directly from the rules of do-calculus. For further details, refer to Chapter 3 of Pearl [8].
at a certain point in time, they receive the same treatment  $a$ , and every unobserved factor  $\pmb{u}$  that may affect their health is also the same. Intuitively, Definition 1 implies that their vitals will also evolve similarly in the immediate future, i.e., the values  $g_{S}(\pmb{s},a,\pmb{u})$  and  $g_{S}(\pmb{s}^{\prime},a,\pmb{u})$  will not differ dramatically. In this context, it is worth mentioning that, when the transition mechanism  $g_{S}$  is modeled by a neural network, it is possible to control its Lipschitz constant during training, and penalizing high values can be seen as a regularization method [36, 37].
Further, we will focus on bijective SCMs [19], a fairly broad class of SCMs, which subsumes multiple models studied in the causal discovery literature, such as additive noise models [20], post-nonlinear causal models [21], location-scale noise models [22] and more complex models with neural network components [23-25].
Definition 2. An SCM  $\mathcal{C}$  is bijective iff the transition mechanism  $g_{S}$  is bijective with respect to its last argument, i.e., there is a well-defined inverse function  $g_{S}^{-1}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathcal{U}$  such that, for every combination of  $\boldsymbol{s}_{t+1}, \boldsymbol{s}_t, a_t, \boldsymbol{u}_t$  with  $\boldsymbol{s}_{t+1} = g_{S}(\boldsymbol{s}_t, a_t, \boldsymbol{u}_t)$ , it holds that  $\boldsymbol{u}_t = g_{S}^{-1}(\boldsymbol{s}_t, a_t, \boldsymbol{s}_{t+1})$ .
Importantly, bijective SCMs allow for a more concise characterization of the counterfactual transition density given in Eq. 4. More specifically, after observing an event  $S_{t + 1} = s_{t + 1}, S_t = s_t, A_t = a_t$ , the value  $\pmb{u}_t$  of the noise variable  $U_{t}$  can only be such that  $\pmb{u}_t = g_S^{-1}(\pmb{s}_t, a_t, \pmb{s}_{t + 1})$ , i.e., the posterior distribution of  $U_{t}$  is a point mass and its density is given by
$$
f _ {\boldsymbol {U} _ {t}} ^ {\mathcal {C} \mid \boldsymbol {S} _ {t + 1} = \boldsymbol {s} _ {t + 1}, \boldsymbol {S} _ {t} = \boldsymbol {s} _ {t}, A _ {t} = a _ {t}} (\boldsymbol {u}) = \mathbb {1} [ \boldsymbol {u} = g _ {S} ^ {- 1} (\boldsymbol {s} _ {t}, a _ {t}, \boldsymbol {s} _ {t + 1}) ], \tag {5}
$$
where  $\mathbb{1}[\cdot ]$  denotes the indicator function. Then, for a given episode  $\tau$  of the decision making process, we have that the (non-stationary) counterfactual transition density is given by
$$
\begin{array}{l} p _ {\tau , t} \left(\boldsymbol {S} _ {t + 1} = \boldsymbol {s} ^ {\prime} \mid \boldsymbol {S} _ {t} = \boldsymbol {s}, A _ {t} = a\right) := p ^ {\mathcal {C} \mid \boldsymbol {S} _ {t + 1} = \boldsymbol {s} _ {t + 1}, \boldsymbol {S} _ {t} = \boldsymbol {s} _ {t}, A _ {t} = a _ {t}; d o (A _ {t} = a)} \left(\boldsymbol {S} _ {t + 1} = \boldsymbol {s} ^ {\prime} \mid \boldsymbol {S} _ {t} = \boldsymbol {s}\right) \\ = \int_ {\boldsymbol {u} \in \mathcal {U}} \mathbb {1} [ \boldsymbol {s} ^ {\prime} = g _ {S} (\boldsymbol {s}, a, \boldsymbol {u}) ] \cdot \mathbb {1} [ \boldsymbol {u} = g _ {S} ^ {- 1} (\boldsymbol {s} _ {t}, a _ {t}, \boldsymbol {s} _ {t + 1}) ] d \boldsymbol {u} \\ = \mathbb {1} \left[ \boldsymbol {s} ^ {\prime} = g _ {S} \left(\boldsymbol {s}, a, g _ {S} ^ {- 1} \left(\boldsymbol {s} _ {t}, a _ {t}, \boldsymbol {s} _ {t + 1}\right)\right) \right]. \tag {6} \\ \end{array}
$$
Since this density is also a point mass, the resulting counterfactual dynamics are purely deterministic. That means, under a bijective SCM, the answer to the question "What would have been the state at time  $t + 1$ , had we been at state  $s$  and taken action  $a$  at time  $t$ , given that, in reality, we were at  $s_t$ , we took  $a_t$  and the environment transitioned to  $s_{t + 1}$ ?" is just given by  $s' = g_S(s, a, g_S^{-1}(s_t, a_t, s_{t + 1}))$ .
On the counterfactual identifiability of bijective SCMs. Very recently, Nasr-Esfahany and Kiciman [38] have shown that bijective SCMs are in general not counterfactually identifiable when the exogenous variable  $U_{t}$  is multi-dimensional. In other words, even with access to an infinite amount of triplets  $(s_{t}, a_{t}, s_{t+1})$  sampled from the true SCM  $\mathcal{C}$ , it is always possible to find an SCM  $\mathcal{M} \neq \mathcal{C}$  with transition mechanism  $h_{S}$  and distributions  $P^{\mathcal{M}}(U_{t})$  that entails the same transition distributions as  $\mathcal{C}$  (i.e., it fits the observational data perfectly), but leads to different counterfactual predictions. Although our subsequent algorithmic results do not require the SCM  $\mathcal{C}$  to be counterfactually identifiable, the subclass of bijective SCMs we will use in our experiments in Section 5 is counterfactually identifiable. The defining attribute of this subclass, which we refer to as element-wise bijective SCMs, is that the transition mechanism  $g_{S}$  can be decoupled into  $D$  independent mechanisms  $g_{S,i}$  such that  $S_{t+1,i} = g_{S,i}(S_{t}, A_{t}, U_{t,i})$  for  $i \in \{1, \ldots, D\}$ . This implies  $S_{t+1,i} \perp U_{t,j} | U_{t,i}, S_{t}, A_{t}$  for  $j \neq i$ , however,  $U_{t,i}, U_{t,j}$  do not need to be independent. Informally, we have the following identifiability result (refer to Appendix C for a formal version of the theorem along with its proof, which follows a similar reasoning to proofs found in related work [12, 19]):
Theorem 3 (Informal). Let  $\mathcal{C}$  and  $\mathcal{M}$  be two element-wise bijective SCMs such that their entailed transition distributions for  $S_{t + 1}$  given any value of  $S_{t}, A_{t}$  are always identical. Then, all their counterfactual predictions based on an observed transition  $(s_t, a_t, s_{t + 1})$  will also be identical.
On the assumption of no unobserved confounding. The assumption that there are no hidden confounders is a frequent assumption made by work at the intersection of counterfactual reasoning and reinforcement learning [10-13] and, more broadly, in the causal inference literature [39-43]. That said, there is growing interest in developing off-policy methods for partially observable MPDs (POMDPs) that are robust to certain types of confounding [44-46], and in learning dynamic treatment regimes in sequential settings with non-Markovian structure [47, 48]. Moreover, there is a line of work focusing on the identification of counterfactual quantities in non-sequential confounded environments [49-51]. In that context, we consider the computation of (approximately) optimal counterfactual action sequences under confounding as a very interesting direction for future work.
3 Problem statement
Let  $\tau$  be an observed episode of a decision making process whose dynamics are characterized by a Lipschitz-continuous bijective SCM. To characterize the counterfactual outcome that any alternative action sequence would have achieved under the circumstances of the particular episode, we build upon the formulation of Section 2, and we define a non-stationary counterfactual MDP  $\mathcal{M}^{+} = (\mathcal{S}^{+},\mathcal{A},F_{\tau ,t}^{+},R^{+},T)$  with deterministic transitions. Here,  $\mathcal{S}^{+} = \mathcal{S}\times [T - 1]$  is an enhanced state space such that each  $s^+ \in S^+$  is a pair  $(s,l)$  indicating that the counterfactual episode would have been at state  $s\in S$  with  $l$  action changes already performed. Accordingly,  $R^{+}$  is a reward function which takes the form  $R^{+}((s,l),a) = R(s,a)$  for all  $(s,l)\in S^{+}$ ,  $a\in \mathcal{A}$ , i.e., it does not change depending on the number of action changes already performed. Finally, the time-dependent transition function  $F_{\tau ,t}^{+}:S^{+}\times \mathcal{A}\to S^{+}$  is defined as
$$
F _ {\tau , t} ^ {+} \left(\left(\boldsymbol {s}, l\right), a\right) = \left\{ \begin{array}{l l} \left(g _ {S} \left(\boldsymbol {s}, a, g _ {S} ^ {- 1} \left(\boldsymbol {s} _ {t}, a _ {t}, \boldsymbol {s} _ {t + 1}\right)\right), l + 1\right) & \text {i f} (a \neq a _ {t}) \\ \left(g _ {S} \left(\boldsymbol {s}, a _ {t}, g _ {S} ^ {- 1} \left(\boldsymbol {s} _ {t}, a _ {t}, \boldsymbol {s} _ {t + 1}\right)\right), l\right) & \text {o t h e r w i s e .} \end{array} \right. \tag {7}
$$
Intuitively, here we set the transition function according to the point mass of the counterfactual transition density given in Eq. 6, and we use the second coordinate to keep track of the changes that have been performed in comparison to the observed action sequence up to the time step  $t$ .
Now, given the initial state  $s_0$  of the episode  $\tau$  and any counterfactual action sequence  $\{a_t'\}_{t=0}^{T-1}$ , we can compute the corresponding counterfactual episode  $\tau' = \{(s_t', l_t), a_t'\}_{t=0}^{T-1}$ . Its sequence of states is given recursively by
$$
\left(\boldsymbol {s} _ {1} ^ {\prime}, l _ {1}\right) = F _ {\tau , 0} ^ {+} \left(\left(\boldsymbol {s} _ {0}, 0\right), a _ {0} ^ {\prime}\right) \text {a n d} \left(\boldsymbol {s} _ {t + 1} ^ {\prime}, l _ {t + 1}\right) = F _ {\tau , 0} ^ {+} \left(\left(\boldsymbol {s} _ {t} ^ {\prime}, l _ {t}\right), a _ {t} ^ {\prime}\right) \text {f o r} t \in \{1, \dots , T - 1 \}, \tag {8}
$$
and its counterfactual outcome is given by  $o^{+}(\tau^{\prime})\coloneqq \sum_{t}R^{+}\left((s_{t}^{\prime},l_{t}),a_{t}^{\prime}\right) = \sum_{t}R\left(s_{t}^{\prime},a_{t}^{\prime}\right)$ .
Then, similarly as in Tsirtsis et al. [13], our ultimate goal is to find the counterfactual action sequence  $\{a_t^t\}_{t=0}^{T-1}$  that, starting from the observed initial state  $s_0$ , maximizes the counterfactual outcome subject to a constraint on the number of counterfactual actions that can differ from the observed ones, i.e.,