SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, we investigate to what extent the counterfactual action sequences generated by our method would have led the patients in our dataset to better outcomes. For each patient, we measure their counterfactual improvement—the relative decrease in cumulative SOFA score between the counterfactual and the observed episode. Figures 3a and 3b summarize the results, which show that: (i) the average counterfactual improvement shows a diminishing increase as  $k$  increases; (ii) the median counterfactual improvement is only  $5\%$ , indicating that, the treatment choices made by the clinicians for most of the patients were close to optimal, even with the benefit of hindsight; and (iii) there are 176 patients for whom our method suggests that a different sequence of actions would have led to an outcome that is at least  $15\%$  better. That said, we view patients at the tail of the distribution as "interesting cases" that should be deferred to domain experts for closer inspection, and we present one such example in Fig. 3c. In this example, our method suggests that, had the patient received an early higher dosage of intravenous fluids while some of the later administered fluids where replaced by vasopressors, their SOFA score would have been lower across time. Although we present this case as purely anecdotal, the counterfactual episode is plausible, since there are indications of decreased mortality when intravenous fluids are administered at the early stages of a septic shock [61].
In this paper, we have introduced the problem of finding counterfactually optimal action sequences in sequential decision making processes with continuous state dynamics. We showed that the problem is NP-hard and, to tackle it, we introduced a search method based on the  $A^{*}$  algorithm that is guaranteed to find the optimal solution, with the caveat that its runtime can vary depending on the problem instance. Lastly, using real clinical data, we have found that our method is very efficient in practice, and it has the potential to offer interesting insights to domain experts by highlighting episodes and time-steps of interest for further inspection.
Our work opens up many interesting avenues for future work. For example, it would be interesting to develop algorithms with approximation guarantees that run in polynomial time, at the expense of not achieving strict counterfactual optimality. Moreover, since the practicality of methods like ours relies on the assumption that the SCM describing the environment is accurate, it would be interesting to develop methods to learn SCMs that align with human domain knowledge. Finally, it would be interesting to validate our method using real datasets from other applications and carry out user studies in which the counterfactual action sequences found by our method are systematically evaluated by the human experts (e.g., clinicians) who took the observed actions.
Acknowledgements. Tsirtsis and Gomez-Rodriguez acknowledge support from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 945719).
A Further related work
Counterfactual reasoning and reinforcement learning. As mentioned in Section 1, there is a closely related line of work [10-13] that focuses on the development of machine learning methods that employ elements of counterfactual reasoning to improve or to retrospectively analyze decisions in sequential settings. Buesing et al. [10] use SCMs to express the transition dynamics in Partially Observable MDPs (POMDPs), and they propose a method to efficiently compute a policy based on counterfactual realizations of logged episodes. Lu et al. [12] adopt a similar modeling framework, and they propose a counterfactual data augmentation approach to speed up standard Q-learning. Oberst and Sontag [11] introduce the Gumbel-Max SCM to express the dynamics of an arbitrary discrete POMPD, and they develop a method for counterfactual off-policy evaluation to identify episodes where a given alternative policy would have achieved a higher reward. However, none of these works aims to find an action sequence, close to the observed sequence of a particular episode, that is counterfactually optimal.
Planning in continuous-state MDPs. Our work has additional connections to pieces of work that aim to approximate the optimal value function in an MDP with continuous states and a finite horizon [27, 62, 63]. Therein, the work most closely related to ours is the one by Bertsekas [27]. It shows that, under a Lipschitz continuity assumption on the transition dynamics, a value function computed via value iteration in an MDP with discretized states, converges to the optimal value function of the original (continuous-state) MDP as the discretization becomes finer. Although some of the proof mechanics of this work are similar to ours, the contributions are orthogonal, as we do not employ any form of discretization, and we leverage the Lipschitz continuity assumption to compute optimal action sequences in continuous states using the  $A^{*}$  algorithm.
Counterfactual reasoning and explainability. Our work has ties to pieces of work that use forms of counterfactual reasoning as a tool towards learning explainable machine learning models. For example, Madumal et al. [64] propose to express the action selection process of a reinforcement learning agent as a causal graph, and they use it to generate explanations for the agent's chosen actions. Bica et al. [65] introduce an inverse reinforcement learning approach to learn an interpretable reward function from expert demonstrated behavior that is expressed in terms of preferences over potential (counterfactual) outcomes. Moreover, our work is broadly related to the work on counterfactual explanations (in classification) that aims to find a minimally different set of features that would have led to a different outcome, in settings where single decisions are taken [66, 67].
B Supporting table and figure
Table 1: It summarizes the most important notation used in the main body of the paper.
Symbol/Definition	Description
T	Time horizon
St∈S,At∈A	State and action (random variables) at time t
st,at	State and action (observed values) at time t
R: S × A → R	Reward function
τ = {(st, at)}T-1t=0	Observed episode
o(τ) = ∑t=0T-1R(st, at)	Outcome of episode τ
Ut ∈ U	Transition noise (random variable) at time t
ut	Transition noise (inferred value) at time t
gs: S × A × U → S	Transition mechanism (def. in Eq. 2)
Ka,u	Lipschitz constant of gs under At = a and Ut = u
Ca	Lipschitz constant of R under At = a
S+ = S × [T-1]	Enhanced state space
(s,l) ∈ S+	Enhanced state representing a c/f state s after l action changes
Fτ,t+: S+ × A → S+	Time-dependent transition function (def. in Eq. 7)
τ' = {(st', at')}T-1t=0	Counterfactual episode
o+(τ') = ∑t=0T-1R(st', at')	Counterfactual outcome of episode τ'
Vτ(s,l,t)	Max. c/f reward achievable for τ from enhanced state (s,l) at time t
(sa,la) = Fτ,t+((s,l),a)	Enhanced state resulting from (s,l) with action a
v = (s,l,t)	Node in the search graph of the A* algorithm
v0 = (s,0,0)	Root node
vT = (s0,k,T)	Goal node
rv	Reward accumulated along the path that A* followed from v0 to v
Lt	Lipschitz constant of Vτ(s,l,t)
ˆVτ(s,l,t)	Heuristic function approximating Vτ(s,l,t)
S†	Anchor set
Figure 4: Directed acyclic graph  $G$  of the SCM  $\mathcal{C}$ , modeling a sequential decision making process (figure adapted from [13]). Green boxes represent endogenous random variables and red boxes represent exogenous noise variables. All exogenous random variables are root nodes of the graph, and each one is independently sampled from its respective prior distribution. Each endogenous variable is an effect of its ancestors in the graph  $G$ , and takes its value based on Eqs. 1 and 2. An intervention  $do(A_{t} = a)$  breaks the dependence of the variable  $A_{t}$  from its ancestors (highlighted by dotted lines) and sets its value to  $a$ . After observing an event  $S_{t + 1} = s_{t + 1}, S_{t} = s_{t}, A_{t} = a_{t}$ , a counterfactual prediction corresponds to an intervention  $do(A_{t} = a)$  in a modified SCM where  $U_{t}$  takes values  $u_{t}$  from a posterior distribution with support such that  $s_{t + 1} = g_{S}(s_{t}, a_{t}, u_{t})$ .
C Counterfactual identifiability of element-wise bijective SCMs
In this section, we show that element-wise bijective SCMs, a subclass of bijective SCMs which we formally define next, are counterfactually identifiable.
Definition 8. An SCM  $\mathcal{C}$  is element-wise bijective iff it is bijective and there exist functions  $g_{S,i}:\mathbb{R}\times \mathcal{A}\times \mathbb{R}\to \mathbb{R}$  with  $i\in \{1,\ldots ,D\}$  such that, for every combination of  $\pmb{s}_{t + 1},\pmb{s}_t,a_t,\pmb{u}_t$  with  $\pmb{s}_{t + 1} = g_S(\pmb {s}_t,a_t,\pmb {u}_t)$ , it holds that  $s_{t + 1,i} = g_{S,i}(s_t,a_t,u_{t,i})$  for  $i\in \{1,\dots ,D\}$ .
Under our assumption that the transition mechanism  $g_{S}$  is continuous with respect to its third argument, it is easy to see that, for any element-wise bijective SCM, the functions  $g_{S,i}$  are always strictly monotonic functions of the respective  $u_{t,i}$ . Based on this observation, we have the following theorem of counterfactual identifiability:
Theorem 9. Let  $\mathcal{C}$  and  $\mathcal{M}$  be two element-wise bijective SCMs with transition mechanisms  $g_{S}$  and  $h_S$ , respectively, and, for any observed transition  $(s_t, a_t, s_{t+1})$ , let  $\boldsymbol{u}_t = g_S^{-1}(\boldsymbol{s}_t, a_t, \boldsymbol{s}_{t+1})$  and  $\tilde{\boldsymbol{u}}_t = h_S^{-1}(\boldsymbol{s}_t, a_t, \boldsymbol{s}_{t+1})$ . Moreover, given any  $\boldsymbol{s} \in S$ ,  $a \in \mathcal{A}$ , let  $\boldsymbol{s}' = g_S(\boldsymbol{s}, a, \boldsymbol{u}_t)$  and  $\boldsymbol{s}'' = h_S(\boldsymbol{s}, a, \tilde{\boldsymbol{u}}_t)$ . If  $P^{\mathcal{C}}(\boldsymbol{S}_{t+1} \mid \boldsymbol{S}_t = \boldsymbol{s}, A_t = a) = P^{\mathcal{M}}(\boldsymbol{S}_{t+1} \mid \boldsymbol{S}_t = \boldsymbol{s}, A_t = a)$  for all  $\boldsymbol{s} \in S$ ,  $a \in \mathcal{A}$ , it must hold that  $\boldsymbol{s}' = \boldsymbol{s}''$ .
Proof. We prove the theorem by induction, starting by establishing the base case  $s_1' = s_1''$ . Without loss of generality, assume that both  $g_{S,1}$  and  $h_{S,1}$  are strictly increasing with respect to their third argument. Since the two SCMs entail the same transition distributions, we have that
$$
\begin{array}{l} P ^ {\mathcal {C}} \left(S _ {t + 1, 1} \leq s _ {t + 1, 1} \mid \boldsymbol {S} _ {t} = \boldsymbol {s} _ {t}, A _ {t} = a _ {t}\right) = P ^ {\mathcal {M}} \left(S _ {t + 1, 1} \leq s _ {t + 1, 1} \mid \boldsymbol {S} _ {t} = \boldsymbol {s} _ {t}, A _ {t} = a _ {t}\right) \overset {(*)} {\Rightarrow} \\ P ^ {\mathcal {C}} \left(g _ {S, 1} \left(\boldsymbol {s} _ {t}, a _ {t}, U _ {t, 1}\right) \leq g _ {S, 1} \left(\boldsymbol {s} _ {t}, a _ {t}, u _ {t, 1}\right)\right) = P ^ {\mathcal {M}} \left(h _ {S, 1} \left(\boldsymbol {s} _ {t}, a _ {t}, U _ {t, 1}\right) \leq h _ {S, 1} \left(\boldsymbol {s} _ {t}, a _ {t}, \tilde {u} _ {t, 1}\right)\right) \stackrel {(*)} {\Rightarrow} \\ P ^ {\mathcal {C}} \left(U _ {t, 1} \leq u _ {t, 1}\right) = P ^ {\mathcal {M}} \left(U _ {t, 1} \leq \tilde {u} _ {t, 1}\right), \\ \end{array}
$$
where  $(\ast)$  holds because both SCMs are element-wise bijective, and  $(\ast\ast)$  holds because  $g_{S,1}$  and  $h_{S,1}$  are increasing with respect to their third argument. Similarly, we have that
$$
\begin{array}{l} P ^ {\mathcal {C}} \left(S _ {t + 1, 1} \leq s _ {1} ^ {\prime} \mid S _ {t} = s, A _ {t} = a\right) = P ^ {\mathcal {C}} \left(g _ {S, 1} (s, a, U _ {t, 1}) \leq g _ {S, 1} (s, a, u _ {t, 1})\right) \\ \stackrel {(\star)} {=} P ^ {\mathcal {C}} \left(U _ {t, 1} \leq u _ {t, 1}\right) \\ \stackrel {(\star \star)} {=} P ^ {\mathcal {M}} (U _ {t, 1} \leq \tilde {u} _ {t, 1}) \\ \stackrel {(\dagger)} {=} P ^ {\mathcal {M}} \left(h _ {S, 1} \left(\boldsymbol {s}, a, U _ {t, 1}\right) \leq h _ {S, 1} \left(\boldsymbol {s}, a, \tilde {u} _ {t, 1}\right)\right) \\ = P ^ {\mathcal {M}} \left(S _ {t + 1, 1} \leq s _ {1} ^ {\prime \prime} \mid \boldsymbol {S} _ {t} = \boldsymbol {s}, A _ {t} = a\right) \\ = P ^ {\mathcal {C}} \left(S _ {t + 1, 1} \leq s _ {1} ^ {\prime \prime} \mid S _ {t} = s, A _ {t} = a\right), \\ \end{array}
$$
where in  $(\star), (\dagger)$  we have used the monotonicity of  $g_{S}$  and  $h_{S}$ , and  $(\star \star)$  follows from the previous result. The last equality implies that  $s_1'$  and  $s_1''$  correspond to the same quantile of the distribution for  $S_{t+1,1} \mid S_t = s, A_t = a$ . Therefore, it is easy to see that  $s_1' = s_1''$  since the opposite would be in contradiction to  $g_{S,1}$  being bijective. Note that, we can reach that conclusion irrespective of the direction of monotonicity of  $g_{S,1}$  and  $h_{S,1}$ , since any change in the direction of the inequalities happening at step  $(\ast \ast)$  is reverted at steps  $(\star)$  and  $(\dagger)$ .
Now, starting from the inductive hypothesis that  $s_i' = s_i''$  for all  $i \in \{1, \dots, n\}$  with  $n < D$ , we show the inductive step, i.e.,  $s_{n+1}' = s_{n+1}''$ . Again, without loss of generality, assume that both  $g_{S,n+1}$  and  $h_{S,n+1}$  are strictly increasing with respect to their last argument. Note that, the two SCMs entail the same transition distributions, i.e., the same joint distributions for  $S_{t+1} \mid S_t, A_t$ . Following from the law of total probability, they also entail the same conditional distributions for  $S_{t+1,n+1} \mid S_{t+1, \leq n}, S_t, A_t$ , where we use the notation  $x_{\leq n}$  to refer to a vector that contains the first  $n$  elements of a  $D$ -dimensional vector  $x$ . Therefore, we have that
$$
\begin{array}{l} P ^ {\mathcal {C}} \left(S _ {t + 1, n + 1} \leq s _ {t + 1, n + 1} \mid \boldsymbol {S} _ {t + 1, \leq n} = \boldsymbol {s} _ {t + 1, \leq n}, \boldsymbol {S} _ {t} = \boldsymbol {s} _ {t}, A _ {t} = a _ {t}\right) = \\ P ^ {\mathcal {M}} \left(S _ {t + 1, n + 1} \leq s _ {t + 1, n + 1} \mid S _ {t + 1, \leq n} = s _ {t + 1, \leq n}, S _ {t} = s _ {t}, A _ {t} = a _ {t}\right) \stackrel {(*)} {\Rightarrow} \\ \end{array}
$$
$$
\begin{array}{l} P ^ {\mathcal {C}} \left(g _ {S, n + 1} \left(\boldsymbol {s} _ {t}, a _ {t}, U _ {t, n + 1}\right) \leq g _ {S, n + 1} \left(\boldsymbol {s} _ {t}, a _ {t}, u _ {t, n + 1}\right) \mid \boldsymbol {U} _ {t, \leq n} = \boldsymbol {u} _ {t, \leq n}\right) \\ = P ^ {\mathcal {M}} \left(h _ {S, n + 1} \left(\boldsymbol {s} _ {t}, a _ {t}, U _ {t, n + 1}\right) \leq h _ {S, n + 1} \left(\boldsymbol {s} _ {t}, a _ {t}, \tilde {\boldsymbol {u}} _ {t, n + 1}\right) \mid \boldsymbol {U} _ {t, \leq n} = \tilde {\boldsymbol {u}} _ {t, \leq n}\right) \stackrel {(*)} {\Rightarrow} \\ \end{array}
$$
$$
P ^ {\mathcal {C}} \left(U _ {t, n + 1} \leq u _ {t, n + 1} \mid \boldsymbol {U} _ {t, \leq n} = \boldsymbol {u} _ {t, \leq n}\right) = P ^ {\mathcal {M}} \left(U _ {t, n + 1} \leq \tilde {u} _ {t, n + 1} \mid \boldsymbol {U} _ {t, \leq n} = \tilde {\boldsymbol {u}} _ {t, \leq n}\right),
$$
where for the first equality we have used the inductive hypothesis,  $(\ast)$  holds because both SCMs are element-wise bijective, and  $(\ast\ast)$  holds because  $g_{S,n+1}$  and  $h_{S,n+1}$  are increasing with respect to their third argument. Similarly, we get that
$$
\begin{array}{l} P ^ {\mathcal {C}} \left(S _ {t + 1, n + 1} \leq s _ {n + 1} ^ {\prime} \mid S _ {t + 1, \leq n} = s _ {t + 1, \leq n}, S _ {t} = s, A _ {t} = a\right) \\ = P ^ {C} \left(g _ {S, n + 1} (\boldsymbol {s}, a, U _ {t, n + 1}) \leq g _ {S, n + 1} (\boldsymbol {s}, a, u _ {t, n + 1}) \mid g _ {S, \leq n} (\boldsymbol {s}, a, U _ {t, \leq n}) = g _ {s, \leq n} (\boldsymbol {s}, a, \boldsymbol {u} _ {t, \leq n})\right) \\ \stackrel {(\star)} {=} P ^ {\mathcal {C}} \left(U _ {t, n + 1} \leq u _ {t, n + 1} \mid U _ {t, \leq n} = \boldsymbol {u} _ {t, \leq n}\right) \\ \stackrel {(\star \star)} {=} P ^ {\mathcal {M}} \left(U _ {t, n + 1} \leq \tilde {u} _ {t, n + 1} \mid U _ {t, \leq n} = \tilde {\boldsymbol {u}} _ {t, \leq n}\right) \\ \stackrel {(\dagger)} {=} P ^ {\mathcal {M}} \left(h _ {S, n + 1} (\boldsymbol {s}, a, U _ {t, 1}) \leq h _ {S, n + 1} (\boldsymbol {s}, a, \tilde {u} _ {t, 1}) \mid h _ {S, \leq n} (\boldsymbol {s}, a, \boldsymbol {U} _ {t, \leq n}) = h _ {s, \leq n} (\boldsymbol {s}, a, \tilde {\boldsymbol {u}} _ {t, \leq n})\right) \\ = P ^ {\mathcal {M}} \left(S _ {t + 1, n + 1} \leq s _ {n + 1} ^ {\prime \prime} \mid \boldsymbol {S} _ {t + 1, \leq n} = \boldsymbol {S} _ {t + 1, \leq n}, \boldsymbol {S} _ {t} = \boldsymbol {s}, A _ {t} = a\right) \\ = P ^ {\mathcal {C}} \left(S _ {t + 1, n + 1} \leq s _ {n + 1} ^ {\prime \prime} \mid \boldsymbol {S} _ {t + 1, \leq n} = s _ {t + 1, \leq n}, \boldsymbol {S} _ {t} = s, A _ {t} = a\right), \\ \end{array}
$$
where in  $(\star), (\dagger)$  we have used the invertibility and monotonicity of  $g_{S}$  and  $h_{S}$ , and  $(\star \star)$  follows from the previous result. With the same argument as in the base case, the last equality implies that  $s_{n+1}' = s_{n+1}''$ . That concludes the proof.
We prove the hardness of our problem as defined in Eq. 9 by performing a reduction from the partition problem [26], which is known to be NP-Complete. In the partition problem, we are given a multiset of  $B$  positive integers  $\mathcal{V} = \{v_{1},\ldots ,v_{B}\}$  and the goal is to decide whether there is a partition of  $\mathcal{V}$  into two subsets  $\mathcal{V}_1,\mathcal{V}_2$  with  $\mathcal{V}_1\cap \mathcal{V}_2 = \emptyset$  and  $\mathcal{V}_1\cup \mathcal{V}_2 = \mathcal{V}$ , such that their sums are equal, i.e.,  $\sum_{v_i\in \mathcal{V}_1}v_i = \sum_{v_j\in \mathcal{V}_2}v_j$ .
Consider an instance of our problem where  $\mathcal{S} = \mathcal{U} = \mathbb{R}^2$ ,  $\mathcal{A}$  contains 2 actions  $a_{\mathrm{diff}}$ ,  $a_{\mathrm{null}}$  and the horizon is  $T = B + 1$ . Let  $\mathcal{C}$  be an element-wise bijective SCM with arbitrary prior distributions  $P^{\mathcal{C}}(\pmb{U}_t)$  such that their support is on  $\mathbb{R}^2$  and a transition mechanism  $g_S$  such that
$$
g _ {S} \left(\boldsymbol {S} _ {t}, a _ {\text {d i f f}}, \boldsymbol {U} _ {t}\right) = \left[ \begin{array}{c} S _ {t, 1} - S _ {t, 2} \\ 0 \end{array} \right] + \boldsymbol {U} _ {t} \quad \text {a n d} \quad g _ {S} \left(\boldsymbol {S} _ {t}, a _ {\text {n u l l}}, \boldsymbol {U} _ {t}\right) = \left[ \begin{array}{c} S _ {t, 1} \\ 0 \end{array} \right] + \boldsymbol {U} _ {t}. \tag {12}
$$
Moreover, assume that the reward function is given by
$$
\begin{array}{l} R (\pmb {S} _ {t}, a _ {\mathrm {d i f f}}) = R (\pmb {S} _ {t}, a _ {\mathrm {n u l l}}) = - \max \left(0, S _ {t, 1} - \frac {s u m (\mathcal {V})}{2} - S _ {t, 2} \frac {s u m (\mathcal {V})}{2}\right) \\ - \max  \left(0, \frac {\operatorname {s u m} (\mathcal {V})}{2} - S _ {t, 1} - S _ {t, 2} \frac {\operatorname {s u m} (\mathcal {V})}{2}\right), \tag {13} \\ \end{array}
$$
where  $sum(\mathcal{V})$  is the sum of all elements  $\sum_{i=1}^{B} v_i$ . Note that, the SCM  $\mathcal{C}$  defined above is Lipschitz-continuous as suggested by the following Lemma (refer to Appendix D.1.1 for a proof).
Lemma 10. The SCM C defined by Equations 12, 13 is Lipschitz-continuous according to Definition 1.
Now, assume that the counterfactual action sequence can differ in an arbitrary number of actions from the action sequence in the observed episode  $\tau$ , i.e.,  $k = T$  and, let the observed action sequence be such that  $a_{t} = a_{\mathrm{null}}$  for  $t \in \{0, \dots, T - 1\}$ . Lastly, let the initial observed state be  $\boldsymbol{s}_0 = [0, v_1]$ , the observed states  $\{\boldsymbol{s}_t\}_{t=1}^{T-2}$  be such that  $\boldsymbol{s}_t = \left[\sum_{i=1}^t v_i, v_{t+1}\right]$  for  $t \in \{1, \dots, T-2\}$  and the last observed state be  $\boldsymbol{s}_{T-1} = [sum(\mathcal{V}), 0]$ . Then, it is easy to see that the noise variables  $\boldsymbol{U}_t$  have posterior distributions with a point mass on the respective values
$$
\boldsymbol {u} _ {t} = \left[ \begin{array}{c} v _ {t + 1} \\ v _ {t + 2} \end{array} \right] \text {f o r} t \in \{0, \ldots , T - 3 \} \qquad \text {a n d} \qquad \boldsymbol {u} _ {T - 2} = \left[ \begin{array}{c} v _ {T - 1} \\ 0 \end{array} \right].
$$
Note that, for all  $t \in \{1, \dots, T - 2\}$ , we have  $0 \leq s_{t,1} < \text{sum}(\mathcal{V})$  and  $s_{t,2} \geq 1$ , hence the immediate reward according to Eq. 13 is equal to 0. Consequently, the outcome of the observed episode  $\tau$  is  $o^{+}(\tau) = R(s_{T - 1}, a_{\mathrm{null}}) = -\max(0, \frac{\text{sum}(\mathcal{V})}{2}) - \max(0, -\frac{\text{sum}(\mathcal{V})}{2}) = -\frac{\text{sum}(\mathcal{V})}{2}$ .
Next, we will characterize the counterfactual outcome  $o(\tau')$  of a counterfactual episode  $\tau'$  with a sequence of states  $\{s_t'\}_{t=0}^{T-1}$  resulting from an alternative sequence of actions  $\{a_t'\}_{t=0}^{T-1}$ . Let  $\mathcal{D}_t', \mathcal{N}_t'$  denote the set of time steps until time  $t$ , where the actions taken in a counterfactual episode  $\tau'$  are  $a_{\mathrm{diff}}$  and  $a_{\mathrm{null}}$  respectively. Formally,  $\mathcal{D}_t' = \{t' \in \{0, \dots, t\} : a_{t'}' = a_{\mathrm{diff}}\}$ ,  $\mathcal{N}_t' = \{t' \in \{0, \dots, t\} : a_{t'}' = a_{\mathrm{null}}\}$ . Then, as an intermediate result, we get the following Lemma (refer to Appendix D.1.2 for a proof).
Lemma 11. It holds that  $s_{t,1}^{\prime} = \sum_{t^{\prime}\in \mathcal{N}_{t - 1}^{\prime}}v_{t^{\prime} + 1}$  for all  $t\in \{1,\ldots T - 1\}$ .
Following from that, we get that  $0 \leq s_{t,1}^{\prime} \leq \text{sum}(\mathcal{V})$  for all  $t \in \{1, \dots, T-1\}$ . Moreover, we can observe that the transition mechanism given in Eq. 12 is such that  $g_{S,2}(S_t, A_t, U_{t,2}) = U_{t,2}$  for all  $t \in \{0, \dots, T-2\}$ , independently of  $S_T$  and  $A_t$ . Therefore, it holds that  $s_{t,2}^{\prime} = u_{t-1,2} \geq 1$  for  $t \in \{1, \dots, T-2\}$ , and  $s_{0,2}^{\prime} = s_{0,2} = v_1 \geq 1$ . As a direct consequence, it is easy to see that  $R(s_t', a_t') = 0$  for all  $t \in \{0, \dots, T-2\}$ , and the counterfactual outcome is given by
$$
o ^ {+} \left(\tau^ {\prime}\right) = R \left(s _ {T - 1} ^ {\prime}, a _ {T - 1} ^ {\prime}\right), \tag {14}
$$
In addition to that, we have that  $u_{T - 2,2} = 0$ , hence
$$
\boldsymbol {s} _ {T - 1} ^ {\prime} = \left[ \begin{array}{c} \sum_ {t \in \mathcal {N} _ {T - 2} ^ {\prime}} v _ {t + 1} \\ 0 \end{array} \right] \tag {15}
$$
Now, we will show that, if we can find the action sequence  $\{a_t^*\}_{t=0}^{T-1}$  that gives the optimal counterfactual outcome  $o^+(\tau^*)$  for the aforementioned instance in polynomial time, then we can make a decision about the corresponding instance of the partition problem, also in polynomial time. To this end, let  $\{s_t^*\}_{t=0}^{T-1}$  be the sequence of states in the optimal counterfactual realization and, let  $\mathcal{D}_{T-2}^* = \{t \in \{0,\dots,T-2\} : a_t^* = a_{\mathrm{diff}}\}, \mathcal{N}_{T-2}^* = \{t' \in \{0,\dots,T-2\} : a_{t'}^* = a_{\mathrm{null}}\}$ .
From Eq. 14, we get that the optimal counterfactual outcome is  $o^{+}(\tau^{*}) = R(s_{T - 1}^{*},a_{T - 1}^{*})$ , and it is easy to see that the reward function given in Eq. 13 is always less or equal than zero. If  $o(\tau^{*}) = 0$ , it has to hold that
$$
\begin{array}{l} \max  \left(0, s _ {T - 1, 1} ^ {*} - \frac {\operatorname {s u m} (\mathcal {V})}{2} - s _ {T - 1, 2} ^ {*} \frac {\operatorname {s u m} (\mathcal {V})}{2}\right) = \\ \max  \left(0, \frac {s u m (\mathcal {V})}{2} - s _ {T - 1, 1} ^ {*} - s _ {T - 1, 2} ^ {*} \frac {s u m (\mathcal {V})}{2}\right) = 0 \stackrel {(*)} {\Rightarrow} \\ \left(\sum_ {t \in \mathcal {N} _ {T - 2} ^ {*}} v _ {t + 1}\right) - \frac {\operatorname {s u m} (\mathcal {V})}{2} \leq 0 \quad \text {a n d} \quad \frac {\operatorname {s u m} (\mathcal {V})}{2} - \left(\sum_ {t \in \mathcal {N} _ {T - 2} ^ {*}} v _ {t + 1}\right) \leq 0 \Rightarrow \\ \sum_ {t \in \mathcal {N} _ {T - 2} ^ {*}} v _ {t + 1} = \frac {\operatorname {s u m} (\mathcal {V})}{2}, \\ \end{array}
$$