SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $(\ast)$  follows from Eq. 15. As a consequence, the subsets  $\mathcal{V}_1 = \{v_i : i - 1 \in \mathcal{N}_{T-2}^*\}$  and  $\mathcal{V}_2 = \{v_i : i - 1 \in \mathcal{D}_{T-2}^*\}$  partition  $\mathcal{V}$  and their sums are equal.
On the other hand, if  $o^{+}(\tau^{*}) < 0$ , as we will show, there is no partition of  $\mathcal{V}$  into two sets with equal sums. For the sake of contradiction, assume there are two sets  $\mathcal{V}_1, \mathcal{V}_2$  that partition  $\mathcal{V}$ , with  $\text{sum}(\mathcal{V}_1) = \text{sum}(\mathcal{V}_2) = \text{sum}(\mathcal{V}) / 2$ , and let  $\mathcal{N}_{T-2}' = \{t \in \{0, \dots, T-2\} : v_{t+1} \in \mathcal{V}_1\}$  and  $\mathcal{D}_{T-2}' = \{t \in \{0, \dots, T-2\} : v_{t+1} \in \mathcal{V}_2\}$ . Then, consider the counterfactual episode  $\tau'$  with an action sequence  $\{a_t'\}_{t=0}^{T-1}$  such that its elements take values  $a_{\text{null}}$  and  $a_{\text{diff}}$  based on the sets  $\mathcal{N}_{T-2}', \mathcal{D}_{T-2}'$  respectively, with  $a_{T-1}'$  taking an arbitrary value. It is easy to see that
$$
\begin{array}{l} o ^ {+} \left(\tau^ {\prime}\right) = R \left(s _ {T - 1} ^ {\prime}, a _ {T - 1} ^ {\prime}\right) \\ = R \left(\left[ \begin{array}{c} \sum_ {t \in \mathcal {N} _ {T - 2} ^ {\prime}} v _ {t + 1} \\ 0 \end{array} \right], a _ {T - 1} ^ {\prime}\right) \\ = - \max  \left(0, \sum_ {t \in \mathcal {N} _ {T - 2} ^ {\prime}} v _ {t + 1} - \frac {s u m (\mathcal {V})}{2}\right) - \max  \left(0, \frac {s u m (\mathcal {V})}{2} - \sum_ {t \in \mathcal {N} _ {T - 2} ^ {\prime}} v _ {t + 1}\right) \\ = - \max  \left(0, \frac {s u m (\mathcal {V})}{2} - \frac {s u m (\mathcal {V})}{2}\right) - \max  \left(0, \frac {s u m (\mathcal {V})}{2} - \frac {s u m (\mathcal {V})}{2}\right) \\ = 0 > o ^ {+} (\tau^ {*}), \\ \end{array}
$$
which is a contradiction. This step concludes the reduction and, therefore, the problem given in Eq. 9 cannot be solved in polynomial time, unless  $P = NP$ .
D.1.1 Proof of Lemma 10
It is easy to see that, for all  $\mathbf{u} \in \mathcal{U}$  and for all  $s, s' \in S$ , the function  $g_S(S_t, a_{\mathrm{null}}, \mathbf{u})$  satisfies  $\| g_S(s, a_{\mathrm{null}}, \mathbf{u}) - g_S(s', a_{\mathrm{null}}, \mathbf{u}) \| \leq \| s - s' \|$ , and therefore  $K_{a_{\mathrm{null}}, \mathbf{u}} = 1$  satisfies Definition 1. For the case of  $A_t = a_{\mathrm{diff}}$ , we have that
$$
\begin{array}{l} \| g _ {S} (\boldsymbol {s}, a _ {\text {d i f f}}, \boldsymbol {u}) - g _ {S} (\boldsymbol {s} ^ {\prime}, a _ {\text {d i f f}}, \boldsymbol {u}) \| = \left\| \left[ \begin{array}{c} s _ {1} - s _ {2} \\ 0 \end{array} \right] - \left[ \begin{array}{c} s _ {1} ^ {\prime} - s _ {2} ^ {\prime} \\ 0 \end{array} \right] \right\| = \left\| \left[ \begin{array}{c} (s _ {1} - s _ {1} ^ {\prime}) + (s _ {2} ^ {\prime} - s _ {2}) \\ 0 \end{array} \right] \right\| \\ = \left| \left(s _ {1} - s _ {1} ^ {\prime}\right) + \left(s _ {2} ^ {\prime} - s _ {2}\right) \right| \leq \left| s _ {1} - s _ {1} ^ {\prime} \right| + \left| s _ {2} - s _ {2} ^ {\prime} \right|, \\ \end{array}
$$
and therefore  $\| g_S(\pmb {s},a_{\mathrm{diff}},\pmb {u}) - g_S(\pmb {s}',a_{\mathrm{diff}},\pmb {u})\| ^2\leq (s_1 - s_1')^2 +(s_2 - s_2')^2 +2|s_1 - s_1'||s_2 - s_2'|$  We also have that
$$
\sqrt {2} \left\| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \right\| = \sqrt {2} \sqrt {(s _ {1} - s _ {1} ^ {\prime}) ^ {2} + (s _ {2} - s _ {2} ^ {\prime}) ^ {2}} \Rightarrow 2 \left\| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \right\| ^ {2} = 2 (s _ {1} - s _ {1} ^ {\prime}) ^ {2} + 2 (s _ {2} - s _ {2} ^ {\prime}) ^ {2}.
$$
By combining these, we get
$$
\begin{array}{l} 2 \left\| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \right\| ^ {2} - \left\| g _ {S} (\boldsymbol {s}, a _ {\text {d i f f}}, \boldsymbol {u}) - g _ {S} \left(\boldsymbol {s} ^ {\prime}, a _ {\text {d i f f}}, \boldsymbol {u}\right) \right\| ^ {2} \geq \left(s _ {1} - s _ {1} ^ {\prime}\right) ^ {2} + \left(s _ {2} - s _ {2} ^ {\prime}\right) ^ {2} - 2 \left| s _ {1} - s _ {1} ^ {\prime} \right| \left| s _ {2} - s _ {2} ^ {\prime} \right| \Rightarrow \\ 2 \left\| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \right\| ^ {2} - \left\| g _ {S} (\boldsymbol {s}, a _ {\text {d i f f}}, \boldsymbol {u}) - g _ {S} (\boldsymbol {s} ^ {\prime}, a _ {\text {d i f f}}, \boldsymbol {u}) \right\| ^ {2} \geq \left(\left| s _ {1} - s _ {1} ^ {\prime} \right| - \left| s _ {2} - s _ {2} ^ {\prime} \right|\right) ^ {2} \geq 0. \\ \end{array}
$$
Hence, we can easily see that  $K_{a_{\mathrm{diff}}, \mathbf{u}} = \sqrt{2}$  satisfies Definition 1.
Next, we need to show that, for all  $a \in \mathcal{A}$  there exists a  $C_a \in \mathbb{R}_+$  such that, for all  $s, s' \in S$ , it holds  $|R(s, a) - R(s', a)| \leq C_a \|s - s'\|$ . Note that, to show that a function of the form  $\max(0, f(s))$  with  $f: \mathbb{R}^2 \to \mathbb{R}$  is Lipschitz continuous, it suffices to show that  $f(s)$  is Lipschitz continuous, since the function  $\max(0, x)$  with  $x \in \mathbb{R}$  has a Lipschitz constant equal to 1.
We start by showing that the function  $f(s) = s_1 - \alpha - s_2 \cdot \alpha$  is Lipschitz continuous, where  $\alpha = \text{sum}(\mathcal{V}) / 2$  is a positive constant. For an arbitrary pair  $s, s' \in S$ , we have that
$$
\begin{array}{l} \left| f (\boldsymbol {s}) - f \left(\boldsymbol {s} ^ {\prime}\right) \right| = \left| s _ {1} - s _ {1} ^ {\prime} - \alpha \left(s _ {2} - s _ {2} ^ {\prime}\right) \right| \leq \left| s _ {1} - s _ {1} ^ {\prime} \right| + \alpha \left| s _ {2} - s _ {2} ^ {\prime} \right| \Rightarrow \\ \left| f (s) - f \left(s ^ {\prime}\right) \right| ^ {2} \leq \left(s _ {1} - s _ {1} ^ {\prime}\right) ^ {2} + \left(s _ {2} - s _ {2} ^ {\prime}\right) ^ {2} + 2 \alpha \left| s _ {1} - s _ {1} ^ {\prime} \right| \left| s _ {2} - s _ {2} ^ {\prime} \right|. \\ \end{array}
$$
We also have that
$$
\begin{array}{l} \sqrt {1 + \alpha} \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| = \sqrt {1 + \alpha} \sqrt {(s _ {1} - s _ {1} ^ {\prime}) ^ {2} + (s _ {2} - s _ {2} ^ {\prime}) ^ {2}} \Rightarrow \\ (1 + \alpha) \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| ^ {2} = (1 + \alpha) (s _ {1} - s _ {1} ^ {\prime}) ^ {2} + (1 + \alpha) (s _ {2} - s _ {2} ^ {\prime}) ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} (1 + \alpha) \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| ^ {2} - | f (\boldsymbol {s}) - f (\boldsymbol {s} ^ {\prime}) | ^ {2} \geq \alpha \left(s _ {1} - s _ {1} ^ {\prime}\right) ^ {2} + \alpha \left(s _ {2} - s _ {2} ^ {\prime}\right) ^ {2} - 2 \alpha \left| s _ {1} - s _ {1} ^ {\prime} \right| \left| s _ {2} - s _ {2} ^ {\prime} \right| \Rightarrow \\ (1 + \alpha) \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| ^ {2} - | f (\boldsymbol {s}) - f (\boldsymbol {s} ^ {\prime}) | ^ {2} \geq \alpha \left(| s _ {1} - s _ {1} ^ {\prime} | - | s _ {2} - s _ {2} ^ {\prime} |\right) ^ {2} \geq 0. \\ \end{array}
$$
Hence, we arrive to  $|f(s) - f(s')| \leq \sqrt{1 + \alpha} \| s - s' \|$ , and the function  $f$  is Lipschitz continuous. It is easy to see that the function  $\phi(s) = \alpha - s_1 - s_2 \cdot \alpha$  is also Lipschitz continuous with the proof being almost identical. As a direct consequence, the reward function given in Equation 13 satisfies
Definition 1 with  $C_{a_{\mathrm{null}}} = C_{a_{\mathrm{diff}}} = 2\sqrt{1 + \frac{\text{sum}(\mathcal{V})}{2}}$ . This concludes the proof of the lemma.
D.1.2 Proof of Lemma 11
We will prove the lemma by induction. For the base case of  $t = 1$ , we distinguish between the cases  $a_0' = a_{\mathrm{diff}}$  and  $a_0' = a_{\mathrm{null}}$ . In the first case, we have  $s_{1,1}' = u_{0,1} + s_{0,1} - s_{0,2} = v_1 + 0 - v_1 = 0$  and  $\mathcal{N}_0' = \emptyset$  and, therefore, the statement holds. In the second case, we have  $s_{1,1}' = u_{0,1} + s_{0,1} = v_1 + 0 = v_1$ ,  $\mathcal{N}_0' = \{0\}$  and  $\sum_{t' \in \mathcal{N}_0'} v_{t' + 1} = v_1$ . Therefore, the statement also holds.
For the inductive step  $(t > 1)$ , we assume that  $s_{t - 1,1}^{\prime} = \sum_{t^{\prime}\in \mathcal{N}_{t - 2}^{\prime}}v_{t^{\prime} + 1}$  and we will show that  $s_{t,1}^{\prime} = \sum_{t^{\prime}\in \mathcal{N}_{t - 1}^{\prime}}v_{t^{\prime} + 1}$ . Again, we distinguish between the cases  $a_{t - 1}^{\prime} = a_{\mathrm{diff}}$  and  $a_{t - 1}^{\prime} = a_{\mathrm{null}}$ . However, note that, in both cases,  $s_{t - 1,2}^{\prime} = u_{t - 2,2} + 0 = v_t$ . Therefore, in the case of  $a_{t - 1}^{\prime} = a_{\mathrm{diff}}$ , we get
$$
s _ {t, 1} ^ {\prime} = u _ {t - 1, 1} + s _ {t - 1, 1} ^ {\prime} - s _ {t - 1, 2} ^ {\prime} = v _ {t} + \sum_ {t ^ {\prime} \in \mathcal {N} _ {t - 2} ^ {\prime}} v _ {t ^ {\prime} + 1} - v _ {t} = \sum_ {t ^ {\prime} \in \mathcal {N} _ {t - 2} ^ {\prime}} v _ {t ^ {\prime} + 1} = \sum_ {t ^ {\prime} \in \mathcal {N} _ {t - 1} ^ {\prime}} v _ {t ^ {\prime} + 1},
$$
where the last equation holds because  $a_{t-1}' = a_{\mathrm{diff}}$  and, therefore,  $\mathcal{N}_{t-1}' = \mathcal{N}_{t-2}'$ . In the case of  $a_{t-1}' = a_{\mathrm{null}}$ , we get
$$
s _ {t, 1} ^ {\prime} = u _ {t - 1, 1} + s _ {t - 1, 1} ^ {\prime} = v _ {t} + \sum_ {t ^ {\prime} \in \mathcal {N} _ {t - 2} ^ {\prime}} v _ {t ^ {\prime} + 1} = \sum_ {t ^ {\prime} \in \mathcal {N} _ {t - 1} ^ {\prime}} v _ {t ^ {\prime} + 1},
$$
where the last equation holds because  $a_{t - 1}^{\prime} = a_{\mathrm{null}}$  and, therefore,  $\mathcal{N}_{t - 1}^{\prime} = \mathcal{N}_{t - 2}^{\prime}\cup \{t - 1\}$
We will prove the proposition by induction, starting from the base case, where  $t = T - 1$ . First, Let  $t = T - 1$  and  $l = k$ . It is easy to see that, if the process is at a state  $s \in S$  in the last
time step with no action changes left, the best reward that can be achieved is  $R(\boldsymbol{s}, a_{T-1})$ , as already discussed after Eq. 10. Therefore, it holds that  $|V_{\tau}(\boldsymbol{s}, k, T-1) - V_{\tau}(\boldsymbol{s}', k, T-1)| = |R(\boldsymbol{s}, a_{T-1}) - R(\boldsymbol{s}', a_{T-1})| \leq C_{a_{T-1}} \| \boldsymbol{s} - \boldsymbol{s}' \| \leq C \| \boldsymbol{s} - \boldsymbol{s}' \|$ , where the last step holds because  $C = \max_{a \in \mathcal{A}} C_a$ . Now, consider the case of  $t = T-1$  with  $l$  taking an arbitrary value in  $\{0, \dots, k-1\}$ . Let  $s, s'$  be two states in  $S$  and  $a^*$  be the action that gives the maximum immediate reward at state  $s$ , that is,  $a^* = \operatorname{argmax}_{a \in \mathcal{A}} \{R(\boldsymbol{s}, a)\}$ . Then, we get
$$
\begin{array}{l} | V _ {\tau} (\boldsymbol {s}, l, T - 1) - V _ {\tau} (\boldsymbol {s} ^ {\prime}, l, T - 1) | = \left| \max  _ {a \in \mathcal {A}} \{R (\boldsymbol {s}, a) \} - \max  _ {a \in \mathcal {A}} \{R (\boldsymbol {s} ^ {\prime}, a) \} \right| \\ \stackrel {(*)} {\leq} \left| R (\boldsymbol {s}, a ^ {*}) - R (\boldsymbol {s} ^ {\prime}, a ^ {*}) \right| \leq C _ {a ^ {*}} \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| \leq C \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \|, \\ \end{array}
$$
where  $(\ast)$  follows from the fact that  $R(\pmb{s}^{\prime},a^{*})\leq \max_{a\in \mathcal{A}}\{R(\pmb{s}^{\prime},a)\}$ . Therefore, for any  $l\in$ $\{0,\dots ,k\}$  and  $\pmb {s},\pmb{s}^{\prime}\in \mathcal{S}$ , it holds that  $|V_{\tau}(\pmb {s},l,T - 1) - V_{\tau}(\pmb{s}^{\prime},l,T - 1)|\leq L_{T - 1}\| \pmb {s} - \pmb{s}^{\prime}\|$ , where  $L_{T - 1} = C$ .
Now, we will proceed to the induction step. Let  $t < T - 1$ ,  $l < k$  and, as an inductive hypothesis, assume that  $L_{t + 1} \in \mathbb{R}_+$  as defined in Lemma 5 is such that, for all  $l \in \{0, \dots, k\}$  and  $s, s' \in S$ , it holds that  $|V_{\tau}(s, l, t + 1) - V_{\tau}(s', l, t + 1)| \leq L_{t + 1} \| s - s' \|$ . Additionally, let  $(s_a, l_a), (s'_a, l'_a)$  denote the enhanced states that follow from  $(s, l), (s', l)$  after taking an action  $a$ , i.e.,  $(s_a, l_a) = F_{\tau, t}^+( (s, l), a)$  and  $(s'_a, l'_a) = F_{\tau, t}^+( (s', l), a)$ . Lastly, let  $a^*$  be the action that maximizes the future total reward starting from state  $s$ , i.e.,  $a^* = \operatorname{argmax}_{a \in \mathcal{A}} \{ R(s, a) + V_{\tau}(s_a, l_a, t + 1) \}$ . Then, we have that
$$
\begin{array}{l} \left| V _ {\tau} (\boldsymbol {s}, l, t) - V _ {\tau} \left(\boldsymbol {s} ^ {\prime}, l, t\right) \right| \\ = \left| \max  _ {a \in \mathcal {A}} \left\{R (\boldsymbol {s}, a) + V _ {\tau} \left(\boldsymbol {s} _ {a}, l _ {a}, t + 1\right) \right\} - \max  _ {a \in \mathcal {A}} \left\{R \left(\boldsymbol {s} ^ {\prime}, a\right) + V _ {\tau} \left(\boldsymbol {s} _ {a} ^ {\prime}, l _ {a}, t + 1\right) \right\} \right| \\ \stackrel {(*)} {\leq} \left| R \left(\boldsymbol {s}, a ^ {*}\right) + V _ {\tau} \left(\boldsymbol {s} _ {a ^ {*}}, l _ {a ^ {*}}, t + 1\right) - R \left(\boldsymbol {s} ^ {\prime}, a ^ {*}\right) - V _ {\tau} \left(\boldsymbol {s} _ {a ^ {*}} ^ {\prime}, l _ {a ^ {*}}, t + 1\right) \right| \\ \leq \left| R \left(\boldsymbol {s}, a ^ {*}\right) - R \left(\boldsymbol {s} ^ {\prime}, a ^ {*}\right) \right| + \left| V _ {\tau} \left(\boldsymbol {s} _ {a ^ {*}}, l _ {a ^ {*}}, t + 1\right) - V _ {\tau} \left(\boldsymbol {s} _ {a ^ {*}} ^ {\prime}, l _ {a ^ {*}}, t + 1\right) \right| \\ \stackrel {(*)} {\leq} C _ {a ^ {*}} \left\| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \right\| + L _ {t + 1} \left\| \boldsymbol {s} _ {a ^ {*}} - \boldsymbol {s} _ {a ^ {*}} ^ {\prime} \right\| \\ \leq C _ {a ^ {*}} \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| + L _ {t + 1} K _ {a ^ {*}, \boldsymbol {u} _ {t}} \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| \\ \stackrel {(* * *)} {\leq} C \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| + L _ {t + 1} K _ {\boldsymbol {u} _ {t}} \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| \\ = \left(C + L _ {t + 1} K _ {\boldsymbol {u} _ {t}}\right) \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| = L _ {t} \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \|. \\ \end{array}
$$
In the above,  $(\ast)$  holds due to  $R(s', a^*) + V_{\tau}(s_{a^*}', l_{a^*}, t + 1) \leq \max_{a \in \mathcal{A}} \{R(s', a) + V_{\tau}(s_a', l_a, t + 1)\}$ ,  $(\ast\ast)$  follows from the inductive hypothesis, and  $(\ast\ast\ast)$  holds because  $C = \max_{a \in \mathcal{A}} C_a$  and  $K_{u_t} = \max_{a \in \mathcal{A}} K_{a,u_t}$ . It is easy to see that, similar arguments hold for the simple case of  $l = k$ , therefore, we omit the details. This concludes the inductive step and the proof of Lemma 5.
D.3 Proof of Proposition 6
We will prove the proposition by induction, starting from the base case, where  $t = T - 1$ . If  $t = T - 1$ , the algorithm initializes  $\hat{V}_{\tau}(s,l,T - 1)$  to  $\max_{a\in \mathcal{A}}R(s,a)$  for all  $s\in S_{\dagger}, l\in \{0,\dots ,k - 1\}$  and  $\hat{V}_{\tau}(s,k,T - 1)$  to  $R(s,a_{T - 1})$ . It is easy to see that those values are optimal, as already discussed after Eq. 10. Therefore, the base case  $\hat{V}_{\tau}(s,l,T - 1)\geq V_{\tau}(s,l,T - 1)$  follows trivially.
Now, we will proceed to the induction step. Let  $t < T - 1$  and, as an inductive hypothesis, assume that  $\hat{V}_{\tau}(s,l,t + 1)\geq V_{\tau}(s,l,t + 1)$  for all  $s\in S_{\dagger},l\in \{0,\ldots ,k\}$ . Our goal is to show that  $\hat{V}_{\tau}(s,l,t)\geq V_{\tau}(s,l,t)$  for all  $s\in S_{\dagger},l\in \{0,\dots ,k\}$ . First, let  $l < k$ . For a given point  $s\in S_{\dagger}$ , Algorithm 1 finds the next state  $s_a$  that would have occurred by taking each action  $a$ , i.e.,  $(s_a,l_a) = F_{\tau ,t}^{+}((s,l),a)$  and it computes the associated value  $V_{a} = \min_{s_{\dagger}\in S_{\dagger}}\{\hat{V}_{\tau}(s_{\dagger},l_{a},t + 1) + L_{t + 1}\| s_{\dagger} - s_{a}\| \}$ . Then, it
simply sets  $\hat{V}_{\tau}(\pmb {s},l,t)$  equal to  $\max_{a\in \mathcal{A}}\{R(\pmb {s},a) + V_a\}$ . We have that
$$
\begin{array}{l} V _ {a} = \min  _ {\boldsymbol {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{\hat {V} _ {\tau} \left(\boldsymbol {s} _ {\dagger}, l _ {a}, t + 1\right) + L _ {t + 1} \left\| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a} \right\| \right\} \\ \stackrel {(*)} {\geq} \min  _ {\boldsymbol {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{V _ {\tau} \left(\boldsymbol {s} _ {\dagger}, l _ {a}, t + 1\right) + L _ {t + 1} \| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a} \| \right\} \\ \stackrel {(* *)} {\geq} \min  _ {\boldsymbol {s} _ {\dagger} \in S _ {\dagger}} \left\{V _ {\tau} \left(\boldsymbol {s} _ {a}, l _ {a}, t + 1\right) \right\} \\ = V _ {\tau} \left(\boldsymbol {s} _ {a}, l _ {a}, t + 1\right), \\ \end{array}
$$
where  $(\ast)$  follows from the inductive hypothesis, and  $(\ast\ast)$  is a consequence of Lemma 5. Then, we get
$$
\hat {V} _ {\tau} (\pmb {s}, l, t) = \max _ {a \in \mathcal {A}} \left\{R (\pmb {s}, a) + V _ {a} \right\} \geq \max _ {a \in \mathcal {A}} \left\{R (\pmb {s}, a) + V _ {\tau} (\pmb {s} _ {a}, l _ {a}, t + 1) \right\} = V _ {\tau} (\pmb {s}, l, t).
$$
Additionally, when  $l = k$ , we have  $\hat{V}_{\tau}(\boldsymbol{s},k,t) = R(\boldsymbol{s},a_{t}) + \min_{\boldsymbol{s}_{\dagger}\in \mathcal{S}_{\dagger}}\{\hat{V}_{\tau}(\boldsymbol{s}_{\dagger},k,t + 1) + L_{t + 1}\| \boldsymbol{s}_{\dagger} - \boldsymbol{s}_{a_{t}}\| \}$  and  $V_{\tau}(\boldsymbol{s},k,t) = R(\boldsymbol{s},a_{t}) + V_{\tau}(\boldsymbol{s}_{a_{t}},k,t + 1)$ . Therefore, the proof for  $\hat{V}_{\tau}(\boldsymbol{s},k,t)\geq V_{\tau}(\boldsymbol {s},k,t)$  is almost identical.
D.4 Proof of Theorem 7
We start from the case where  $t = T - 1$ . Let  $v = (s, l, T - 1)$  and, consider an edge associated with action  $a^*$  connecting  $v$  to the goal node  $v_T = (s_{\emptyset}, k, T)$  that carries a reward  $R(\pmb{s}, a^*)$ . Then, we have
$$
\hat {V} _ {\tau} (\boldsymbol {s}, l, T - 1) = \max _ {a \in \mathcal {A} ^ {\prime}} R (\boldsymbol {s}, a) \geq R (\boldsymbol {s}, a ^ {*}) + 0 = R (\boldsymbol {s}, a ^ {*}) + \hat {V} _ {\tau} (\boldsymbol {s} _ {\emptyset}, k, T),
$$
and the base case holds.
For the more general case, where  $t < T - 1$ , we first establish the following intermediate result, whose proof is given in Appendix D.4.1.
Lemma 12. For every  $s, s' \in S$ ,  $l \in \{0, \dots, k\}$ ,  $t \in \{0, \dots, T - 1\}$ , it holds that  $|\hat{V}_{\tau}(s, l, t) - \hat{V}_{\tau}(s', l, t)| \leq L_t \| s - s' \|$ , where  $L_t$  is as defined in Lemma 5.
That said, consider an edge associated with an action  $a^*$  connecting node  $v = (s, l, t)$  to node  $v_{a^*} = (s_{a^*}, l_{a^*}, t + 1)$ . Then, we have
$$
\begin{array}{l} \hat {V} _ {\tau} (\boldsymbol {s}, l, t) = \max  _ {a \in \mathcal {A} ^ {\prime}} \left\{R (\boldsymbol {s}, a) + \min  _ {\boldsymbol {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{\hat {V} _ {\tau} (\boldsymbol {s} _ {\dagger}, l _ {a}, t + 1) + L _ {t + 1} \| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a} \| \right\} \right\} \\ \geq R (\pmb {s}, a ^ {*}) + \min _ {\pmb {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{\hat {V} _ {\tau} (\pmb {s} _ {\dagger}, l _ {a ^ {*}}, t + 1) + L _ {t + 1} \| \pmb {s} _ {\dagger} - \pmb {s} _ {a ^ {*}} \| \right\} \\ \geq R \left(\boldsymbol {s}, a ^ {*}\right) + \min  _ {\boldsymbol {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{\hat {V} _ {\tau} \left(\boldsymbol {s} _ {a ^ {*}}, l _ {a ^ {*}}, t + 1\right) \right\} \\ = R \left(\boldsymbol {s}, a ^ {*}\right) + \hat {V} _ {\tau} \left(\boldsymbol {s} _ {a ^ {*}}, l _ {a ^ {*}}, t + 1\right). \\ \end{array}
$$
That concludes the proof and, therefore, the heuristic function  $\hat{V}_{\tau}$  is consistent.
D.4.1 Proof of Lemma 12
Without loss of generality, we will assume that  $l < k$ , since the proof for the case of  $l = k$  is similar and more straightforward. We start from the case where  $t = T - 1$  and, for two states  $s, s' \in S$  we have
$$
\begin{array}{l} | \hat {V} _ {\tau} (\boldsymbol {s}, l, T - 1) - \hat {V} _ {\tau} (\boldsymbol {s} ^ {\prime}, l, T - 1) | = \left| \max  _ {a \in \mathcal {A}} R (\boldsymbol {s}, a) - \max  _ {a \in \mathcal {A}} R (\boldsymbol {s} ^ {\prime}, a) \right| \\ = \left| V _ {\tau} (\boldsymbol {s}, l, T - 1) - V _ {\tau} \left(\boldsymbol {s} ^ {\prime}, l, T - 1\right) \right| \leq C \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \| = L _ {T - 1} \| \boldsymbol {s} - \boldsymbol {s} ^ {\prime} \|, \\ \end{array}
$$
where the last inequality follows from Lemma 5.
Now, consider the case  $t < T - 1$ , and let  $(s_a, l_a)$  denote the enhanced state that follows from  $(s, l)$  after taking an action  $a$  at time  $t$ , i.e.,  $(s_a, l_a) = F_{\tau, t}^{+}((s, l), a)$ . Then, we have
$$
\begin{array}{l} \left| \hat {V} _ {\tau} (\boldsymbol {s}, l, t) - \hat {V} _ {\tau} \left(\boldsymbol {s} ^ {\prime}, l, t\right) \right| \\ = \left| \max  _ {a \in \mathcal {A}} \left\{R (\boldsymbol {s}, a) + \min  _ {\boldsymbol {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{\hat {V} _ {\tau} \left(\boldsymbol {s} _ {\dagger}, l _ {a}, t + 1\right) + L _ {t + 1} \| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a} \| \right\} \right\} \right. \\ \left. - \max  _ {a \in \mathcal {A}} \left\{R \left(\boldsymbol {s} ^ {\prime}, a\right) + \min  _ {\boldsymbol {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{\hat {V} _ {\tau} \left(\boldsymbol {s} _ {\dagger}, l _ {a}, t + 1\right) + L _ {t + 1} \left\| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a} ^ {\prime} \right\| \right\} \right\} \right\rvert . \tag {16} \\ \end{array}
$$
Let  $a^*$  be the action  $a \in \mathcal{A}$  that maximizes the first part of the above subtraction, i.e.,
$$
a ^ {*} = \operatorname * {a r g m a x} _ {a \in \mathcal {A}} \left\{R (\boldsymbol {s}, a) + \min  _ {\boldsymbol {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{\hat {V} _ {\tau} (\boldsymbol {s} _ {\dagger}, l _ {a}, t + 1) + L _ {t + 1} \| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a} \| \right\} \right\}
$$
Then, Eq. 16 implies that
$$
\begin{array}{l} | \hat {V} _ {\tau} (\boldsymbol {s}, l, t) - \hat {V} _ {\tau} (\boldsymbol {s} ^ {\prime}, l, t) | \leq \left| R (\boldsymbol {s}, a ^ {*}) + \min  _ {\boldsymbol {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{\hat {V} _ {\tau} (\boldsymbol {s} _ {\dagger}, l _ {a ^ {*}}, t + 1) + L _ {t + 1} \| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a ^ {*}} \| \right\} \right. \\ \left. - R \left(\boldsymbol {s} ^ {\prime}, a ^ {*}\right) - \min  _ {\boldsymbol {s} _ {\dagger} \in S _ {\dagger}} \left\{\hat {V} _ {\tau} \left(\boldsymbol {s} _ {\dagger}, l _ {a ^ {*}}, t + 1\right) + L _ {t + 1} \| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a ^ {*}} ^ {\prime} \| \right\} \right| \\ \leq \left| R \left(\boldsymbol {s}, a ^ {*}\right) - R \left(\boldsymbol {s} ^ {\prime}, a ^ {*}\right) \right| \\ + \left| \min  _ {\boldsymbol {s} _ {\dagger} \in S _ {\dagger}} \left\{\hat {V} _ {\tau} \left(\boldsymbol {s} _ {\dagger}, l _ {a ^ {*}}, t + 1\right) + L _ {t + 1} \| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a ^ {*}} \| \right\} \right. \\ \left. - \min  _ {\boldsymbol {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{\hat {V} _ {\tau} \left(\boldsymbol {s} _ {\dagger}, l _ {a ^ {*}}, t + 1\right) + L _ {t + 1} \left\| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a ^ {*}} ^ {\prime} \right\| \right\} \right\rvert \tag {17} \\ \end{array}
$$
Now, let  $\tilde{s}$  be the  $s_{\dagger} \in S_{\dagger}$  that minimizes the second part of the above subtraction, i.e.,
$$
\tilde {\boldsymbol {s}} = \operatorname * {a r g m i n} _ {\boldsymbol {s} _ {\dagger} \in \mathcal {S} _ {\dagger}} \left\{\hat {V} _ {\tau} (\boldsymbol {s} _ {\dagger}, l _ {a ^ {*}}, t + 1) + L _ {t + 1} \| \boldsymbol {s} _ {\dagger} - \boldsymbol {s} _ {a ^ {*}} ^ {\prime} \| \right\}.
$$
As a consequence and in combination with Eq. 17, we get