SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Delta \left(V _ {\theta_ {1}, \theta_ {2}} ^ {p}\right) ^ {\eta} (\boldsymbol {x}) \leq - \beta_ {\theta_ {1}, \theta_ {2}} ^ {\eta} \left(V _ {\theta_ {1}, \theta_ {2}} ^ {p} (\boldsymbol {x})\right) ^ {\alpha_ {\theta_ {1}, \theta_ {2}} ^ {p} + \eta - 1} + b _ {\theta_ {1}, \theta_ {2}} ^ {\eta} \mathbb {I} _ {C _ {\theta_ {1}, \theta_ {2}} ^ {p}} (\boldsymbol {x}), \quad \boldsymbol {x} \in \mathcal {X}, \tag {37}
$$
where for  $\eta \in (0,1)$ ,  $\tilde{\beta}_{\theta_1,\theta_2}^p \coloneqq \min (\beta_{\theta_1,\theta_2}^p, 1)$  and
$$
\beta_ {\theta_ {1}, \theta_ {2}} ^ {\eta} = \eta \tilde {\beta} _ {\theta_ {1}, \theta_ {2}} ^ {p}, b _ {\theta_ {1}, \theta_ {2}} ^ {\eta} = \left(b _ {\theta_ {1}, \theta_ {2}} ^ {p}\right) ^ {\eta} + \eta \tilde {\beta} _ {\theta_ {1}, \theta_ {2}} ^ {p} \max  \left(1, \left(\tilde {\beta} _ {\theta_ {1}, \theta_ {2}} ^ {p}\right) ^ {\left(\alpha_ {\theta_ {1}, \theta_ {2}} ^ {p} + \eta - 1\right) / \left(1 - \alpha_ {\theta_ {1}, \theta_ {2}} ^ {p}\right)}\right), \tag {38}
$$
and for  $\eta = 1$ ,  $\beta_{\theta_1,\theta_2}^{\eta} = \beta_{\theta_1,\theta_2}^p$  and  $b_{\theta_1,\theta_2}^{\eta} = b_{\theta_1,\theta_2}^p$ . Consequently, the following result is immediate from the proof of [29, Theorem 3.6]; for completeness, we provide the proof in Appendix D.1.
Lemma 9. Suppose a finite set  $C_{\theta_1,\theta_2}^p$ , constants  $\beta_{\theta_1,\theta_2}^p, b_{\theta_1,\theta_2}^p > 0$ ,  $r / (r + 1) \leq \alpha_{\theta_1,\theta_2}^p < 1$ , and a function  $V_{\theta_1,\theta_2}^p: \mathcal{X} \to [1, +\infty)$  exists such that (4) holds. Then, there exist a sequence of non-negative functions  $V_{\theta_1,\theta_2}^i: \mathcal{X} \to [1, +\infty)$  for  $i = 0, \dots, r + 1$  that satisfy the following system of drift equations for finite sets  $C_{\theta_1,\theta_2}^i$ , constants  $b_{\theta_1,\theta_2}^i \geq 0$  and  $\beta_{\theta_1,\theta_2}^i > 0$ :
$$
\Delta V _ {\theta_ {1}, \theta_ {2}} ^ {i - 1} (\boldsymbol {x}) \leq - \beta_ {\theta_ {1}, \theta_ {2}} ^ {i} V _ {\theta_ {1}, \theta_ {2}} ^ {i} (\boldsymbol {x}) + b _ {\theta_ {1}, \theta_ {2}} ^ {i} \mathbb {I} _ {C _ {\theta_ {1}, \theta_ {2}} ^ {i}} (\boldsymbol {x}), \quad \boldsymbol {x} \in \mathcal {X}, i = 1, \dots , r + 1. \tag {39}
$$
Notice that  $r$  is the maximum degree of the cost function  $c$  defined in Assumption 1. Following the proof and approach of [29] and using the set of equations (39), we can find an upper-bound for  $\mathbb{E}_{\boldsymbol{x}}[\tau_{0^d}^i]$  for  $i = 1, \dots, r + 1$  in Lemma 10. In order to establish upper bounds for the first  $r + 1$  moments of  $\tau_{0^d}$ , it is crucial to choose the value of  $\alpha_{\theta_1, \theta_2}^p$  greater than or equal to  $\frac{r}{r + 1}$ , as demonstrated in the proof of Lemma 10 in Appendix D.2
Lemma 10. For  $i = 1,\dots ,r + 1$  and for all  $\pmb {x}\in \mathcal{X}$
$$
\mathbb {E} _ {\boldsymbol {x}} ^ {\pi_ {\theta_ {2}} ^ {*}} \left[ \left(\tau_ {0 ^ {d}}\right) ^ {i} \right] \leq i \phi_ {\theta_ {1}, \theta_ {2}} ^ {p} (i) \left(V _ {\theta_ {1}, \theta_ {2}} ^ {p} (\boldsymbol {x}) + b _ {\theta_ {1}, \theta_ {2}} ^ {p} \alpha_ {C _ {\theta_ {1}, \theta_ {2}} ^ {p}}\right),
$$
where  $\phi_{\theta_1,\theta_2}^p (i)\coloneqq \prod_{j = 1}^i\frac{1}{\beta_{\theta_1,\theta_2}^{\eta_j}}\Big(2^{j - 1} + (j - 1)\alpha_{C_{\theta_1,\theta_2}^p}b_{\theta_1,\theta_2}^{\eta_j}\Big),$ $\eta_i = 1 - (i - 1)(1 - \alpha_{\theta_1,\theta_2}^p)$
$b_{\theta_1,\theta_2}^{\eta_i}$  and  $\beta_{\theta_1,\theta_2}^{\eta_i}$  defined in (38), and  $\alpha_{C_{\theta_1,\theta_2}^p} = \left(\min_{\boldsymbol {y}\in C_{\theta_1,\theta_2}^p}K_{\theta_1,\theta_2}(\boldsymbol {y})\right)^{-1}$ .
Based on Lemma 10, we impose the conditions of Assumption 4 to obtain uniform (over model class) and polynomial (in norm of the state) upper-bounds on the moments of hitting times to  $0^d$ . Moreover, these conditions lead to a uniform characterization of parameters of Lemma 10 over all models in our class.
C.2 Distribution of return times to state  $0^{d}$
For any  $\theta_1, \theta_2 \in \Theta$ , consider the Markov process with transition kernel  $P_{\theta_1}^{\pi_{\hat{\theta}_2}}$  obtained from the MDP  $(\mathcal{X}, \mathcal{A}, c, P_{\theta_1})$  by following policy  $\pi_{\theta_2}^*$ . In the following lemma, we show that the tail probabilities of the return times to the common state  $0^d$ , again  $\tau_{0^d}$ , converge geometrically fast to 0, and characterize the convergence parameters in terms of the constants given in Assumption 3. Explicitly, we show
$$
\mathbb {P} _ {0 ^ {d}} (\tau_ {0 ^ {d}} > n) \leq c _ {\theta_ {1}, \theta_ {2}} ^ {g} \left(\tilde {\gamma} _ {\theta_ {1}, \theta_ {2}} ^ {g}\right) ^ {n},
$$
for problem and policy dependent constants  $c_{\theta_1,\theta_2}^g$  and  $\tilde{\gamma}_{\theta_1,\theta_2}^g$ . We will follow the method outlined in [27] with the goal to identify problem dependent parameters that will be relevant to our results. Proof of the following lemma is given in Appendix D.3 and follows the methodology of [27].
Lemma 11. For every  $\theta_{1},\theta_{2}\in \Theta$  in the Markov process obtained from the Markov decision process  $(\mathcal{X},\mathcal{A},c,P_{\theta_1})$  following policy  $\pi_{\theta_2}^*$ , the return time to state 0 starting from state 0 satisfies the following:
$$
c _ {\theta_ {1}, \theta_ {2}} ^ {g} = \frac {b _ {\theta_ {1} , \theta_ {2}} ^ {g} \left(\tilde {b} _ {\theta_ {1} , \theta_ {2}} ^ {g}\right) ^ {2}}{\tilde {b} _ {\theta_ {1} , \theta_ {2}} ^ {g} - 1} \quad a n d \quad \tilde {\gamma} _ {\theta_ {1}, \theta_ {2}} ^ {g} = 1 - \frac {1}{\tilde {b} _ {\theta_ {1} , \theta_ {2}} ^ {g}},
$$
$$
\tilde{b}_{\theta_{1},\theta_{2}}^{g} = \frac{3b_{\theta_{1},\theta_{2}}^{g} + 1}{1 - \gamma_{\theta_{1},\theta_{2}}^{g}}\left(|C_{\theta_{1},\theta_{2}}^{g}|^{2}\max \left(1,\max_{\boldsymbol {u}\in C_{\theta_{1},\theta_{2}}^{g}\setminus \{0^{d}\}}\mathbb{E}_{\boldsymbol{u}}^{\pi_{\theta_{2}}^{*}}[\tau_{0^{d}}]\right)\right).
$$
Based on Lemma 11, it is necessary to impose the conditions in Assumption 3 to obtain uniform tail probability bounds on  $\tau_{0^d}$  for all model parameters and policy choices in  $\Theta$ . Moreover, these conditions lead to a uniform characterization of  $c_{\theta_1,\theta_2}^g$  and  $\tilde{\gamma}_{\theta_1,\theta_2}^g$  over  $\Theta$ . Furthermore, as a result of Lemma 10 and uniformity conditions of Assumption 4,  $\mathbb{E}_{\boldsymbol{u}}^{\pi_{\theta_2}^*}[\tau_{0^d}]$  has a uniform bound over  $\Theta$  and  $C_{\theta_1,\theta_2}^g \setminus \{0^d\}$ , which can be characterized in terms of the polynomial Lyapunov function.
D Proofs of hitting time bounds
D.1 Proof of Lemma 9
Proof. In the proof, to avoid cumbersome notation we will drop the indices  $\theta_{1},\theta_{2}$ . Following the proof of Theorem 3.6 in [29], we choose  $\eta_{i} = 1 - (i - 1)(1 - \alpha^{p})$  for  $i = 1,\ldots ,r + 1$  and note that as  $\alpha^p\in [\frac{r}{r + 1},1)$ , we have  $\eta_i\in [\frac{1}{r + 1},1]$ . As a result, we can apply (37) to each  $\eta_{i}$  to get
$$
\Delta \left(V ^ {p}\right) ^ {\eta_ {i}} (\boldsymbol {x}) \leq - \beta^ {\eta_ {i}} \left(V ^ {p} (\boldsymbol {x})\right) ^ {i \alpha^ {p} - i + 1} + b ^ {\eta_ {i}} \mathbb {I} _ {C ^ {p}} (\boldsymbol {x}), \quad i = 1, \dots , r + 1.
$$
Thus, the system of drift equations (39) hold for
$$
V _ {i} = \left(V ^ {p}\right) ^ {1 - i \left(1 - \alpha^ {p}\right)}, \quad i = 0, \dots , r + 1,
$$
$$
\beta_ {i} = \beta^ {\eta_ {i}}, \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad i = 1, \dots , r + 1,
$$
$$
b _ {i} = b ^ {\eta_ {i}}, \quad i = 1, \dots , r + 1,
$$
$$
C _ {i} = C ^ {p}, \quad i = 1, \dots , r + 1,
$$
where  $\beta^{\eta_i}$  and  $b^{\eta_i}$  are defined in (38).
D.2 Proof of Lemma 10
The proof of Lemma 10 uses the following lemma.
Lemma 12 (Proposition 11.3.2, [43]). Suppose for nonnegative functions  $f$ ,  $g$ , and  $V$  on the state space  $\mathcal{X}$  and every  $k \in \mathbb{Z}_+$ , the following holds:
$$
\mathbb {E} \left[ V \left(X _ {k + 1}\right) \mid \mathcal {F} _ {k} \right] \leq V \left(X _ {k}\right) - f \left(X _ {k}\right) + g \left(X _ {k}\right).
$$
Then, for any initial condition  $x$  and stopping time  $\tau$
$$
\mathbb {E} _ {x} \left[ \sum_ {k = 0} ^ {\tau - 1} f (X _ {k}) \right] \leq V (x) + \mathbb {E} _ {x} \left[ \sum_ {k = 0} ^ {\tau - 1} g (X _ {k}) \right].
$$
Proof of Lemma 10. Following [29], the proof uses an induction argument. We will use the notation of Lemma 9 for simplicity. Similarly, in this proof we will also denote  $\phi_{\theta_1,\theta_2}^p (i)$  as  $\phi (i)$ ,  $K_{\theta_1,\theta_2}(\cdot)$  as  $K(\cdot)$ , and  $V_{\theta_1,\theta_2}^i,b_{\theta_1,\theta_2}^i,\beta_{\theta_1,\theta_2}^i,C_{\theta_1,\theta_2}^i$  as  $V_{i},b_{i},\beta_{i},C_{i}$ .
From irreducibility, for all  $\pmb{x} \in \mathcal{X}$ ,  $K(\pmb{x})$  is positive and finite. Considering the system of drift equations found in Lemma 9,  $C_i = C^p$  is a finite set for all  $i = 1, \dots, r + 1$ . Thus,  $\min_{\pmb{y} \in C_i} K(\pmb{y})$  is strictly positive. For all  $\pmb{x} \in \mathcal{X}$  and  $i = 1, \dots, r + 1$ , we have
$$
\mathbb {I} _ {C _ {i}} (\boldsymbol {x}) \leq \left(\min  _ {\boldsymbol {y} \in C _ {i}} K (\boldsymbol {y})\right) ^ {- 1} K (\boldsymbol {x}). \tag {40}
$$
We set  $\alpha_{C^p} := (\min_{\boldsymbol{y} \in C_i} K(\boldsymbol{y}))^{-1} = (\min_{\boldsymbol{y} \in C^p} K(\boldsymbol{y}))^{-1}$ . From Lemma 9, for  $j = 1$  and  $\boldsymbol{x} \in \mathcal{X}$
$$
\Delta V _ {0} (\boldsymbol {x}) \leq - \beta_ {1} V _ {1} (\boldsymbol {x}) + b _ {1} \mathbb {I} _ {C _ {1}} (\boldsymbol {x}).
$$
By applying Lemma 12, for all  $\pmb{x} \in \mathcal{X}$  we get
$$
\beta_ {1} \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {k = 0} ^ {\tau_ {0} d - 1} V _ {1} (\boldsymbol {X} _ {k}) \right] \leq V _ {0} (\boldsymbol {x}) + b _ {1} \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {k = 0} ^ {\tau_ {0} d - 1} \mathbb {I} _ {C _ {1}} (\boldsymbol {X} _ {k}) \right]. \tag {41}
$$
Using (40) and (41), followed by noting that
$$
K (\boldsymbol {x}) = \sum_ {n = 0} ^ {\infty} 2 ^ {- n - 2} P ^ {n} (\boldsymbol {x}, 0 ^ {d}) = \sum_ {n = 0} ^ {\infty} 2 ^ {- n - 2} \mathbb {E} _ {\boldsymbol {x}} \left[ \mathbb {I} _ {0 ^ {d}} \left(\boldsymbol {X} _ {n}\right) \right],
$$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {k = 0} ^ {\tau_ {0} d - 1} V _ {1} \left(\boldsymbol {X} _ {k}\right) \right] \leq \frac {1}{\beta_ {1}} V _ {0} (\boldsymbol {x}) + \frac {b _ {1} \alpha_ {C ^ {p}}}{\beta_ {1}} \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {n = 0} ^ {\infty} 2 ^ {- n - 2} \sum_ {k = 0} ^ {\tau_ {0} d - 1} \mathbb {I} _ {0 ^ {d}} \left(\boldsymbol {X} _ {k + n}\right) \right] \\ = \frac {1}{\beta_ {1}} V _ {0} (\boldsymbol {x}) + \frac {b _ {1} \alpha_ {C ^ {p}}}{\beta_ {1}} \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {n = 0} ^ {\infty} 2 ^ {- n - 2} \sum_ {k = n} ^ {\tau_ {0 ^ {d}} - 1 + n} \mathbb {I} _ {0 ^ {d}} \left(\boldsymbol {X} _ {k}\right) \right] \\ \leq \frac {1}{\beta_ {1}} V _ {0} (\boldsymbol {x}) + \frac {b _ {1} \alpha_ {C ^ {p}}}{\beta_ {1}} \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {n = 0} ^ {\infty} 2 ^ {- n - 2} \sum_ {k = n \vee \tau_ {0 ^ {d}}} ^ {\tau_ {0 ^ {d}} - 1 + n} \mathbb {I} _ {0 ^ {d}} \left(\boldsymbol {X} _ {k}\right) \right] \\ \leq \frac {1}{\beta_ {1}} V _ {0} (\boldsymbol {x}) + \frac {b _ {1} \alpha_ {C ^ {p}}}{\beta_ {1}} \sum_ {n = 0} ^ {\infty} 2 ^ {- n - 2} (n + 1) \\ = \frac {1}{\beta_ {1}} V _ {0} (\boldsymbol {x}) + \frac {b _ {1} \alpha_ {C ^ {p}}}{\beta_ {1}}. \\ \end{array}
$$
As  $V_{1}(\pmb {x})\geq 1$  , this gives us a bound on  $\mathbb{E}_{\pmb{x}}[\tau_{0^d}]$  as follows:
$$
\mathbb {E} _ {\boldsymbol {x}} \left[ \tau_ {0 ^ {d}} \right] \leq \frac {1}{\beta_ {1}} V _ {0} (\boldsymbol {x}) + \frac {b _ {1} \alpha_ {C ^ {p}}}{\beta_ {1}}.
$$
Assume for  $i \geq 1$ , by the induction assumption we have
$$
\mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {k = 0} ^ {\tau_ {0} d - 1} (k + 1) ^ {i - 1} V _ {i} \left(\boldsymbol {X} _ {k}\right) \right] \leq \phi (i) \left(V _ {0} (\boldsymbol {x}) + b _ {1} \alpha_ {C ^ {p}}\right). \tag {42}
$$
Set  $j = i + 1$  in (39), which yields
$$
\Delta V _ {i} (\boldsymbol {x}) \leq - \beta_ {i + 1} V _ {i + 1} (\boldsymbol {x}) + b _ {i + 1} \mathbb {I} _ {C ^ {p}} (\boldsymbol {x}).
$$
Define  $Z_{k} = k^{i}V_{i}(\mathbf{X}_{k})$ . From the above equation, we have
$$
\begin{array}{l} \mathbb {E} \left[ Z _ {k + 1} \mid \boldsymbol {X} _ {k} \right] \leq (k + 1) ^ {i} \left(V _ {i} \left(\boldsymbol {X} _ {k}\right) - \beta_ {i + 1} V _ {i + 1} \left(\boldsymbol {X} _ {k}\right) + b _ {i + 1} \mathbb {I} _ {C ^ {p}} \left(\boldsymbol {X} _ {k}\right)\right) \\ \leq Z _ {k} + 2 ^ {i} (k + 1) ^ {i - 1} V _ {i} \left(\boldsymbol {X} _ {k}\right) + (k + 1) ^ {i} b _ {i + 1} \mathbb {I} _ {C ^ {p}} \left(\boldsymbol {X} _ {k}\right) - (k + 1) ^ {i} \beta_ {i + 1} V _ {i + 1} \left(\boldsymbol {X} _ {k}\right). \\ \end{array}
$$
By applying Lemma 12 to the above equation, we get
$$
\begin{array}{l} \beta_ {i + 1} \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {k = 0} ^ {\tau_ {0} d - 1} (k + 1) ^ {i} V _ {i + 1} (\boldsymbol {X} _ {k}) \right] \\ \leq 2 ^ {i} \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {k = 0} ^ {\tau_ {0 d} - 1} (k + 1) ^ {i - 1} V _ {i} (\boldsymbol {X} _ {k}) \right] + b _ {i + 1} \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {k = 0} ^ {\tau_ {0 d} - 1} (k + 1) ^ {i} \mathbb {I} _ {C ^ {p}} (\boldsymbol {X} _ {k}) \right] \\ \leq 2 ^ {i} \phi (i) \left(V _ {0} (\boldsymbol {x}) + b _ {1} \alpha_ {C ^ {p}}\right) + \alpha_ {C ^ {p}} b _ {i + 1} \mathbb {E} _ {\boldsymbol {x}} \left[ \left(\tau_ {0 ^ {d}}\right) ^ {i} \right], \tag {43} \\ \end{array}
$$
where the second inequality follows from (40) and the induction hypothesis (42). Thereafter, from (42) (by using integral lower bound after using  $V_{i} \geq 1$ ), we have
$$
\frac {1}{i} \mathbb {E} _ {\boldsymbol {x}} \left[ \left(\tau_ {0 ^ {d}}\right) ^ {i} \right] \leq \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {k = 0} ^ {\tau_ {0 ^ {d}} - 1} (k + 1) ^ {i - 1} V _ {i} \left(\boldsymbol {X} _ {k}\right) \right] \leq \phi (i) \left(V _ {0} (\boldsymbol {x}) + b _ {1} \alpha_ {C ^ {p}}\right).
$$
Substituting in (43), we get
$$
\begin{array}{l} \beta_ {i + 1} \mathbb {E} _ {\boldsymbol {x}} \left[ \sum_ {k = 0} ^ {\tau_ {0 ^ {d}} - 1} (k + 1) ^ {i} V _ {i + 1} (\boldsymbol {X} _ {k}) \right] \leq 2 ^ {i} \phi (i) \left(V _ {0} (\boldsymbol {x}) + b _ {1} \alpha_ {C ^ {p}}\right) + i b _ {i + 1} \alpha_ {C ^ {p}} \phi (i) \left(V _ {0} (\boldsymbol {x}) + b _ {1} \alpha_ {C ^ {p}}\right) \\ = \left(2 ^ {i} + i b _ {i + 1} \alpha_ {C ^ {p}}\right) \phi (i) \left(V _ {0} (\boldsymbol {x}) + b _ {1} \alpha_ {C ^ {p}}\right) \\ = \beta_ {i + 1} \phi (i + 1) \left(V _ {0} (\boldsymbol {x}) + b _ {1} \alpha_ {C ^ {p}}\right). \\ \end{array}
$$
D.3 Proof of Lemma 11
Proof. In the proof, to avoid cumbersome notation we will drop the indices  $\theta_{1},\theta_{2}$ . Based on Assumption 3, there exists a finite set  $C^g$ , constants  $b^{g},\gamma^{g}\in (0,1)$ , and a function  $V^{g}:\mathcal{X}\to [1, + \infty)$  satisfying
$$
\Delta V ^ {g} (\boldsymbol {x}) \leq - (1 - \gamma^ {g}) V ^ {g} (\boldsymbol {x}) + b ^ {g} \mathbb {I} _ {C ^ {g}} (\boldsymbol {x}), \quad \boldsymbol {x} \in \mathcal {X}. \tag {44}
$$
For  $n\geq 1$  , define the  $n$  -step taboo probabilities [43] as
where  $A, B \subseteq \mathcal{X}$ , and  $\tau_A$  is the first hitting time of set  $A$ . We also let  ${}_A P_{\boldsymbol{x}B}^0 = \mathbb{I}_B(\boldsymbol{x})$  and  $\tilde{V}^g = \sum_{n=0}^\infty {}_0^{d^n} P^n V^g$ . Applying the last exit decomposition on  $C^g \setminus \{0^d\}$  for all  $x \in \mathcal{X}$ , we obtain
$$
\begin{array}{l} \tilde {V} ^ {g} (\boldsymbol {x}) \\ = \sum_ {n = 0} ^ {\infty} \sum P _ {\boldsymbol {x y}} ^ {n} V ^ {g} (\boldsymbol {y}) \\ = V ^ {g} (\boldsymbol {x}) + \sum_ {n = 1} ^ {\infty} \sum P _ {\boldsymbol {x y}} ^ {n} V ^ {g} (\boldsymbol {y}) \\ + \sum_ {n = 1} ^ {\infty} \sum_ {\boldsymbol {y} \in \mathcal {X}} \sum_ {m = 1} ^ {n - 1} \sum_ {\boldsymbol {z} \in C ^ {g} \backslash \{0 ^ {d} \}} _ {0 ^ {d}} P _ {\boldsymbol {x z}} ^ {m} _ {C ^ {g}} P _ {\boldsymbol {z y}} ^ {n - m} V ^ {g} (\boldsymbol {y}) + \sum_ {n = 1} ^ {\infty} \sum_ {\boldsymbol {y} \in \mathcal {X}} \sum_ {\boldsymbol {z} \in C ^ {g} \backslash \{0 ^ {d} \}} _ {0 ^ {d}} P _ {\boldsymbol {x z}} ^ {n} _ {C ^ {g}} P _ {\boldsymbol {z y}} ^ {0} V ^ {g} (\boldsymbol {y}) \\ = V ^ {g} (\boldsymbol {x}) + \sum_ {n = 1} ^ {\infty} \sum P _ {\boldsymbol {x y}} ^ {n} V ^ {g} (\boldsymbol {y}) (45) \\ + \underbrace {\sum_ {\boldsymbol {y} \in \mathcal {X}} \sum_ {\boldsymbol {z} \in C ^ {g} \backslash \{0 ^ {d} \}} \left(\sum_ {m = 1} ^ {\infty} _ {0 ^ {d}} P _ {\boldsymbol {x z}} ^ {m}\right) \left(\sum_ {n = 1} ^ {\infty} _ {C ^ {g}} P _ {\boldsymbol {z y}} ^ {n} V ^ {g} (\boldsymbol {y})\right)} _ {\text {T e r m 1}} + \underbrace {\sum_ {n = 1} ^ {\infty} \sum_ {\boldsymbol {z} \in C ^ {g} \backslash \{0 ^ {d} \}} _ {0 ^ {d}} P _ {\boldsymbol {x z}} ^ {n} V ^ {g} (\boldsymbol {z})} _ {\text {T e r m 2}}, (46) \\ \end{array}
$$
where we break up the trajectories starting at state  $\pmb{x}$  and reaching state  $\pmb{y}$  while avoiding state  $0^d$  into two: ones that never visit the set  $C^g$ , and the others that visit  $C^g \setminus \{0^d\}$  up until time  $m$  but not afterwards and exit  $C^g \setminus \{0^d\}$  at time  $m$ .
We first bound Term 1 in (46) by finding an upper bound for the probability term  $\sum_{m=1}^{\infty} P_{\boldsymbol{x}\boldsymbol{z}}^{m}$  using the first entrance decomposition on  $C^{g} \setminus \{0^{d}\}$  while noting that  $\boldsymbol{z} \in C^{g} \setminus \{0^{d}\}$ :
$$
\begin{array}{l} \sum_{m = 1}^{\infty}{}_{0^{d}}P_{\boldsymbol {x}\boldsymbol{z}}^{m} = \sum_{m = 1}^{\infty}\sum_{l = 1}^{m}\sum_{\substack{\boldsymbol {u}\in C^{g}\setminus \{0^{d}\} \\ \boldsymbol {v}\not\in C^{g}}}{}_{C^{g}}P_{\boldsymbol {x}\boldsymbol{v}}^{l - 1}P_{\boldsymbol {v}\boldsymbol{u}}{}_{0^{d}}P_{\boldsymbol {u}\boldsymbol{z}}^{m - l} \\ = \sum_ {\boldsymbol {u} \in C ^ {g} \backslash \{0 ^ {d} \}} \left(\sum_ {l = 0} ^ {\infty} \sum_ {\boldsymbol {v} \notin C ^ {g}} P _ {\boldsymbol {x v}} ^ {l} P _ {\boldsymbol {v u}}\right) \left(\sum_ {m = 0} ^ {\infty} \sum_ {0 ^ {d}} P _ {\boldsymbol {u z}} ^ {m}\right) \\ \leq \sum_ {\boldsymbol {u} \in C ^ {g} \backslash \{0 ^ {d} \}} \sum_ {m = 0} ^ {\infty} P _ {\boldsymbol {u z}} ^ {m} \\ \leq \sum_ {\boldsymbol {u} \in C ^ {g} \backslash \{0 ^ {d} \}} \sum_ {m = 0} ^ {\infty} \mathbb {P} _ {\boldsymbol {u}} \left(\tau_ {0 ^ {d}} > m\right) \\ \leq \left| C ^ {g} \right| \max  _ {\boldsymbol {u} \in C ^ {g} \backslash \{0 ^ {d} \}} \mathbb {E} _ {\boldsymbol {u}} \left[ \tau_ {0 ^ {d}} \right], \tag {47} \\ \end{array}
$$
where the third line follows from the fact that  $\sum_{l=0}^{\infty} \sum_{\boldsymbol{v} \notin C^{g}} P_{\boldsymbol{x}\boldsymbol{v}}^{l} P_{\boldsymbol{v}\boldsymbol{u}}$  is the probability of entrance to  $C^{g}$  through  $\boldsymbol{u} \in C^{g} \setminus \{0\}$ , so it is less than 1. Irreducibility and positive recurrence combined with  $|C^{g}| < \infty$  imply that  $\max_{\boldsymbol{u} \in C^{g} \setminus \{0^{d}\}} \mathbb{E}_{\boldsymbol{u}}[\tau_{0^{d}}] < \infty$ , which shows  $\sum_{m=0}^{\infty} P_{\boldsymbol{x}\boldsymbol{z}}^{m}$  is finite. Next,
by induction we prove that for  $n \geq 1$  and  $z \in C^g \setminus \{0^d\}$  we have
$$
\sum P _ {\boldsymbol {z y}} ^ {n} V ^ {g} (\boldsymbol {y}) \leq \left(\gamma^ {g}\right) ^ {n - 1} b ^ {g}. \tag {48}
$$
For  $n = 1$ , we have using Assumption 3 that
$$
\sum P _ {\boldsymbol {z y}} V ^ {g} (\boldsymbol {y}) \leq \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\boldsymbol {z y}} V ^ {g} (\boldsymbol {y}) \leq b ^ {g}.
$$
Assuming that (48) holds for  $n$ , for  $n + 1$  we have
$$
\begin{array}{l} \sum P _ {\boldsymbol {z y}} ^ {n + 1} V ^ {g} (\boldsymbol {y}) \leq \sum_ {\substack {\boldsymbol {y} \in \mathcal {X} \\ \boldsymbol {v} \notin C ^ {g}}} _ {C ^ {g}} P _ {\boldsymbol {z v}} ^ {n} P _ {\boldsymbol {v y}} V ^ {g} (\boldsymbol {y}) \leq \gamma^ {g} \sum P _ {\boldsymbol {z v}} ^ {n} V ^ {g} (\boldsymbol {v}) \quad \text {(U s i n g (4 4))} \\ \leq \gamma \sum_ {\boldsymbol {v} \in \mathcal {X}} C ^ {g} P _ {\boldsymbol {z} \boldsymbol {v}} ^ {n} V ^ {g} (\boldsymbol {v}) \leq (\gamma^ {g}) ^ {n} b ^ {g}, \quad \text {(B y i n d u c t i o n s t e p)} \\ \end{array}
$$
so (48) is shown. We collect these bounds later on for our result on Term 2.
We now simplify the summation in (45). Similar to previous arguments, we will use induction for  $n \geq 1$  and show for all  $x \in \mathcal{X}$
$$
\sum P _ {\boldsymbol {x y}} ^ {n} V ^ {g} (\boldsymbol {y}) \leq \left(\gamma^ {g}\right) ^ {n - 1} \left(\gamma^ {g} V ^ {g} (\boldsymbol {x}) + b ^ {g}\right). \tag {49}
$$
For  $n = 1$ , we have
$$
\sum P _ {\boldsymbol {x y}} V ^ {g} (\boldsymbol {y}) \leq \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\boldsymbol {x y}} V ^ {g} (\boldsymbol {y}) \leq \gamma^ {g} V ^ {g} (\boldsymbol {x}) + b ^ {g}.
$$
Assuming that (49) holds for  $n$ , for  $n + 1$  we have
$$
\begin{array}{l} \sum_ {\boldsymbol {y} \in \mathcal {X}} C ^ {g} P _ {\boldsymbol {x y}} ^ {n + 1} V ^ {g} (\boldsymbol {y}) \leq \sum_ {\boldsymbol {z} \notin C ^ {g}} C ^ {g} P _ {\boldsymbol {x z}} ^ {n} \sum_ {\boldsymbol {y} \in \mathcal {X}} P _ {\boldsymbol {z y}} V ^ {g} (\boldsymbol {y}) \leq \gamma^ {g} \sum_ {\boldsymbol {z} \notin C ^ {g}} C ^ {g} P _ {\boldsymbol {x z}} ^ {n} V ^ {g} (\boldsymbol {z}) \\ \leq \gamma^ {g} \sum P _ {\boldsymbol {x z}} ^ {n} V ^ {g} (\boldsymbol {z}) \leq \left(\gamma^ {g}\right) ^ {n} \left(\gamma^ {g} V ^ {g} (\boldsymbol {x}) + b ^ {g}\right), \\ \end{array}
$$
where the first and second inequalities follow from the definition of taboo probabilities and (44). Thus, (49) is proved. Lastly, for Term 2 in (46), we note
$$
\begin{array}{l} \sum_ {n = 1} ^ {\infty} \sum_ {\boldsymbol {z} \in C ^ {g} \backslash \{0 ^ {d} \}} _ {0 ^ {d}} P _ {\boldsymbol {x z}} ^ {n} V ^ {g} (\boldsymbol {z}) \leq \max  _ {\boldsymbol {y} \in C ^ {g} \backslash \{0 ^ {d} \}} V ^ {g} (\boldsymbol {y}) \sum_ {\boldsymbol {z} \in C ^ {g} \backslash \{0 ^ {d} \}} \sum_ {n = 1} ^ {\infty} _ {0 ^ {d}} P _ {\boldsymbol {x z}} ^ {n} \\ \leq b ^ {g} \left| C ^ {g} \right| ^ {2} \max  _ {\boldsymbol {u} \in C ^ {g} \backslash \{0 ^ {d} \}} \mathbb {E} _ {\boldsymbol {u}} \left[ \tau_ {0 ^ {d}} \right] \quad (\text {F r o m (4 7)}). \\ \end{array}
$$
From the above equation, (47), (48), and (49), we bound  $\tilde{V}^g(\pmb{x})$  as follows:
$$
\begin{array}{l} \tilde {V} ^ {g} (\boldsymbol {x}) \\ \leq V ^ {g} (\boldsymbol {x}) + \left(\gamma^ {g} V ^ {g} (\boldsymbol {x}) + b ^ {g}\right) \sum_ {n = 1} ^ {\infty} \left(\gamma^ {g}\right) ^ {n - 1} + | C ^ {g} | ^ {2} b ^ {g} \max  _ {\boldsymbol {u} \in C ^ {g} \backslash \{0 ^ {d} \}} \mathbb {E} _ {\boldsymbol {u}} \left[ \tau_ {0 ^ {d}} \right] \left(1 + \sum_ {n = 1} ^ {\infty} \left(\gamma^ {g}\right) ^ {n - 1}\right) \\ \leq \frac {V ^ {g} (\boldsymbol {x})}{1 - \gamma^ {g}} + \frac {3 | C ^ {g} | ^ {2} b ^ {g}}{1 - \gamma^ {g}} \max  \left(1, \max  _ {\boldsymbol {u} \in C ^ {g} \backslash \{0 ^ {d} \}} \mathbb {E} _ {\boldsymbol {u}} [ \tau_ {0 ^ {d}} ]\right) \\ \leq V ^ {g} (\boldsymbol {x}) \left(\frac {3 b ^ {g} + 1}{1 - \gamma^ {g}} \left(| C ^ {g} | ^ {2} \max  \left(1, \max  _ {\boldsymbol {u} \in C ^ {g} \backslash \{0 ^ {d} \}} \mathbb {E} _ {\boldsymbol {u}} [ \tau_ {0 ^ {d}} ]\right)\right)\right), \\ \end{array}
$$
where the last line is due to  $V^g(\pmb{x}) \geq 1$ . Taking
$$
\tilde {b} ^ {g} := \frac {3 b ^ {g} + 1}{1 - \gamma^ {g}} \left(| C ^ {g} | ^ {2} \max  \left(1, \max  _ {\boldsymbol {u} \in C ^ {g} \backslash \{0 ^ {d} \}} \mathbb {E} _ {\boldsymbol {u}} [ \tau_ {0 ^ {d}} ]\right)\right) > 1,
$$
we have shown that
$$
\tilde {V} ^ {g} (\boldsymbol {x}) \leq \tilde {b} ^ {g} V ^ {g} (\boldsymbol {x}), \quad \boldsymbol {x} \in \mathcal {X}. \tag {50}
$$