# APPENDIX II
## PROOFS OF SECTION IV-F

### A. Proof of Theorem 5

Let $x \in \mathbb{R}^{n_x}$ and $i \in \mathbb{Z}_{>0}$. From item (i) of Lemma 6 and Bellman equation, it follows $V_r^i(x) - V^*(x) = \ell(x, h_r^{*,i}(x)) + V_r^i(\phi(1, x, h_r^{*,i})) - \ell(x, h^*(x)) - V^*(\phi(1, x, h^*))$, for any $h_r^{*,i} \in H_r^{*,i}$ and $h^* \in H^*$. Moreover, $V_r^i(x) = \ell(x, h_r^{*,i}(x)) + V_r^i(\phi(1, x, h_r^{*,i})) \le \ell(x, h^*(x)) + V_r^i(f(x, h^*(x))) \le \ell(x, u) + V_r^{i-1}(f(x, u))$ for any $h^* \in H^*$ and $u \in \mathcal{U}(x)$ as consequence of (PI$^+$.2), (PI$^+$.5) and item (ii) of Lemma 6. We derive $V_r^i(x) - V^*(x) \le \ell(x, u) + V_r^{i-1}(f(x, u)) - \ell(x, h^*(x)) - V^*(\phi(1, x, h^*)) \le \ell(x, h^*(x)) + V_r^{i-1}(\phi(1, x, h^*)) - \ell(x, h^*(x)) - V^*(\phi(1, x, h^*)) = V_r^{i-1}(\phi(1, x, h^*)) - V^*(\phi(1, x, h^*))$. We repeat the above reasoning $i-1$ times, hence $V_r^i(x) - V^*(x) \le V_r^0(\phi(i, x, h^*)) - V^*(\phi(i, x, h^*))$. The first part of Theorem 5 is obtained. We now show (16) holds and we distinguish two cases as in the proof of Lemma 2 in Appendix I-C.

**Case where $\chi_W \le I$:** For any $z \in \mathbb{R}^{n_x}$, $V_r^0(z) - V^*(z) = V_r^0(z) - V^*(z) \pm W(z) = Y^0(z) - Y^*(z)$, where $Y^* = V^* + W$ defined in Section V-B, in view of the proof of Lemma 3 in Appendix I-C. Moreover that $V_r^0(z) - V^*(z) \le V_r^0(z)$ since $V^*(z) \ge 0$. Hence $V_r^0(z) - V^*(z) \le \min\{V_r^0(z), Y^0(z) - Y^*(z)\}$. On the one hand, $V_r^0(z) \le \bar{\alpha}_V(\sigma(z))$ for any $z \in \mathbb{R}^{n_x}$ from SA3, and on the other hand, $\underline{\alpha}_Y(\sigma(z)) \le Y^*(z)$ moreover $Y^0(z) \le \bar{\alpha}_Y(\sigma(z))$, $Y^0(z) - Y^*(z) \le \bar{\alpha}_Y(\sigma(z)) - \underline{\alpha}_Y(\sigma(z))$. Therefore $V_r^0(z) - V^*(z) \le \min\{V_r^0(z), Y^0(z) - Y^*(z)\} \le \min\{\bar{\alpha}_V(\sigma(z)), \bar{\alpha}_Y(\sigma(z)) - \underline{\alpha}_Y(\sigma(z))\} =: \hat{\alpha}(\sigma(z))$, where $\hat{\alpha} \circ \sigma$ is continuous and positive semidefinite but not necessarily non-decreasing. Thus, we construct $\tilde{\alpha}(s) := \max_{\hat{s} \in [0, s]} \hat{\alpha}(\hat{s})$ and we obtain suitable $\tilde{\alpha}$ continuous, positive definite and non-decreasing such that $V_r^0(z) - V^*(z) \le \tilde{\alpha}(\sigma(z))$, and this concludes this case.

**Case where $\chi(s) > s$ for some $s > 0$:** For any $z \in \mathbb{R}^{n_x}$, since $V^*(z) \ge \ell(z, u)$ for any $u \in \mathcal{U}(x)$ and $V_r^0(z) \le \bar{\alpha}_V(\sigma(z))$ for any $z \in \mathbb{R}^{n_x}$ from SA3, $V_r^0(z) - V^*(z) \le \bar{\alpha}_V(\sigma(z)) - \ell(z, u)$. From SA2 and $\ell(z, u) \ge 0$ for any $(z, u) \in \mathcal{W}$, we have $\max\{0, \alpha_W(\sigma(z)) - \bar{\alpha}_W(\sigma(z))\} \le \chi_W(\ell(z, u))$ hence $-\ell(z, u) \le -\chi_W^{-1}(\max\{0, \alpha_W(\sigma(z)) - \bar{\alpha}_W(\sigma(z))\}) =: -\tilde{\alpha}_W(\sigma(z))$, where $\tilde{\alpha}_W \circ \sigma$ is continuous and positive semidefinite but not necessarily class $K_\infty$. Nevertheless, $V_r^0(z) - V^*(z) \le \bar{\alpha}_V(\sigma(z)) - \ell(z, u) \le \bar{\alpha}_V(\sigma(z)) - \tilde{\alpha}_W(\sigma(z)) =: \hat{\alpha}(\sigma(z))$. As done above, we construct $\tilde{\alpha}(s) := \max_{\hat{s} \in [0, s]} \hat{\alpha}(\hat{s})$ and we obtain suitable $\tilde{\alpha}$ continuous, positive definite and non-decreasing such that $V_r^0(z) - V^*(z) \le \tilde{\alpha}(\sigma(z))$, and this concludes this case.

All that remains is to show that $\sigma(\phi(i, x, h^*)) \le \beta(\sigma(x), i)$ holds for any $x \in \mathbb{R}^{n_x}$ and $i \in \mathbb{Z}_{\ge 0}$. To see this, it suffices to note that, in view of SA1 and SA3, $V^* \le V^0 \le \bar{\alpha} \circ \sigma$, and then to follow the same steps as in the proof of Lemma 2 in Appendix I-C and the proof of Lemma 3 in Appendix I-D for $h^* \in H^*$ as in (4) instead of $H_r^{*,i+1}$. Then, since $\tilde{\alpha}$ is non-decreasing, $V_r^i(x) - V^*(x) \le \tilde{\alpha}(\beta(\sigma(x), i))$ and this concludes the proof.

### B. Proof of Proposition 2

Item (i) of Proposition 2 holds in view of item (ii) of Lemma 6. We now focus on item (ii) of Proposition 2. Let $\delta > 0$ and $K \subset \mathbb{R}^{n_x}$ compact. Let $\Delta > 0$ sufficiently large such that $\{z \in \mathbb{R}^{n_x} : \sigma(z) \le \Delta\} \supseteq K$ and let $x \in K$, thus $\sigma(x) \le \Delta$. Since $\beta \in \mathcal{KL}$ and $\tilde{\alpha}$ is continuous and zero at zero, there exists $i^*$ sufficiently large that $\tilde{\alpha}(\beta(\Delta, i^*)) \le \delta$. As $\tilde{\alpha}$ is non-decreasing and $\beta \in \mathcal{KL}$, $\tilde{\alpha}(\beta(\sigma(x), i)) \le \tilde{\alpha}(\beta(\Delta, i)) \le \tilde{\alpha}(\beta(\Delta, i^*)) \le \delta$ for $i \ge i^*$, and the proof is finished by invoking (16).

# APPENDIX III
## SKETCH OF PROOFS OF SECTION III

In view of Assumption 1, we do not require the interplay between stability and feasibility as in Proposition 4 in Appendix I. Hence, to derive stability properties as in Theorem 1, it suffices to show that: (i) for all $i \in \mathbb{Z}_{\ge 0}$, $x \in \mathbb{R}^{n_x}$ and any $h^{i+1} \in H^{i+1}$, $J(x, h^{i+1}) \le V^i(x)$, which is standard in PI literature, see e.g., [28, Section 4.2], and is akin to Lemma 6 for PI$^+$; (ii) invoke $V^0(x) \le \bar{\alpha}_V \circ \sigma(x)$ for any $x \in \mathbb{R}^{n_x}$ by SA3 and then to follow the Lyapunov arguments made in the proof of Lemmas 2 and 3. On the other hand, for near-optimality of PI in Theorem 2 and Proposition 1, it suffices to follow the proofs in Appendix II with $V^i$ in place of $V_r^i$.

# APPENDIX IV
## PROOFS OF SECTION V

### A. Proof of Theorem 6

As explained in Section V-A, we will invoke [21, Theorem 2.8] to obtain robust $\mathcal{KL}$-stability with respect to some measures $(\sigma_1, \sigma_2)$ on $\mathcal{X} \subseteq \mathbb{R}^{n_x}$. The first requirement, regarding the compactness of $F_r^i(x)$ for all $x \in \mathcal{X}$ is established in Lemma 1. All that remains is to show that $Y^* = \rho_V(V^*) + \rho_W(W)$ is continuous and is a suitable Lyapunov function for the considered stability property of system (13) with respect to measures $(\sigma_1, \sigma_2)$ on $\mathcal{X} \subseteq \mathbb{R}^{n_x}$.

First, we show in the next lemma that $V^*$ satisfies a useful dissipation inequality along (13).

**Lemma 10:** For any $i \in \mathbb{Z}_{\ge 0}$, $x \in \mathbb{R}^{n_x}$, $v \in F_r^i(x)$, $h_r^{*,i} \in H_r^{*,i}$ such that $f(x, h_r^{*,i}(x)) = v$ and $h^* \in H^*$, $V^*(v) - V^*(x) \le -\ell(x, h_r^{*,i}(x)) + (V_r^0 - V^*)(\phi(i, x, h^*))$. $\square$

**Proof:** Let $i \in \mathbb{Z}_{\ge 0}$, $x \in \mathbb{R}^{n_x}$, $v \in F_r^i(x)$ and $h_r^{*,i} \in H_r^{*,i}$ such that $f(x, h_r^{*,i}(x)) = v$. Since $V_r^i(x) = \ell(x, h_r^{*,i}(x)) + V_r^i(v)$ and $V^* \le V_r^i$ for all $i \in \mathbb{Z}_{>0}$ by definition of $V^*$ in (3), $V_r^i(x) \ge \ell(x, h_r^{*,i}(x)) + V^*(v)$ holds. In view of (15), $V_r^i(x) - V^*(x) \le (V_r^0 - V^*)(\phi(i, x, h^*))$, and the desired result holds by combining the previous inequalities. $\blacksquare$

We then follow similar steps as in the proof of Lemma 7 to derive there exists $i^* \in \mathbb{Z}_{\ge 0}$ such that, for any $i > i^*$, $\sigma_1(\phi^i(k, x)) = \sigma(\phi^i(k, x)) - \delta \le \max\{\hat{\beta}(k, \sigma(x)) - \delta, 0\} \le \hat{\beta}(k, \sigma_2(x))$ holds for any $x \in \mathcal{X} := \{z \in \mathbb{R}^{n_x} : \sigma(z) < \Delta\}$. The Lyapunov function used to establish this stability property is $Y^*$, which is continuous by Assumption 3 and following similar steps as [21, Theorem 6]. As $F_r^i$ is non-empty and compact on $\mathcal{X}$ by Lemma 1, we invoke [21, Theorem 2.8] to establish the desired result.
# APPENDIX II
## PROOFS OF SECTION IV-F

### A. Proof of Theorem 5

Let $x \in \mathbb{R}^{n_x}$ and $i \in \mathbb{Z}_{>0}$. From item (i) of Lemma 6 and Bellman equation, it follows $V_r^i(x) - V^*(x) = \ell(x, h_r^{*,i}(x)) + V_r^i(\phi(1, x, h_r^{*,i})) - \ell(x, h^*(x)) - V^*(\phi(1, x, h^*))$, for any $h_r^{*,i} \in H_r^{*,i}$ and $h^* \in H^*$. Moreover, $V_r^i(x) = \ell(x, h_r^{*,i}(x)) + V_r^i(\phi(1, x, h_r^{*,i})) \le \ell(x, h^*(x)) + V_r^i(f(x, h^*(x))) \le \ell(x, u) + V_r^{i-1}(f(x, u))$ for any $h^* \in H^*$ and $u \in \mathcal{U}(x)$ as consequence of (PI$^+$.2), (PI$^+$.5) and item (ii) of Lemma 6. We derive $V_r^i(x) - V^*(x) \le \ell(x, u) + V_r^{i-1}(f(x, u)) - \ell(x, h^*(x)) - V^*(\phi(1, x, h^*)) \le \ell(x, h^*(x)) + V_r^{i-1}(\phi(1, x, h^*)) - \ell(x, h^*(x)) - V^*(\phi(1, x, h^*)) = V_r^{i-1}(\phi(1, x, h^*)) - V^*(\phi(1, x, h^*))$. We repeat the above reasoning $i-1$ times, hence $V_r^i(x) - V^*(x) \le V_r^0(\phi(i, x, h^*)) - V^*(\phi(i, x, h^*))$. The first part of Theorem 5 is obtained. We now show (16) holds and we distinguish two cases as in the proof of Lemma 2 in Appendix I-C.

**Case where $\chi_W \le I$:** For any $z \in \mathbb{R}^{n_x}$, $V_r^0(z) - V^*(z) = V_r^0(z) - V^*(z) \pm W(z) = Y^0(z) - Y^*(z)$, where $Y^* = V^* + W$ defined in Section V-B, in view of the proof of Lemma 3 in Appendix I-C. Moreover that $V_r^0(z) - V^*(z) \le V_r^0(z)$ since $V^*(z) \ge 0$. Hence $V_r^0(z) - V^*(z) \le \min\{V_r^0(z), Y^0(z) - Y^*(z)\}$. On the one hand, $V_r^0(z) \le \bar{\alpha}_V(\sigma(z))$ for any $z \in \mathbb{R}^{n_x}$ from SA3, and on the other hand, $\underline{\alpha}_Y(\sigma(z)) \le Y^*(z)$ moreover $Y^0(z) \le \bar{\alpha}_Y(\sigma(z))$, $Y^0(z) - Y^*(z) \le \bar{\alpha}_Y(\sigma(z)) - \underline{\alpha}_Y(\sigma(z))$. Therefore $V_r^0(z) - V^*(z) \le \min\{V_r^0(z), Y^0(z) - Y^*(z)\} \le \min\{\bar{\alpha}_V(\sigma(z)), \bar{\alpha}_Y(\sigma(z)) - \underline{\alpha}_Y(\sigma(z))\} =: \hat{\alpha}(\sigma(z))$, where $\hat{\alpha} \circ \sigma$ is continuous and positive semidefinite but not necessarily non-decreasing. Thus, we construct $\tilde{\alpha}(s) := \max_{\hat{s} \in [0, s]} \hat{\alpha}(\hat{s})$ and we obtain suitable $\tilde{\alpha}$ continuous, positive definite and non-decreasing such that $V_r^0(z) - V^*(z) \le \tilde{\alpha}(\sigma(z))$, and this concludes this case.

**Case where $\chi(s) > s$ for some $s > 0$:** For any $z \in \mathbb{R}^{n_x}$, since $V^*(z) \ge \ell(z, u)$ for any $u \in \mathcal{U}(x)$ and $V_r^0(z) \le \bar{\alpha}_V(\sigma(z))$ for any $z \in \mathbb{R}^{n_x}$ from SA3, $V_r^0(z) - V^*(z) \le \bar{\alpha}_V(\sigma(z)) - \ell(z, u)$. From SA2 and $\ell(z, u) \ge 0$ for any $(z, u) \in \mathcal{W}$, we have $\max\{0, \alpha_W(\sigma(z)) - \bar{\alpha}_W(\sigma(z))\} \le \chi_W(\ell(z, u))$ hence $-\ell(z, u) \le -\chi_W^{-1}(\max\{0, \alpha_W(\sigma(z)) - \bar{\alpha}_W(\sigma(z))\}) =: -\tilde{\alpha}_W(\sigma(z))$, where $\tilde{\alpha}_W \circ \sigma$ is continuous and positive semidefinite but not necessarily class $\mathcal{K}_\infty$. Nevertheless, $V_r^0(z) - V^*(z) \le \bar{\alpha}_V(\sigma(z)) - \ell(z, u) \le \bar{\alpha}_V(\sigma(z)) - \tilde{\alpha}_W(\sigma(z)) =: \hat{\alpha}(\sigma(z))$. As done above, we construct $\tilde{\alpha}(s) := \max_{\hat{s} \in [0, s]} \hat{\alpha}(\hat{s})$ and we obtain suitable $\tilde{\alpha}$ continuous, positive definite and non-decreasing such that $V_r^0(z) - V^*(z) \le \tilde{\alpha}(\sigma(z))$, and this concludes this case.

All that remains is to show that $\sigma(\phi(i, x, h^*)) \le \beta(\sigma(x), i)$ holds for any $x \in \mathbb{R}^{n_x}$ and $i \in \mathbb{Z}_{\ge 0}$. To see this, it suffices to note that, in view of SA1 and SA3, $V^* \le V^0 \le \bar{\alpha} \circ \sigma$, and then to follow the same steps as in the proof of Lemma 2 in Appendix I-C and the proof of Lemma 3 in Appendix I-D for $h^* \in H^*$ as in (4) instead of $H_r^{*,i+1}$. Then, since $\tilde{\alpha}$ is non-decreasing, $V_r^i(x) - V^*(x) \le \tilde{\alpha}(\beta(\sigma(x), i))$ and this concludes the proof.

### B. Proof of Proposition 2

Item (i) of Proposition 2 holds in view of item (ii) of Lemma 6. We now focus on item (ii) of Proposition 2. Let $\delta > 0$ and $K \subset \mathbb{R}^{n_x}$ compact. Let $\Delta > 0$ sufficiently large such that $\{z \in \mathbb{R}^{n_x} : \sigma(z) \le \Delta\} \supseteq K$ and let $x \in K$, thus $\sigma(x) \le \Delta$. Since $\beta \in \mathcal{KL}$ and $\tilde{\alpha}$ is continuous and zero at zero, there exists $i^*$ sufficiently large that $\tilde{\alpha}(\beta(\Delta, i^*)) \le \delta$. As $\tilde{\alpha}$ is non-decreasing and $\beta \in \mathcal{KL}$, $\tilde{\alpha}(\beta(\sigma(x), i)) \le \tilde{\alpha}(\beta(\Delta, i)) \le \tilde{\alpha}(\beta(\Delta, i^*)) \le \delta$ for $i \ge i^*$, and the proof is finished by invoking (16).

# APPENDIX III
## SKETCH OF PROOFS OF SECTION III

In view of Assumption 1, we do not require the interplay between stability and feasibility as in Proposition 4 in Appendix I. Hence, to derive stability properties as in Theorem 1, it suffices to show that: (i) for all $i \in \mathbb{Z}_{\ge 0}$, $x \in \mathbb{R}^{n_x}$ and any $h^{i+1} \in H^{i+1}$, $J(x, h^{i+1}) \le V^i(x)$, which is standard in PI literature, see e.g., [28, Section 4.2], and is akin to Lemma 6 for PI$^+$; (ii) invoke $V^0(x) \le \bar{\alpha}_V \circ \sigma(x)$ for any $x \in \mathbb{R}^{n_x}$ by SA3 and then to follow the Lyapunov arguments made in the proof of Lemmas 2 and 3. On the other hand, for near-optimality of PI in Theorem 2 and Proposition 1, it suffices to follow the proofs in Appendix II with $V^i$ in place of $V_r^i$.

# APPENDIX IV
## PROOFS OF SECTION V

### A. Proof of Theorem 6

As explained in Section V-A, we will invoke [21, Theorem 2.8] to obtain robust $\mathcal{KL}$-stability with respect to some measures $(\sigma_1, \sigma_2)$ on $\mathcal{X} \subseteq \mathbb{R}^{n_x}$. The first requirement, regarding the compactness of $F_r^i(x)$ for all $x \in \mathcal{X}$ is established in Lemma 1. All that remains is to show that $Y^* = \rho_V(V^*) + \rho_W(W)$ is continuous and is a suitable Lyapunov function for the considered stability property of system (13) with respect to measures $(\sigma_1, \sigma_2)$ on $\mathcal{X} \subseteq \mathbb{R}^{n_x}$.

First, we show in the next lemma that $V^*$ satisfies a useful dissipation inequality along (13).

**Lemma 10:** For any $i \in \mathbb{Z}_{\ge 0}$, $x \in \mathbb{R}^{n_x}$, $v \in F_r^i(x)$, $h_r^{*,i} \in H_r^{*,i}$ such that $f(x, h_r^{*,i}(x)) = v$ and $h^* \in H^*$, $V^*(v) - V^*(x) \le -\ell(x, h_r^{*,i}(x)) + (V_r^0 - V^*)(\phi(i, x, h^*))$. $\square$

**Proof:** Let $i \in \mathbb{Z}_{\ge 0}$, $x \in \mathbb{R}^{n_x}$, $v \in F_r^i(x)$ and $h_r^{*,i} \in H_r^{*,i}$ such that $f(x, h_r^{*,i}(x)) = v$. Since $V_r^i(x) = \ell(x, h_r^{*,i}(x)) + V_r^i(v)$ and $V^* \le V_r^i$ for all $i \in \mathbb{Z}_{>0}$ by definition of $V^*$ in (3), $V_r^i(x) \ge \ell(x, h_r^{*,i}(x)) + V^*(v)$ holds. In view of (15), $V_r^i(x) - V^*(x) \le (V_r^0 - V^*)(\phi(i, x, h^*))$, and the desired result holds by combining the previous inequalities. $\blacksquare$

We then follow similar steps as in the proof of Lemma 7 to derive there exists $i^* \in \mathbb{Z}_{\ge 0}$ such that, for any $i > i^*$, $\sigma_1(\phi^i(k, x)) = \sigma(\phi^i(k, x)) - \delta \le \max\{\hat{\beta}(k, \sigma(x)) - \delta, 0\} \le \hat{\beta}(k, \sigma_2(x))$ holds for any $x \in \mathcal{X} := \{z \in \mathbb{R}^{n_x} : \sigma(z) < \Delta\}$. The Lyapunov function used to establish this stability property is $Y^*$, which is continuous by Assumption 3 and following similar steps as [21, Theorem 6]. As $F_r^i$ is non-empty and compact on $\mathcal{X}$ by Lemma 1, we invoke [21, Theorem 2.8] to establish the desired result.
## B. Proof of Proposition 3

We first show that, under Assumption 4, when $V_r^i$ is continuous for some $i \in \mathbb{Z}_{\ge 0}$, $V_r^{i+1}$ is continuous. Since $V_r^0$ is continuous, the desired result then follows by induction.

**Lemma 11:** When Assumption 4 holds, $V_r^i$ continuous on $\mathbb{R}^{n_x}$ for $i \in \mathbb{Z}_{\ge 0}$ implies $V_r^{i+1}$ continuous on $\mathbb{R}^{n_x}$. $\square$

**Proof:** In light of Lemma 9, all that remains to be shown is that $V_r^{i+1}$ is upper semicontinuous on $\mathbb{R}^{n_x}$ given $V_r^i$ continuous on $\mathbb{R}^{n_x}$ and Assumption 4. Hence, we extend here the proof of Lemma 9 in Appendix I-G to show, for any $\varepsilon > 0$, for any $x \in \mathbb{R}^{n_x}$ and sequence $x_n \to x$, there exists $N \in \mathbb{Z}_{>0}$, for any $n \ge N$ such that $V^{i+1}(x) \ge V^{i+1}(x_n) - \varepsilon$.

Let $x \in \mathbb{R}^{n_x}$, $\varepsilon \in \mathbb{R}_{\ge 0}$ and $n$ sufficiently large such that any invoked inequalities of the proof of Lemma 9 in Appendix I-G hold in the sequel. Given the continuity of $V_r^i$, we have that $H^{i+1}$ is outer semicontinuous in view of [27, Theorem 7.41(b)], hence $H^{i+1} = H_r^{i+1}$, and in view of Assumption 4 we have $H^{i+1} = H_r^{i+1} = H_r^{*,i+1}$. Thus, for any two solutions $\psi, \tilde{\psi} \in S^{i+1}(z)$, $\sum_{k=0}^{\infty} \ell(\psi(k, z)) = \sum_{k=0}^{\infty} \ell(\tilde{\psi}(k, z)) = V_r^{i+1}(z)$ for all $z \in \mathbb{R}^{n_x}$, that is, all solutions are optimal. Therefore, we have $V^{i+1}(x) = \sum_{k=0}^{\infty} \ell(\hat{\psi}_{i+1,n}(k, x))$ for $\hat{\psi}_{i+1,n} = (\hat{\phi}_{i+1,n}, \hat{v}_{i+1,n}) \in S^{i+1}(x)$ as defined in Appendix I-G. From (34), we deduce that $V^{i+1}(x) \ge \sum_{k=0}^{\bar{N}-1} \ell(\hat{\psi}_{i+1,n}(k, x)) \ge \sum_{k=0}^{\bar{N}-1} \ell(\psi_{i+1,n}(k, x_n)) - \frac{\varepsilon}{2}$ for $\psi_{i+1,n} = (\phi_{i+1,n}, v_{i+1,n}) \in S^{i+1}(x_n)$ as defined in Appendix I-G. By adding and subtracting $\sum_{k=\bar{N}}^{\infty} \ell(\psi_{i+1,n}(k, x_n))$ to the previous inequality and in view of the optimality of $\psi_{i+1,n}$, $V^{i+1}(x) \ge \sum_{k=0}^{\infty} \ell(\psi_{i+1,n}(k, x_n)) - \frac{\varepsilon}{2} - \sum_{k=\bar{N}}^{\infty} \ell(\psi_{i+1,n}(k, x_n)) = V^{i+1}(x_n) - \frac{\varepsilon}{2} - \sum_{k=\bar{N}}^{\infty} \ell(\psi_{i+1,n}(k, x_n)).$ Again, in view of the optimality of $\psi_{i+1,n}$, $V^{i+1}(\phi_{i+1,n}(\bar{N}, x_n)) = \sum_{k=\bar{N}}^{\infty} \ell(\psi_{i+1,n}(k, x_n)).$ Hence it follows from the previous inequality that

$$V^{i+1}(x) \ge V^{i+1}(x_n) - \varepsilon/2 - V^{i+1}(\phi_{i+1,n}(\bar{N}, x_n)). \quad (43)$$

The following inequality is deduced from $V^{i+1}(z) \le \bar{\alpha}_V(\sigma(z))$ for any $z \in \mathbb{R}^{n_x}$ in view of Lemma 6, SA3, (35), and optimality of $\psi_{i+1,n}$, $V^{i+1}(\phi_{i+1,n}(\bar{N}, x_n)) \le \bar{\alpha}_V(\sigma(\phi_{i+1,n}(\bar{N}, x_n))) \le \bar{\alpha}_V(\sigma(\hat{\phi}_{i+1,n}(\bar{N}, x)) + \varepsilon_1)$. Given $n \in \mathbb{Z}_{\ge 0}$ sufficiently big so that (32) holds, by proceeding like in the end of the proof of Lemma 9, we obtain $V^{i+1}(\phi_{i+1,n}(\bar{N}, x_n)) \le 2\bar{\alpha}_V(2\varepsilon_1)$. By definition of $\varepsilon_1$ in (31) we deduce $V^{i+1}(\phi_{i+1,n}(\bar{N}, x_n)) \le \varepsilon/2$. Finally, by (43), $V^{i+1}(x) \ge V^{i+1}(x_n) - \varepsilon$. Hence, $V^{i+1}$ is upper semicontinuous at $x$. We have that $V^{i+1}$ is continuous at arbitrary $x$, it is thus continuous on $\mathbb{R}^{n_x}$.

# APPENDIX V
## TECHNICAL LEMMAS

**Lemma 12:** Given an outer-semicontinuous set-valued map $H: \mathbb{R}^n \rightrightarrows \mathbb{R}^m$ with $n, m \in \mathbb{Z}_{>0}$, the function $(x, u) \mapsto \delta_{H(x)}(u)$ is lower semicontinuous on $\mathbb{R}^n \times \mathbb{R}^m$. $\square$

**Proof:** Let $x \in \mathbb{R}^n$ and $u \in \mathbb{R}^m$. To prove that $\delta_{H(\cdot)}(\cdot)$ is lower semicontinuous at $(x, u)$, we show that $\liminf_{(\tilde{x}, \tilde{u}) \to (x, u)} \delta_{H(\tilde{x})}(\tilde{u}) \ge \delta_{H(x)}(u)$, see [27, Definition 1.5]. This is equivalent to $\min\{\alpha \in \mathbb{R} : \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\} \ge \delta_{H(x)}(u)$, as $\liminf_{(\tilde{x}, \tilde{u}) \to (x, u)} \delta_{H(\tilde{x})}(\tilde{u}) = \min\{\alpha \in \mathbb{R} : \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\}$ by [27, Lemma 1.7]. For what follows, we employ the fact that both $\min\{\alpha \in \mathbb{R} : \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\}$ and $\delta_{H(x)}(u)$ only take values in $\{0, \infty\}$.

We distinguish two cases. First, when $\min\{\alpha \in \mathbb{R} : \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\} = \infty$. The desired result holds. Second, when $\min\{\alpha \in \mathbb{R} : \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\} = 0$. This implies that there exists a (sub)sequence $(x_n, u_n) \to (x, u)$ such that $u_n \in H(x_n)$ as $\delta_{H(x_n)}(u_n) \to 0$. Hence, since $H$ is outer semicontinuous, we have $u \in H(x)$. Therefore, $\delta_{H(x)}(u) = 0$, and $\min\{\alpha \in \mathbb{R} : \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\} = 0 = \delta_{H(x)}(u)$. We have proved that $\delta_{H(\cdot)}(\cdot)$ is lower semicontinuous at arbitrary $(x, u)$, it is therefore lower semicontinuous on $\mathbb{R}^n \times \mathbb{R}^m$. $\blacksquare$

**Lemma 13:** Let $n, m \in \mathbb{Z}_{>0}$ and $g, \ell: \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}_{\ge 0}$ with $g(x, u) \ge \ell(x, u)$ for any $(x, u) \in \mathbb{R}^n \times \mathbb{R}^m$. If $\ell$ is level-bounded in $u$, locally uniform in $x$, then so is $g$. $\square$

**Proof:** Let $x \in \mathbb{R}^n$, $\alpha \in \mathbb{R}$, $S$ be a neighbourhood of $x$ and $B \subset \mathbb{R}^m$ be bounded such that $\{u \in \mathbb{R}^m : \ell(z, u) \le \alpha\} \subset B$ for any $z \in S$, which exist by item (ii) of Assumption 2, see Definition 1. Necessarily $\{u \in \mathbb{R}^m : g(z, u) \le \alpha\} \subset \{u \in \mathbb{R}^m : \ell(z, u) \le \alpha\}$ for $z \in S$, as $\ell(z, u) \le g(z, u)$, hence $\{u \in \mathbb{R}^m : g(z, u) \le \alpha\} \subset B$ for any $z \in S$. As $x$ and $\alpha$ have been arbitrarily, the desired result holds. $\blacksquare$

**Lemma 14:** Let $\alpha: \mathbb{R}_{\ge 0} \to \mathbb{R}_{\ge 0}$ be continuous and $\{s_k\}_{k \in \mathbb{Z}_{\ge 0}}$ with $0 \le s_{k+1} \le \alpha(s_k)$ and $s_k \ge 0$. Then $s_k \le \tilde{\beta}(s_0, k) := \tilde{\alpha}^{(k)}(s_0)$ with $\tilde{\alpha}(s_0) := \max_{s \in [0, s_0]} \alpha(s)$. Moreover, when $\alpha(s) < s$ for all $s \in \mathbb{R}_{>0}$, $\tilde{\beta} \in \mathcal{KL}$. $\square$

**Proof:** Note that $\tilde{\alpha}(s_0) < \infty$ for any $s_0 \ge 0$ as $\alpha$ is continuous. Moreover, $\tilde{\alpha}$ is non-decreasing in view of its definition. We show that $s_k \le \tilde{\alpha}^{(k)}(s_0)$ for all $k \in \mathbb{Z}_{\ge 0}$ by induction. This property is true for $k=0$ as $s_0 \le \tilde{\alpha}^{(0)}(s_0) = s_0$. Let $k \in \mathbb{Z}_{\ge 0}$ and assume that $s_k \le \tilde{\alpha}^{(k)}(s_0)$. By combining $s_{k+1} \le \alpha(s_k)$, $\alpha(s) \le \tilde{\alpha}(s)$ for all $s \ge a$, $\tilde{\alpha}$ non-decreasing and the assumed upper-bound for $s_k$, we obtain $s_{k+1} \le \alpha(s_k) \le \tilde{\alpha}(s_k) \le \tilde{\alpha}(\tilde{\alpha}^{(k)}(s_0)) = \tilde{\alpha}^{(k+1)}(s_0)$. The induction is complete. Now we establish $\tilde{\beta} \in \mathcal{KL}$ when $\alpha(s) < s$ for all $s > 0$. First, it is clear that $\tilde{\alpha}^{(k)}$ is continuous and zero at zero for any $k \in \mathbb{Z}_{\ge 0}$, hence $\tilde{\beta}(\cdot, k)$ is continuous, zero at zero and non-decreasing for any $k \in \mathbb{Z}_{\ge 0}$, thus $\tilde{\beta}$ is class-$\mathcal{K}$ in its first argument. Moreover, as $\alpha(s) < s$ for all $s > 0$ and $\alpha(0) = 0$, we have $\tilde{\beta}(s, k+1) < \tilde{\beta}(s, k)$ when $\tilde{\beta}(s, k) > 0$ and $\tilde{\beta}(s, k+1) = \tilde{\beta}(s, k) = 0$ otherwise. Therefore $\tilde{\beta}(s, \cdot)$ monotonically decreases to zero. $\blacksquare$

<footnote>[1] F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause. Safe model-based reinforcement learning with stability guarantees. In *Adv. Neural Inf. Process Syst.*, pages 908–918, 2017.</footnote>

<footnote>[2] D.P. Bertsekas. *Dynamic Programming and Optimal Control*, volume 2. Athena Scientific, Belmont, U.S.A., 4th edition, 2012.</footnote>

<footnote>[3] D.P. Bertsekas. Value and policy iterations in optimal control and adaptive dynamic programming. *IEEE Trans. Neural Netw. Learn. Syst.*, 28(3):500–509, 2015.</footnote>

<footnote>[4] D.P. Bertsekas. *Reinforcement Learning and Optimal Control*. Athena Scientific, 2019.</footnote>

<footnote>[5] T. Bian, Y. Jiang, and Z.-P. Jiang. Adaptive dynamic programming and optimal control of nonlinear nonaffine systems. *Automatica*, 50(10):2624–2632, 2014.</footnote>
## B. Proof of Proposition 3

We first show that, under Assumption 4, when $V_r^i$ is continuous for some $i \in \mathbb{Z}_{\ge 0}$, $V_r^{i+1}$ is continuous. Since $V_r^0$ is continuous, the desired result then follows by induction.

**Lemma 11:** When Assumption 4 holds, $V_r^i$ continuous on $\mathbb{R}^{n_x}$ for $i \in \mathbb{Z}_{\ge 0}$ implies $V_r^{i+1}$ continuous on $\mathbb{R}^{n_x}$. $\square$

**Proof:** In light of Lemma 9, all that remains to be shown is that $V_r^{i+1}$ is upper semicontinuous on $\mathbb{R}^{n_x}$ given $V_r^i$ continuous on $\mathbb{R}^{n_x}$ and Assumption 4. Hence, we extend here the proof of Lemma 9 in Appendix I-G to show, for any $\varepsilon > 0$, for any $x \in \mathbb{R}^{n_x}$ and sequence $x_n \to x$, there exists $N \in \mathbb{Z}_{>0}$, for any $n \ge N$ such that $V^{i+1}(x) \ge V^{i+1}(x_n) - \varepsilon$.

Let $x \in \mathbb{R}^{n_x}$, $\varepsilon \in \mathbb{R}_{\ge 0}$ and $n$ sufficiently large such that any invoked inequalities of the proof of Lemma 9 in Appendix I-G hold in the sequel. Given the continuity of $V_r^i$, we have that $H^{i+1}$ is outer semicontinuous in view of [27, Theorem 7.41(b)], hence $H^{i+1} = H_r^{i+1}$, and in view of Assumption 4 we have $H^{i+1} = H_r^{i+1} = H_r^{*,i+1}$. Thus, for any two solutions $\psi, \tilde{\psi} \in S^{i+1}(z)$, $\sum_{k=0}^{\infty} \ell(\psi(k, z)) = \sum_{k=0}^{\infty} \ell(\tilde{\psi}(k, z)) = V_r^{i+1}(z)$ for all $z \in \mathbb{R}^{n_x}$, that is, all solutions are optimal. Therefore, we have $V^{i+1}(x) = \sum_{k=0}^{\infty} \ell(\hat{\psi}_{i+1,n}(k, x))$ for $\hat{\psi}_{i+1,n} = (\hat{\phi}_{i+1,n}, \hat{v}_{i+1,n}) \in S^{i+1}(x)$ as defined in Appendix I-G. From (34), we deduce that $V^{i+1}(x) \ge \sum_{k=0}^{\bar{N}-1} \ell(\hat{\psi}_{i+1,n}(k, x)) \ge \sum_{k=0}^{\bar{N}-1} \ell(\psi_{i+1,n}(k, x_n)) - \frac{\varepsilon}{2}$ for $\psi_{i+1,n} = (\phi_{i+1,n}, v_{i+1,n}) \in S^{i+1}(x_n)$ as defined in Appendix I-G. By adding and subtracting $\sum_{k=\bar{N}}^{\infty} \ell(\psi_{i+1,n}(k, x_n))$ to the previous inequality and in view of the optimality of $\psi_{i+1,n}$, $V^{i+1}(x) \ge \sum_{k=0}^{\infty} \ell(\psi_{i+1,n}(k, x_n)) - \frac{\varepsilon}{2} - \sum_{k=\bar{N}}^{\infty} \ell(\psi_{i+1,n}(k, x_n)) = V^{i+1}(x_n) - \frac{\varepsilon}{2} - \sum_{k=\bar{N}}^{\infty} \ell(\psi_{i+1,n}(k, x_n)).$ Again, in view of the optimality of $\psi_{i+1,n}$, $V^{i+1}(\phi_{i+1,n}(\bar{N}, x_n)) = \sum_{k=\bar{N}}^{\infty} \ell(\psi_{i+1,n}(k, x_n)).$ Hence it follows from the previous inequality that

$$V^{i+1}(x) \geq V^{i+1}(x_n) - \varepsilon/2 - V^{i+1}(\phi_{i+1,n}(\bar{N}, x_n)). \quad (43)$$

The following inequality is deduced from $V^{i+1}(z) \le \bar{\alpha}_V(\sigma(z))$ for any $z \in \mathbb{R}^{n_x}$ in view of Lemma 6, SA3, (35), and optimality of $\psi_{i+1,n}$, $V^{i+1}(\phi_{i+1,n}(\bar{N}, x_n)) \le \bar{\alpha}_V(\sigma(\phi_{i+1,n}(\bar{N}, x_n))) \le \bar{\alpha}_V(\sigma(\hat{\phi}_{i+1,n}(\bar{N}, x)) + \varepsilon_1)$. Given $n \in \mathbb{Z}_{\ge 0}$ sufficiently big so that (32) holds, by proceeding like in the end of the proof of Lemma 9, we obtain $V^{i+1}(\phi_{i+1,n}(\bar{N}, x_n)) \le 2\bar{\alpha}_V(2\varepsilon_1)$. By definition of $\varepsilon_1$ in (31) we deduce $V^{i+1}(\phi_{i+1,n}(\bar{N}, x_n)) \le \varepsilon/2$. Finally, by (43), $V^{i+1}(x) \ge V^{i+1}(x_n) - \varepsilon$. Hence, $V^{i+1}$ is upper semicontinuous at $x$. We have that $V^{i+1}$ is continuous at arbitrary $x$, it is thus continuous on $\mathbb{R}^{n_x}$.

# APPENDIX V
## TECHNICAL LEMMAS

**Lemma 12:** Given an outer-semicontinuous set-valued map $H: \mathbb{R}^n \to \mathbb{R}^m$ with $n, m \in \mathbb{Z}_{>0}$, the function $(x, u) \mapsto \delta_{H(x)}(u)$ is lower semicontinuous on $\mathbb{R}^n \times \mathbb{R}^m$. $\square$

**Proof:** Let $x \in \mathbb{R}^n$ and $u \in \mathbb{R}^m$. To prove that $\delta_{H(\cdot)}(\cdot)$ is lower semicontinuous at $(x, u)$, we show that $\liminf_{(\tilde{x}, \tilde{u}) \to (x, u)} \delta_{H(\tilde{x})}(\tilde{u}) \ge \delta_{H(x)}(u)$, see [27, Definition 1.5]. This is equivalent to $\min\{\alpha \in \mathbb{R}: \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\} \ge \delta_{H(x)}(u)$, as $\liminf_{(\tilde{x}, \tilde{u}) \to (x, u)} \delta_{H(\tilde{x})}(\tilde{u}) = \min\{\alpha \in \mathbb{R}: \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\}$ by [27, Lemma 1.7].

For what follows, we employ the fact that both $\min\{\alpha \in \mathbb{R}: \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\}$ and $\delta_{H(x)}(u)$ only take values in $\{0, \infty\}$.
We distinguish two cases. First, when $\min\{\alpha \in \mathbb{R}: \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\} = \infty$. The desired result holds. Second, when $\min\{\alpha \in \mathbb{R}: \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\} = 0$. This implies that there exists a (sub)sequence $(x_n, u_n) \to (x, u)$ such that $u_n \in H(x_n)$ as $\delta_{H(x_n)}(u_n) \to 0$. Hence, since $H$ is outer semicontinuous, we have $u \in H(x)$. Therefore, $\delta_{H(x)}(u) = 0$, and $\min\{\alpha \in \mathbb{R}: \exists(x_n, u_n) \to (x, u), \delta_{H(x_n)}(u_n) \to \alpha\} = 0 = \delta_{H(x)}(u)$. We have proved that $\delta_{H(\cdot)}(\cdot)$ is lower semicontinuous at arbitrary $(x, u)$, it is therefore lower semicontinuous on $\mathbb{R}^n \times \mathbb{R}^m$. $\blacksquare$

**Lemma 13:** Let $n, m \in \mathbb{Z}_{>0}$ and $g, \ell: \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}_{\ge 0}$ with $g(x, u) \ge \ell(x, u)$ for any $(x, u) \in \mathbb{R}^n \times \mathbb{R}^m$. If $\ell$ is level-bounded in $u$, locally uniform in $x$, then so is $g$. $\square$

**Proof:** Let $x \in \mathbb{R}^n$, $\alpha \in \mathbb{R}$, $S$ be a neighbourhood of $x$ and $B \subset \mathbb{R}^m$ be bounded such that $\{u \in \mathbb{R}^m: \ell(z, u) \le \alpha\} \subset B$ for any $z \in S$, which exist by item (ii) of Assumption 2, see Definition 1. Necessarily $\{u \in \mathbb{R}^m: g(z, u) \le \alpha\} \subset \{u \in \mathbb{R}^m: \ell(z, u) \le \alpha\}$ for $z \in S$, as $\ell(z, u) \le g(z, u)$, hence $\{u \in \mathbb{R}^m: g(z, u) \le \alpha\} \subset B$ for any $z \in S$. As $x$ and $\alpha$ have been arbitrarily, the desired result holds. $\blacksquare$

**Lemma 14:** Let $\alpha: \mathbb{R}_{\ge 0} \to \mathbb{R}_{\ge 0}$ be continuous and $\{s_k\}_{k \in \mathbb{Z}_{\ge 0}}$ with $0 \le s_{k+1} \le \alpha(s_k)$ and $s_k \ge 0$. Then $s_k \le \tilde{\beta}(s_0, k) := \tilde{\alpha}^{(k)}(s_0)$ with $\tilde{\alpha}(s_0) := \max_{s \in [0, s_0]} \alpha(s)$. Moreover, when $\alpha(s) < s$ for all $s \in \mathbb{R}_{>0}$, $\tilde{\beta} \in \mathcal{KL}$. $\square$

**Proof:** Note that $\tilde{\alpha}(s_0) < \infty$ for any $s_0 \ge 0$ as $\alpha$ is continuous. Moreover, $\tilde{\alpha}$ is non-decreasing in view of its definition. We show that $s_k \le \tilde{\alpha}^{(k)}(s_0)$ for all $k \in \mathbb{Z}_{\ge 0}$ by induction. This property is true for $k=0$ as $s_0 \le \tilde{\alpha}^{(0)}(s_0) = s_0$. Let $k \in \mathbb{Z}_{\ge 0}$ and assume that $s_k \le \tilde{\alpha}^{(k)}(s_0)$. By combining $s_{k+1} \le \alpha(s_k)$, $\alpha(s) \le \tilde{\alpha}(s)$ for all $s \ge a$, $\tilde{\alpha}$ non-decreasing and the assumed upper-bound for $s_k$, we obtain $s_{k+1} \le \alpha(s_k) \le \tilde{\alpha}(s_k) \le \tilde{\alpha}(\tilde{\alpha}^{(k)}(s_0)) = \tilde{\alpha}^{(k+1)}(s_0)$. The induction is complete. Now we establish $\tilde{\beta} \in \mathcal{KL}$ when $\alpha(s) < s$ for all $s > 0$. First, it is clear that $\tilde{\alpha}^{(k)}$ is continuous and zero at zero for any $k \in \mathbb{Z}_{\ge 0}$, hence $\tilde{\beta}(\cdot, k)$ is continuous, zero at zero and non-decreasing for any $k \in \mathbb{Z}_{\ge 0}$, thus $\tilde{\beta}$ is class-$\mathcal{K}$ in its first argument. Moreover, as $\alpha(s) < s$ for all $s > 0$ and $\alpha(0) = 0$, we have $\tilde{\beta}(s, k+1) < \tilde{\beta}(s, k)$ when $\tilde{\beta}(s, k) > 0$ and $\tilde{\beta}(s, k+1) = \tilde{\beta}(s, k) = 0$ otherwise. Therefore $\tilde{\beta}(s, \cdot)$ monotonically decreases to zero. $\blacksquare$

<footnote>[1] F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause. Safe model-based reinforcement learning with stability guarantees. In *Adv. Neural Inf. Process Syst.*, pages 908–918, 2017.</footnote>

<footnote>[2] D.P. Bertsekas. *Dynamic Programming and Optimal Control*, volume 2. Athena Scientific, Belmont, U.S.A., 4th edition, 2012.</footnote>

<footnote>[3] D.P. Bertsekas. Value and policy iterations in optimal control and adaptive dynamic programming. *IEEE Trans. Neural Netw. Learn. Syst.*, 28(3):500–509, 2015.</footnote>

<footnote>[4] D.P. Bertsekas. *Reinforcement Learning and Optimal Control*. Athena Scientific, 2019.</footnote>

<footnote>[5] T. Bian, Y. Jiang, and Z.-P. Jiang. Adaptive dynamic programming and optimal control of nonlinear nonaffine systems. *Automatica*, 50(10):2624–2632, 2014.</footnote>
[6] L. Buşoniu, D. Ernst, B. De Schutter, and R. Babuška. Approximate reinforcement learning: An overview. In *IEEE Symp. Adapt. Dyn. Program. Reinf. Learn.*, pages 1–8, Paris, France, April 2011.

[7] T.Y. Chun, J.Y. Lee, J.B. Park, and Y.H. Choi. Stability and monotone convergence of generalised policy iteration for discrete-time linear quadratic regulations. *Int. J. of Control*, 89(3):437–450, 2016.

[8] W. H. Fleming and H. M. Soner. *Controlled Markov Processes and Viscosity Solutions*. Springer, New York, USA, 2006.

[9] R. Goebel, R.G. Sanfelice, and A.R. Teel. *Hybrid Dynamical Systems*. Princeton University Press, Princeton, U.S.A., 2012.

[10] M. Granzotto, O. Lindamulage de Silva, R. Postoyan, D. Nešić, and Z.-P. Jiang. Regularizing policy iteration for recursive feasibility and stability. In *IEEE CDC, Cancún, Mexico*, 2022.

[11] M. Granzotto, R. Postoyan, L. Buşoniu, D. Nešić, and J. Daafouz. Finite-horizon discounted optimal control: stability and performance. *IEEE Trans. Autom. Control*, 66(2):550–565, 2021.

[12] G. Grimm, M.J. Messina, S.E. Tuna, and A.R. Teel. Examples when nonlinear model predictive control is nonrobust. *Automatica*, 40(10):1729–1738, 2004.

[13] G. Grimm, M.J. Messina, S.E. Tuna, and A.R. Teel. Model predictive control: for want of a local control Lyapunov function, all is not lost. *IEEE Trans. Autom. Control*, 50(5):546–558, 2005.

[14] W. Guo, J. Si, F. Liu, and S. Mei. Policy approximation in policy iteration approximate dynamic programming for discrete-time nonlinear systems. *IEEE Trans. Neural Netw. Learn. Syst.*, 29(7):2794–2807, 2018.

[15] A. Heydari. Analyzing policy iteration in optimal control. In *ACC, Boston, U.S.A.*, pages 5728–5733, 2016.

[16] M. Höger and L. Grüne. On the relation between detectability and strict dissipativity for nonlinear discrete time systems. *IEEE Control Sys. Lett.*, 3(2):458–462, 2019.

[17] Z.-P. Jiang, T. Bian, and W. Gao. Learning-based control: A tutorial and some recent results. *Found. Trends® Sys. Control*, 8(3):176–284, 2020.

[18] R.E. Kalman. Contributions to the theory of optimal control. *Bol. Soc. Mat. Mexicana*, 5(2):102–119, 1960.

[19] S.S. Keerthi and E.G. Gilbert. An existence theorem for discrete-time infinite-horizon optimal control problems. *IEEE Trans. Autom. Control*, 30(9):907–909, 1985.

[20] C.M. Kellett and A.R. Teel. Discrete-time asymptotic controllability implies smooth control-Lyapunov function. *Sys. Control Lett.*, 52:349–359, 2004.

[21] C.M. Kellett and A.R. Teel. On the robustness of *KL*-stability for difference inclusions: smooth discrete-time Lyapunov functions. *SIAM J. Control Optim.*, 44(3):777–800, 2005.

[22] Y. Li, K.P. Tee, R. Yan, W.L. Chan, and Y. Wu. A framework of human-robot coordination based on game theory and policy iteration. *IEEE Trans. Robot.*, 32(6):1408–1418, 2016.

[23] D. Liu and Q. Wei. Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. *IEEE Trans. Neural Netw. Learn. Syst.*, 25(3):621–634, 2013.

[24] H. Modares, M. N. Sistani, and F. L. Lewis. A policy iteration approach to online optimal control of continuous-time constrained-input systems. *ISA Trans.*, 52(5):611–621, 2013.

[25] B. Pang, T. Bian, and Z.-P. Jiang. Robust policy iteration for continuous-time linear quadratic regulation. *IEEE Trans. Autom. Control*, 67(1):504–511, 2022.

[26] R. Postoyan, L. Buşoniu, D. Nešić, and J. Daafouz. Stability analysis of discrete-time infinite-horizon optimal control with discounted cost. *IEEE Trans. Autom. Control*, 62(6):2736–2749, 2017.

[27] R.T. Rockafellar and R.J.-B. Wets. *Variational analysis*. Springer, Dordrecht, Germany, 3rd edition, 1998.

[28] R.S. Sutton and A.G. Barto. *Reinforcement Learning: An Introduction*. MIT press, 2018.

[29] J. Wang, X. Xu, D. Liu, Z. Sun, and Q. Chen. Self-learning cruise control using kernel-based least squares policy iteration. *IEEE Trans. Control Sys. Technol.*, 22(3):1078–1087, 2014.

[30] Y. Wu and T. Shen. Policy iteration approach to control residual gas fraction in ic engines under the framework of stochastic logical dynamics. *IEEE Trans. Control Sys. Technol.*, 25(3):1100–1107, 2017.

<figure><img src="image_4.png" /></figure>

**Mathieu Granzotto** received his engineering degree in Control and Automation in 2016 from "Universidade Federal de Santa Catarina", Brazil. In 2019, he received his Ph.D. in Control Theory from Université de Lorraine, France, where he was a Temporary Research and Teaching Attaché at CRAN. Since 2022, he is a research fellow at the Department of Electrical and Electronic Engineering (DEEE) at the University of Melbourne, Australia.

<figure><img src="image_5.png" /></figure>

**Olivier Lindamulage De Silva** received the "Ingénieur" degree in Digital Systems Engineering from ENSEM (France) in 2020. He obtained the M.Sc. by Research in Robotic Vision Learning from Université de Lorraine (France) in 2020, and the Ph.D. both from Université de Lorraine in 2023.

<figure><img src="image_6.png" /></figure>

**Romain Postoyan** received the "Ingénieur" degree in Electrical and Control Engineering from ENSEEIHT (France) in 2005. He obtained the M.Sc. by Research in Control Theory & Application from Coventry University (United Kingdom) in 2006 and the Ph.D. in Control Theory from Université Paris-Sud (France) in 2009. In 2010, he was a research assistant at the University of Melbourne (Australia). Since 2011, he is a CNRS researcher at the CRAN (France).

<figure><img src="image_7.png" /></figure>

**Dragan Nešić** is a Professor at The University of Melbourne, Australia. He currently serves as Associate Dean Research at the Melbourne School of Engineering. His research interests include networked control systems, reset systems, extremum seeking control, hybrid control systems, event-triggered control, security and privacy in cyber-physical systems, and so on. He is a Fellow of the Institute of Electrical and Electronic Engineers (IEEE, 2006) and Fellow of the International Federation for Automatic Control (IFAC, 2019). He was a co-recipient of the George S. Axelby Outstanding Paper Award for the Best Paper in IEEE Transactions on Automatic Control (2018).

<figure><img src="image_8.png" /></figure>

**Zhong-Ping Jiang** (Fellow, IEEE) received the M.Sc. degree in statistics from the University of Paris XI, France, in 1989, and the Ph.D. degree in automatic control and mathematics from the Ecole des Mines de Paris (now, called ParisTech-Mines), France, in 1993, under the direction of Prof. Laurent Praly. Currently, he is a Professor of Electrical and Computer Engineering at the Tandon School of Engineering, New York University. His main research interests include stability theory, robust/adaptive/distributed nonlinear control, robust adaptive dynamic programming, reinforcement learning and their applications to information, mechanical and biological systems. He has served as Deputy Editor-in-Chief, Senior Editor and Associate Editor for numerous journals. Prof. Jiang is a Fellow of the IEEE, IFAC, and CAA, a foreign member of the Academia Europaea (Academy of Europe), and is among the Clarivate Analytics Highly Cited Researchers. In 2022, he received the Excellence in Research Award from the NYU Tandon School of Engineering.
[6] L. Buşoniu, D. Ernst, B. De Schutter, and R. Babuška. Approximate reinforcement learning: An overview. In *IEEE Symp. Adapt. Dyn. Program. Reinf. Learn.*, pages 1–8, Paris, France, April 2011.

[7] T.Y. Chun, J.Y. Lee, J.B. Park, and Y.H. Choi. Stability and monotone convergence of generalised policy iteration for discrete-time linear quadratic regulations. *Int. J. of Control*, 89(3):437–450, 2016.

[8] W. H. Fleming and H. M. Soner. *Controlled Markov Processes and Viscosity Solutions*. Springer, New York, USA, 2006.

[9] R. Goebel, R.G. Sanfelice, and A.R. Teel. *Hybrid Dynamical Systems*. Princeton University Press, Princeton, U.S.A., 2012.

[10] M. Granzotto, O. Lindamulage de Silva, R. Postoyan, D. Nešić, and Z.-P. Jiang. Regularizing policy iteration for recursive feasibility and stability. In *IEEE CDC, Cancún, Mexico*, 2022.

[11] M. Granzotto, R. Postoyan, L. Buşoniu, D. Nešić, and J. Daafouz. Finite-horizon discounted optimal control: stability and performance. *IEEE Trans. Autom. Control*, 66(2):550–565, 2021.

[12] G. Grimm, M.J. Messina, S.E. Tuna, and A.R. Teel. Examples when nonlinear model predictive control is nonrobust. *Automatica*, 40(10):1729–1738, 2004.

[13] G. Grimm, M.J. Messina, S.E. Tuna, and A.R. Teel. Model predictive control: for want of a local control Lyapunov function, all is not lost. *IEEE Trans. Autom. Control*, 50(5):546–558, 2005.

[14] W. Guo, J. Si, F. Liu, and S. Mei. Policy approximation in policy iteration approximate dynamic programming for discrete-time nonlinear systems. *IEEE Trans. Neural Netw. Learn. Syst.*, 29(7):2794–2807, 2018.

[15] A. Heydari. Analyzing policy iteration in optimal control. In *ACC, Boston, U.S.A.*, pages 5728–5733, 2016.

[16] M. Höger and L. Grüne. On the relation between detectability and strict dissipativity for nonlinear discrete time systems. *IEEE Control Sys. Lett.*, 3(2):458–462, 2019.

[17] Z.-P. Jiang, T. Bian, and W. Gao. Learning-based control: A tutorial and some recent results. *Found. Trends® Sys. Control*, 8(3):176–284, 2020.

[18] R.E. Kalman. Contributions to the theory of optimal control. *Bol. Soc. Mat. Mexicana*, 5(2):102–119, 1960.

[19] S.S. Keerthi and E.G. Gilbert. An existence theorem for discrete-time infinite-horizon optimal control problems. *IEEE Trans. Autom. Control*, 30(9):907–909, 1985.

[20] C.M. Kellett and A.R. Teel. Discrete-time asymptotic controllability implies smooth control-Lyapunov function. *Sys. Control Lett.*, 52:349–359, 2004.

[21] C.M. Kellett and A.R. Teel. On the robustness of *KL*-stability for difference inclusions: smooth discrete-time Lyapunov functions. *SIAM J. Control Optim.*, 44(3):777–800, 2005.

[22] Y. Li, K.P. Tee, R. Yan, W.L. Chan, and Y. Wu. A framework of human-robot coordination based on game theory and policy iteration. *IEEE Trans. Robot.*, 32(6):1408–1418, 2016.

[23] D. Liu and Q. Wei. Policy iteration adaptive dynamic programming algorithm for discrete-time nonlinear systems. *IEEE Trans. Neural Netw. Learn. Syst.*, 25(3):621–634, 2013.

[24] H. Modares, M. N. Sistani, and F. L. Lewis. A policy iteration approach to online optimal control of continuous-time constrained-input systems. *ISA Trans.*, 52(5):611–621, 2013.

[25] B. Pang, T. Bian, and Z.-P. Jiang. Robust policy iteration for continuous-time linear quadratic regulation. *IEEE Trans. Autom. Control*, 67(1):504–511, 2022.

[26] R. Postoyan, L. Buşoniu, D. Nešić, and J. Daafouz. Stability analysis of discrete-time infinite-horizon optimal control with discounted cost. *IEEE Trans. Autom. Control*, 62(6):2736–2749, 2017.

[27] R.T. Rockafellar and R.J.-B. Wets. *Variational analysis*. Springer, Dordrecht, Germany, 3rd edition, 1998.

[28] R.S. Sutton and A.G. Barto. *Reinforcement Learning: An Introduction*. MIT press, 2018.

[29] J. Wang, X. Xu, D. Liu, Z. Sun, and Q. Chen. Self-learning cruise control using kernel-based least squares policy iteration. *IEEE Trans. Control Sys. Technol.*, 22(3):1078–1087, 2014.

[30] Y. Wu and T. Shen. Policy iteration approach to control residual gas fraction in ic engines under the framework of stochastic logical dynamics. *IEEE Trans. Control Sys. Technol.*, 25(3):1100–1107, 2017.

<figure><img src="image_4.png" /></figure>

**Mathieu Granzotto** received his engineering degree in Control and Automation in 2016 from "Universidade Federal de Santa Catarina", Brazil. In 2019, he received his Ph.D. in Control Theory from Université de Lorraine, France, where he was a Temporary Research and Teaching Attaché at CRAN. Since 2022, he is a research fellow at the Department of Electrical and Electronic Engineering (DEEE) at the University of Melbourne, Australia.

<figure><img src="image_5.png" /></figure>

**Olivier Lindamulage De Silva** received the "Ingénieur" degree in Digital Systems Engineering from ENSEM (France) in 2020. He obtained the M.Sc. by Research in Robotic Vision Learning from Université de Lorraine (France) in 2020, and the Ph.D. both from Université de Lorraine in 2023.

<figure><img src="image_6.png" /></figure>

**Romain Postoyan** received the "Ingénieur" degree in Electrical and Control Engineering from ENSEEIHT (France) in 2005. He obtained the M.Sc. by Research in Control Theory & Application from Coventry University (United Kingdom) in 2006 and the Ph.D. in Control Theory from Université Paris-Sud (France) in 2009. In 2010, he was a research assistant at the University of Melbourne (Australia). Since 2011, he is a CNRS researcher at the CRAN (France).

<figure><img src="image_7.png" /></figure>

**Dragan Nešić** is a Professor at The University of Melbourne, Australia. He currently serves as Associate Dean Research at the Melbourne School of Engineering. His research interests include networked control systems, reset systems, extremum seeking control, hybrid control systems, event-triggered control, security and privacy in cyber-physical systems, and so on. He is a Fellow of the Institute of Electrical and Electronic Engineers (IEEE, 2006) and Fellow of the International Federation for Automatic Control (IFAC, 2019). He was a co-recipient of the George S. Axelby Outstanding Paper Award for the Best Paper in IEEE Transactions on Automatic Control (2018).

<figure><img src="image_8.png" /></figure>

**Zhong-Ping Jiang** (Fellow, IEEE) received the M.Sc. degree in statistics from the University of Paris XI, France, in 1989, and the Ph.D. degree in automatic control and mathematics from the Ecole des Mines de Paris (now, called ParisTech-Mines), France, in 1993, under the direction of Prof. Laurent Praly. Currently, he is a Professor of Electrical and Computer Engineering at the Tandon School of Engineering, New York University. His main research interests include stability theory, robust/adaptive/distributed nonlinear control, robust adaptive dynamic programming, reinforcement learning and their applications to information, mechanical and biological systems. He has served as Deputy Editor-in-Chief, Senior Editor and Associate Editor for numerous journals. Prof. Jiang is a Fellow of the IEEE, IFAC, and CAA, a foreign member of the Academia Europaea (Academy of Europe), and is among the Clarivate Analytics Highly Cited Researchers. In 2022, he received the Excellence in Research Award from the NYU Tandon School of Engineering.