SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. The result is a direct consequence of (Nesterov, 2003, Theorem 4.1.3, Theorem 4.1.5, Theorem 4.1.6).  $\square$
We now introduce  $\alpha$ -regularity, which slightly strengthens self-concordance, by ensuring that  $D^3\phi$  is  $\alpha (\alpha +1)$ -Lipschitz continuous with respect to the local norm induced by  $\mathfrak{g}$  (see (11)). We state below the definition of  $\alpha$ -regularity as well as some of its properties in Lemma 11.
Definition 10 (Nesterov & Nemirovski (1998)). Let  $\alpha \geq 1$  and  $\mathsf{U}$  be non-empty open convex domain in  $\mathbb{R}^d$ . A self-concordant function  $\phi$  on  $\mathsf{U}$  is said  $\alpha$ -regular, if  $\phi \in \mathrm{C}^4(\mathsf{U},\mathbb{R})$  and for any  $x \in \mathsf{U}$  and any  $h \in \mathbb{R}^d$
$$
\left| D ^ {4} \phi (x) [ h, h, h, h ] \right| \leq \alpha (\alpha + 1) \| h \| _ {\mathfrak {g} (x)} ^ {2} \| h \| _ {\mathrm {U}, x} ^ {2},
$$
where  $\| h \|_{\mathsf{U}, x} \coloneqq \inf \{ t^{-1} \mid t > 0, x \pm th \in \mathsf{U} \}$ .
Lemma 11. Let  $\phi : \mathsf{U} \to \mathbb{R}$  be a  $\alpha$ -regular function with  $\mathfrak{g} = D^2\phi$ . Assume that  $\mathsf{U}$  contains no straight line.
(a) For any  $x\in \mathsf{U}$  and any  $h\in \mathbb{R}^d$
$$
\left| D ^ {4} \phi (x) [ h, h, h, h ] \right| \leq \alpha (\alpha + 1) \| h \| _ {\mathfrak {g} (x)} ^ {4}. \tag {11}
$$
(b) For any  $x\in \mathsf{U}$ , any  $(h_1,h_2,h_3,h_4)\in \mathbb{R}^d\times \mathbb{R}^d\times \mathbb{R}^d\times \mathbb{R}^d$
$$
\left| D ^ {4} \phi (x) \left[ h _ {1}, h _ {2}, h _ {3}, h _ {4} \right] \right| \leq \alpha (\alpha + 1) \prod_ {i = 1} ^ {4} \| h _ {i} \| _ {\mathfrak {g} (x)}. \tag {12}
$$
(c) For any  $x\in \mathsf{U}$ , any  $y\in \mathsf{W}^0 (x,1)$  and any  $(h_1,h_2)\in \mathbb{R}^d\times \mathbb{R}^d$
$$
\left\| D ^ {3} \phi (x) \left[ h _ {1}, h _ {2} \right] - D ^ {3} \phi (y) \left[ h _ {1}, h _ {2} \right] \right\| _ {\mathfrak {g} (x) ^ {- 1}} \leq \alpha (\alpha + 1) / 3 \| h _ {1} \| _ {\mathfrak {g} (x)} \| h _ {2} \| _ {\mathfrak {g} (x)} ((1 - \| y - x \| _ {\mathfrak {g} (x)}) ^ {- 3} - 1). \tag {13}
$$
Proof. We first prove (11). First remark that the result is true for  $h = 0$ . Consider now  $h \neq 0$ . Let  $\varepsilon > 0$ . We define  $t_{\varepsilon} = (1 + \varepsilon)^{-1}\|h\|_{\mathfrak{g}(x)}^{-1} > 0$ ,  $y_{\varepsilon}^{+} = x + t_{\varepsilon}h$  and  $y_{\varepsilon}^{-} = x - t_{\varepsilon}h$ . We have  $\|x - y_{\varepsilon}^{+}\|_{\mathfrak{g}(x)} = \|x - y_{\varepsilon}^{-}\|_{\mathfrak{g}(x)} = 1/(1 + \varepsilon) < 1$ . Hence,  $y_{\varepsilon}^{-} \in \mathsf{U}$  and  $y_{\varepsilon}^{+} \in \mathsf{U}$  by Lemma 9. Therefore,  $\|h\|_{\mathsf{U},x} \leq t_{\varepsilon}^{-1}$  and
$$
\left| D ^ {4} \phi (x) [ h, h, h, h ] \right| \leq \alpha (\alpha + 1) \| h \| _ {\mathfrak {g} (x)} ^ {4} (1 + \varepsilon) ^ {2}.
$$
We conclude upon taking  $\varepsilon \rightarrow 0$ . Then, (12) is an immediate consequence of (11) using (Nesterov & Nemirovskii, 1994, Proposition 9.1.1). We now prove (13). Using Lemma 9, we have
$$
\begin{array}{l} \| D ^ {3} \phi (x) [ h _ {1}, h _ {2} ] - D ^ {3} \phi (y) [ h _ {1}, h _ {2} ] \| _ {\mathfrak {g} (x) ^ {- 1}} \\ \leq \int_ {0} ^ {1} \| D ^ {4} \phi (x + t (y - x)) [ h _ {1}, h _ {2}, y - x ] \| _ {\mathfrak {g} (x) ^ {- 1}} d t \\ \leq \int_ {0} ^ {1} (1 - t \| y - x \| _ {\mathfrak {g} (x)}) ^ {- 1} \| D ^ {4} \phi (x + t (y - x)) [ h _ {1}, h _ {2}, y - x ] \| _ {\mathfrak {g} (x + t (y - x)) ^ {- 1}} \mathrm {d} t \\ \leq \int_ {0} ^ {1} \alpha (\alpha + 1) / (1 - t \| y - x \| _ {\mathfrak {g} (x)}) \| h _ {1} \| _ {\mathfrak {g} (x + t (y - x))} \| h _ {2} \| _ {\mathfrak {g} (x + t (y - x))} \| y - x \| _ {\mathfrak {g} (x + t (y - x))} d t \\ \leq \alpha (\alpha + 1) \| h _ {1} \| _ {\mathfrak {g} (x)} \| h _ {2} \| _ {\mathfrak {g} (x)} \int_ {0} ^ {1} \| y - x \| _ {\mathfrak {g} (x)} / (1 - t \| y - x \| _ {\mathfrak {g} (x)}) ^ {4} d t \\ \leq \alpha (\alpha + 1) / 3 \| h _ {1} \| _ {\mathfrak {g} (x)} \| h _ {2} \| _ {\mathfrak {g} (x)} ((1 - \| y - x \| _ {\mathfrak {g} (x)}) ^ {- 3} - 1), \\ \end{array}
$$
We now prove the equivalence between the Euclidean norm and the norms induced by  $\mathfrak{g}$  and  $\mathfrak{g}^{-1}$ .
Lemma 12. Let  $(\mathsf{U},\mathfrak{g})$  be non-empty Riemannian manifold in  $\mathbb{R}^d$ . For any  $x_0\in \mathsf{U}$ , any  $(x,p)\in \mathsf{U}\times \mathbb{R}^d$ , we have
$$
\begin{array}{l} \begin{array}{l l} \| \mathfrak {g} (x _ {0}) ^ {- 1} \| _ {2} ^ {- 1 / 2} \| x \| _ {2} \leq \| x \| _ {\mathfrak {g} (x _ {0})} & \leq \| \mathfrak {g} (x _ {0}) \| _ {2} ^ {1 / 2} \| x \| _ {2}, \end{array} \\ \left\| \mathfrak {g} (x _ {0}) \right\| _ {2} ^ {- 1 / 2} \| p \| _ {2} \leq \| p \| _ {\mathfrak {g} (x _ {0}) ^ {- 1}} \leq \| \mathfrak {g} (x _ {0}) ^ {- 1} \| _ {2} ^ {1 / 2} \| p \| _ {2}. \\ \end{array}
$$
In addition, let
$$
C _ {x _ {0}} = \max  \left(\left\| \mathfrak {g} (x _ {0}) \right\| _ {2} ^ {1 / 2}, \left\| \mathfrak {g} (x _ {0}) ^ {- 1} \right\| _ {2} ^ {1 / 2}\right) > 0.
$$
$$
1 / C _ {x _ {0}} \left(\| x \| _ {2} + \| p \| _ {2}\right) \leq \| x \| _ {\mathfrak {g} (x _ {0})} + \| p \| _ {\mathfrak {g} (x _ {0}) ^ {- 1}} \leq C _ {x _ {0}} \left(\| x \| _ {2} + \| p \| _ {2}\right).
$$
D C-BHMC: Algorithm and Results
In this section, we first state general results on the Hamiltonian dynamics (3) under our main assumptions. It allows us to introduce the continuous version of BHMC (c-BHMC), for which we derive the reversibility with respect to  $\pi$ . Full proofs of this section are provided in Appendix E.
Results on the Hamiltonian dynamics. Using Cauchy-Lipschitz's theorem, we obtain in Proposition 13 the existence and uniqueness of the Hamiltonian dynamics (3), starting from any  $z_0 \in \mathbf{T}^*\mathsf{M}$ .
Proposition 13. Assume A1, A2. Let  $z_0 \in \mathrm{T}^\star \mathsf{M}$ . Then (3) admits a unique maximal solution  $(J_{z_0}, z)$ , where (i)  $J_{z_0} \subset \mathbb{R}$  is an open neighbourhood of 0, (ii)  $z \in \mathrm{C}^1(J_{z_0}, \mathrm{T}^\star \mathsf{M})$ , (iii)  $z(0) = z_0$  and (iv) for any  $t \in J_{z_0}$ ,  $H(z(t)) = H(z_0)$ .
In contrast to the Euclidean case (i.e.,  $\mathfrak{g}(x) = \mathrm{I}_d$ ), the Hamiltonian dynamics (3) relies on a metric derived from a self-concordant barrier, and thus is defined up to an explosion time. In the next result, we characterize the behaviour of the solutions when this explosion time is finite.
Proposition 14. Assume A1, A2. Let  $z_0 \in \mathrm{T}^* \mathsf{M}$ ,  $(J_{z_0}, z)$  as in Proposition 13. Assume  $T_{z_0} = \sup J_{z_0} < +\infty$ , then we have
(a)  $\lim_{t\to T_{z_0}}\| p(t)\| _2 = +\infty .$
(b) There exists  $x_{\mathsf{M}} \in \partial \mathsf{M}$  such that  $\lim_{t \to T_{z_0}} x(t) = x_{\mathsf{M}}$ .
In the rest of this section, we define the Hamiltonian flow as the map  $\mathrm{T}:\mathrm{D}\to \mathrm{T}^{\star}\mathsf{M}$ , where  $\mathrm{D} = \{(h,z_0):z_0\in \mathrm{T}^\star \mathsf{M},h < \sup J_{z_0}\}$ , such that  $\mathrm{T}_h(z_0) = z(h)$  for any  $(h,z_0)\in \mathrm{D},z$  being uniquely defined from  $z_{0}$  in Proposition 13.
Introduction to c-BHMC. In the case where  $\mathsf{D} = \mathsf{T}^{\star}\mathsf{M}\times \mathbb{R}$ , the definition of the Hamiltonian dynamics ensures that  $\mathrm{T}_h$  is involutive (up to momentum reversal) for any  $h > 0$ , which is key to derive reversibility in BHMC. However, in our manifold setting, there is one subtlety that needs to be checked, namely that the Hamiltonian flow does not leave the cotangent bundle in finite time. As detailed in Proposition 14, there is indeed no guarantee that this flow is defined for all times, i.e., that
we have  $\mathsf{D} = \mathbb{T}^{\star}\mathsf{M}\times \mathbb{R}$ , due to the ill-conditioned behavior of the dynamics near the boundary of the manifold. Therefore, given a time horizon  $h > 0$ , we restrict the cotangent bundle to the points where the Hamiltonian flow is defined up to time  $h$ . Hence, we define, for any  $h > 0$ , the sets
$$
\mathsf {E} _ {h} = \left\{z _ {0} \in \mathrm {T} ^ {\star} \mathsf {M}: h <   \sup  J _ {z _ {0}} \right\}, \quad \bar {\mathsf {E}} _ {h} = \left\{z _ {0} \in \mathsf {E} _ {h}: h <   \sup  J _ {(s \circ \mathrm {T} _ {h}) (z _ {0})} \right\}.
$$
Note that elements of  $\bar{\mathsf{E}}_h$  correspond to points  $z_{0}$  for which:
(a) we are able to compute exactly the Hamiltonian dynamics (3) starting from  $z_0$  on the time interval  $[0, h]$ , by considering  $\{\mathrm{T}_t(z_0)\}_{t \in [0, h]}$  (since  $\bar{\mathsf{E}}_h \subset \mathsf{E}_h$ ),
(b) we are also able to compute exactly the reversed dynamics - up to momentum reversal - starting from  $(s\circ \mathrm{T}_h)(z_0)$  on the time interval  $[0,h]$ .
In Appendix E, we prove that  $\mathrm{T}_h$  is symplectic (up to momentum reversal) on  $\bar{\mathsf{E}}_h$  for any  $h > 0$ , which thus ensures that the Hamiltonian dynamics is reversible with respect to  $\bar{\pi}$ , see Lemma 16.
Let  $h > 0$ . Our algorithm c-BHMC, whose pseudo-code is provided in Algorithm 2, then proceeds as follows at stage  $n \geq 1$ . Assume that the current state is  $(X_{n-1}, P_{n-1}) \in \mathrm{T}^{\star}\mathsf{M}$ . First define a partial refreshment of the momentum  $\tilde{P}_n \gets \sqrt{1 - \beta} P_{n-1} + \sqrt{\beta} G_n$ , where  $G_n \sim \mathrm{N}_x(0, \mathrm{I}_d)$  is independent from  $P_{n-1}$ . Then, verify that  $(X_{n-1}, \tilde{P}_n) \in \bar{\mathsf{E}}_h$ , which ensures that the Hamiltonian flow is involutive on the time interval  $[0, h]$ , starting at this state. In the same spirit as in Algorithm 1, we refer to this step as an "involution checking step", which is highlighted in yellow in Algorithm 2, although the involution property directly derives from the Hamiltonian dynamics. If this step is valid, define  $(X_n, \bar{P}_n) = \mathrm{T}_h(X_{n-1}, \tilde{P}_n)$ ; otherwise, set  $(X_n, \bar{P}_n) \gets s(X_{n-1}, \tilde{P}_n)$ . Finally, update the momentum as in the beginning of the iteration to obtain the new state  $(X_n, P_n)$ .
Algorithm 2: c-BHMC with Momentum Refresh
Input:  $(X_0,P_0)\in \mathrm{T}^{\star}\mathsf{M},\beta \in (0,1],N\in \mathbb{N},h > 0$  Output:  $(X_{n},P_{n})_{n\in [N]}$    
for  $n = 1,\dots ,N$  do Step 1:  $G_{n}\sim \mathrm{N}_{x}(0,\mathrm{I}_{d}),\tilde{P}_{n}\gets \sqrt{1 - \beta} P_{n - 1} + \sqrt{\beta} G_{n}$  Step 2: solving continuous ODE (3) if  $(X_{n - 1},\tilde{P}_n)\in \bar{\mathsf{E}}_h$  then  $(X_{n},\bar{P}_{n})\leftarrow \mathrm{T}_{h}(X_{n - 1},\tilde{P}_{n})$  else  $(X_{n},\bar{P}_{n})\gets s(X_{n - 1},\tilde{P}_{n})$  Step 3:  $G_{n}^{\prime}\sim \mathrm{N}_{x}(0,\mathrm{I}_{d}),P_{n}\gets \sqrt{1 - \beta}\bar{P}_{n} + \sqrt{\beta} G_{n}^{\prime}$    
7 end
Denote by  $\mathrm{Q}_c: \mathrm{T}^*\mathsf{M} \times \mathcal{B}(\mathrm{T}^*\mathsf{M}) \to [0,1]$ , the transition kernel of the (homogeneous) Markov chain  $(X_n, P_n)_{n \in [N]}$  obtained with Algorithm 2. Using the properties of the Hamiltonian dynamics, we get the following result.
Theorem 15. Assume A1, A2. Then,  $\mathrm{Q}_c$  is reversible up to momentum reversal, i.e., we have for any  $f\in \mathrm{C}(\mathrm{T}^{\star}\mathsf{M}\times \mathrm{T}^{\star}\mathsf{M},\mathbb{R})$  with compact support
$$
\int_ {\mathrm {T} ^ {*} \mathrm {M} \times \mathrm {T} ^ {*} \mathrm {M}} f (z, z ^ {\prime}) \mathrm {d} \bar {\pi} (z) \mathrm {Q} _ {c} (z, \mathrm {d} z ^ {\prime}) = \int_ {\mathrm {T} ^ {*} \mathrm {M} \times \mathrm {T} ^ {*} \mathrm {M}} f (s (z ^ {\prime}), s (z)) \mathrm {d} \bar {\pi} (z) \mathrm {Q} _ {c} (z, \mathrm {d} z ^ {\prime}).
$$
In particular,  $\bar{\pi}$  is an invariant measure for  $\mathrm{Q}_c$ .
E Proofs of Appendix D
E.1 Existence, uniqueness and explosion time of Hamiltonian dynamics
We prove below Proposition 13 and Proposition 14.
Proof of Proposition 13. Assume A1, A2. In particular,  $\mathsf{M}$  contains no straight line, and Lemmas 9 and 11 apply here. We rewrite the ODE (3) as a Cauchy problem defined on Banach space  $\mathsf{E} = (\mathbb{R}^d\times \mathbb{R}^d,\| \cdot \|_{\mathsf{E}})$  where  $\| (a,b)\|_{\mathsf{E}} = \| a\| _2 + \| b\| _2$  for any pair  $(a,b)\in \mathbb{R}^d\times \mathbb{R}^d$ . We define  $\Omega = \mathsf{M}\times \mathbb{R}^{d}$ , which is an open set in  $\mathsf{E}$ . A solution  $(J,z)$  to this Cauchy problem is defined for all  $t\in J$  by
$$
\dot {z} (t) = F (z (t)), z (0) = z _ {0}, \quad \text {w h e r e} \quad F (x, p) = \left(\partial_ {p} H (x, p), - \partial_ {x} H (x, p)\right),
$$
such that  $J$  is an open interval of  $\mathbb{R}$  with  $0 \in J$ , and  $z = (x, p): J \to \Omega$  is differentiable on  $J$ . We now prove two results on  $F$  to establish existence and uniqueness of a maximal solution  $(J, z)$  for this Cauchy problem:
(a)  $F$  is continuously differentiable on  $\Omega$
(b)  $F$  is locally Lipschitz-continuous on  $\Omega$  with respect to  $\| \cdot \|_{\mathsf{E}}$
Since  $\nabla V, D\mathfrak{g} \in \mathrm{C}^1(\mathsf{M}, \mathbb{R})$ , we directly obtain Item (a). We now prove Item (b). Let  $\bar{z} = (\bar{x}, \bar{p}) \in \Omega$ . We endow  $\mathsf{E}$  with the norm  $\|\cdot\|_{\bar{z}}$ . Since  $\mathsf{M}$  is an open subset of  $\mathbb{R}^d$ , there exists  $0 < r \leq 1/(11C_{\bar{x}})$  where we recall that  $C_{\bar{x}}$  is defined in Lemma 12, such that  $\mathrm{B}_{\|\cdot\|_2}(\bar{x}, r) \subset \mathsf{M}$ . We consider such  $r$ , and we define  $\mathsf{B} = \mathrm{B}_{\|\cdot\|_{\mathsf{E}}}(\bar{z}, r)$  and  $\mathsf{B}_{\mathfrak{g}} = \mathrm{B}_{\|\cdot\|_{\bar{z}}}(\bar{z}, \bar{r})$ , where  $\bar{r} = C_{\bar{x}}r \leq 1/11$ . In particular, we have by Lemma 9-(b) that  $\mathsf{B}_{\mathfrak{g}} \subset \mathrm{B}_{\|\cdot\|_{\mathfrak{g}(\bar{x})}}(\bar{x}, \bar{r}) \times \mathrm{B}_{\|\cdot\|_{\mathfrak{g}(\bar{x})^{-1}}}(\bar{p}, \bar{r}) \subset \Omega$  since  $\bar{r} < 1$ . Moreover, using Lemma 12, we have  $\mathsf{B} \subset \mathsf{B}_{\mathfrak{g}} \subset \Omega$ .
Since  $V \in \mathrm{C}^2(\mathsf{M},\mathbb{R})$ ,  $\nabla V$  is Lipschitz-continuous on  $\mathsf{B}_{\mathfrak{g}}$ , i.e., there exists  $C_V > 0$  such that for any  $(z,z') \in \mathsf{B}_{\mathfrak{g}} \times \mathsf{B}_{\mathfrak{g}}$ ,
$$
\left\| \nabla V (x) - \nabla V \left(x ^ {\prime}\right) \right\| _ {\mathfrak {g} (\bar {x}) ^ {- 1}} \leq C _ {V} \| x - x ^ {\prime} \| _ {\mathfrak {g} (\bar {x})}. \tag {14}
$$
We first show that  $F$  is Lipschitz-continuous on  $\mathsf{B}_{\mathfrak{g}}$  with respect to  $\| \cdot \|_{\bar{z}}$ . Consider  $(z,z^{\prime})\in \mathsf{B}_{\mathfrak{g}}\times \mathsf{B}_{\mathfrak{g}}$  denoted by  $z = (x,p)$  and  $z^{\prime} = (x^{\prime},p^{\prime})$ . Note that  $(x,x^{\prime})\in W^{0}(\bar{x},1)\times W^{0}(\bar{x},1)$ , where  $W^{0}(\bar{x},1)$  is defined in Lemma 9. We first bound  $\| F^{(1)}(z) - F^{(1)}(z^{\prime})\|_{\mathfrak{g}(\bar{x})} = \| \partial_{p}H(z) - \partial_{p}H(z^{\prime})\|_{\mathfrak{g}(\bar{x})}$ . We have
$$
\partial_ {p} H (z) - \partial_ {p} H \left(z ^ {\prime}\right) = \mathfrak {g} (x) ^ {- 1} p - \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1} p ^ {\prime} = \mathfrak {g} (x) ^ {- 1} \left(p - p ^ {\prime}\right) + \left(\mathfrak {g} (x) ^ {- 1} - \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1}\right) p ^ {\prime},
$$
$$
\begin{array}{l} \left\| \partial_ {p} H (z) - \partial_ {p} H \left(z ^ {\prime}\right) \right\| _ {\mathfrak {g} (\bar {x})} \leq \left\| \mathfrak {g} (\bar {x}) ^ {1 / 2} \mathfrak {g} (x) ^ {- 1} \mathfrak {g} (\bar {x}) ^ {1 / 2} \right\| _ {2} \| p - p ^ {\prime} \| _ {\mathfrak {g} (\bar {x}) ^ {- 1}} \tag {15} \\ + \| \mathfrak {g} (\bar {x}) ^ {1 / 2} (\mathfrak {g} (x) ^ {- 1} - \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1}) \mathfrak {g} (\bar {x}) ^ {1 / 2} \| _ {2} \| p ^ {\prime} \| _ {\mathfrak {g} (\bar {x}) ^ {- 1}}. \\ \end{array}
$$
Using Lemma 9-(b), we have
$$
\left(1 - \| \bar {x} - x \| _ {\mathfrak {g} (x)}\right) ^ {2} \mathfrak {g} (\bar {x}) ^ {- 1} \preceq \mathfrak {g} (x) ^ {- 1} \preceq \left(1 - \| \bar {x} - x \| _ {\mathfrak {g} (x)}\right) ^ {- 2} \mathfrak {g} (\bar {x}) ^ {- 1},
$$
$$
\begin{array}{l} \left. \left(1 - \| \bar {x} - x \| _ {\mathfrak {g} (x)}\right) ^ {2} I _ {d} \preceq \mathfrak {g} (\bar {x}) ^ {1 / 2} \mathfrak {g} (x) ^ {- 1} \mathfrak {g} (\bar {x}) ^ {1 / 2} \preceq \left(1 - \| \bar {x} - x \| _ {\mathfrak {g} (x)}\right) ^ {- 2} I _ {d} \right. \\ (1 - \bar {r}) ^ {2} I _ {d} \prec \mathfrak {g} (\bar {x}) ^ {1 / 2} \mathfrak {g} (x) ^ {- 1} \mathfrak {g} (\bar {x}) ^ {1 / 2} \prec (1 - \bar {r}) ^ {- 2} I _ {d}. \\ \end{array}
$$
$$
\left\| \mathfrak {g} (\bar {x}) ^ {1 / 2} \mathfrak {g} (x) ^ {- 1} \mathfrak {g} (\bar {x}) ^ {1 / 2} \right\| _ {2} \leq (1 - \bar {r}) ^ {- 2}. \tag {16}
$$
$$
\begin{array}{l} \left\| x - x ^ {\prime} \right\| _ {\mathfrak {g} (x)} \leq \left(1 - \left\| \bar {x} - x ^ {\prime} \right\| _ {\mathfrak {g} (\bar {x})}\right) ^ {- 1} \left\| x - x \right\| _ {\mathfrak {g} (\bar {x})} \\ <   (1 - \bar {r}) ^ {- 1} 2 \bar {r} \\ <   1 / 5, \tag {17} \\ \end{array}
$$
where we used Lemma 9-(c) in the first inequality and  $\bar{r} \leq 1/11$  in the last inequality. Therefore,  $x' \in \mathsf{W}^0(x, 1)$  and we have using Lemma 9-(b)
$$
\left\{\left(1 - \left\| x ^ {\prime} - x \right\| _ {\mathfrak {g} (x)}\right) ^ {2} - 1 \right\} \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1} \preceq \mathfrak {g} (x) ^ {- 1} - \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1}
$$
$$
\preceq \left\{\left(1 - \| x ^ {\prime} - x \| _ {\mathfrak {g} (x)}\right) ^ {- 2} - 1 \right\} \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1}
$$
$$
\begin{array}{l} \{(1 - \| x ^ {\prime} - x \| _ {\mathfrak {g} (x)}) ^ {2} - 1 \} (1 - \bar {r}) ^ {2} I _ {d} \preceq \mathfrak {g} (\bar {x}) ^ {1 / 2} (\mathfrak {g} (x) ^ {- 1} - \mathfrak {g} (x ^ {\prime}) ^ {- 1}) \mathfrak {g} (\bar {x}) ^ {1 / 2} \\ \preceq \left\{\left(1 - \| x ^ {\prime} - x \| _ {\mathfrak {g} (x)}\right) ^ {- 2} - 1 \right\} \left(1 - \bar {r}\right) ^ {- 2} \mathrm {I} _ {d}. \\ \end{array}
$$
$$
\begin{array}{l} \left\| \mathfrak {g} (\bar {x}) ^ {1 / 2} \left(\mathfrak {g} (x) ^ {- 1} - \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1}\right) \mathfrak {g} (\bar {x}) ^ {1 / 2} \right\| _ {2} \tag {18} \\ \leq \max  \left(\left| \left(1 - \| x ^ {\prime} - x \| _ {\mathfrak {g} (x)}\right) ^ {2} - 1 \right| \left(1 - \bar {r}\right) ^ {2}, \left\{\left(1 - \| x ^ {\prime} - x \| _ {\mathfrak {g} (x)}\right) ^ {- 2} - 1 \right\} \left(1 - \bar {r}\right) ^ {- 2}\right) \\ \leq (1 - \bar {r}) ^ {- 2} \max  \left(\left| \left(1 - \| x ^ {\prime} - x \| _ {\mathfrak {g} (x)}\right) ^ {2} - 1 \right|, \left(1 - \| x ^ {\prime} - x \| _ {\mathfrak {g} (x)}\right) ^ {- 2} - 1\right). \\ \end{array}
$$
Using Inequalities (a) and (c) with  $u = \| x' - x\|_{\mathfrak{g}(x)}$ , where  $u \leq 1/5$  by (17), we obtain
(a)  $|(1 - \| x^{\prime} - x\|_{\mathfrak{g}(x)})^{2} - 1|\leq 3\| x^{\prime} - x\|_{\mathfrak{g}(x)} <   3(1 - \bar{r})^{-1}\| x - x^{\prime}\|_{\mathfrak{g}(\bar{x})}$
(b)  $(1 - \| x' - x\|_{\mathfrak{g}(x)})^{-2} - 1\leq 3\| x' - x\|_{\mathfrak{g}(x)} <   3(1 - \bar{r})^{-1}\| x - x'\|_{\mathfrak{g}(\bar{x})}$
$$
\left\| p ^ {\prime} \right\| _ {\mathfrak {g} (\bar {x}) ^ {- 1}} = \left\| p ^ {\prime} - \bar {p} + \bar {p} \right\| _ {\mathfrak {g} (\bar {x}) ^ {- 1}} \leq \bar {r} + \left\| \bar {p} \right\| _ {\mathfrak {g} (\bar {x}) ^ {- 1}}.
$$
Combining (15), (16) and (18), it comes that
$$
\left\| F ^ {(1)} (z) - F ^ {(1)} \left(z ^ {\prime}\right) \right\| _ {\mathfrak {g} (\bar {x})} \leq (1 - \bar {r}) ^ {- 2} \| p - p ^ {\prime} \| _ {\mathfrak {g} (\bar {x}) ^ {- 1}} + 3 (1 - \bar {r}) ^ {- 3} (\bar {r} + \| \bar {p} \| _ {\mathfrak {g} (\bar {x}) ^ {- 1}}) \| x - x ^ {\prime} \| _ {\mathfrak {g} (\bar {x})}. \tag {19}
$$
We define  $A_{\bar{x},1} = (1 - \bar{r})^{-2} + 3(1 - \bar{r})^{-3}(\bar{r} + \| \bar{p}\|_{\mathfrak{g}(\bar{x})^{-1}})$  such that  $\| F^{(1)}(z) - F^{(1)}(z')\|_{\mathfrak{g}(\bar{x})} \leq A_{\bar{x},1}\| z - z' \|_{\bar{z}}$  and we aim to bound  $\| F^{(2)}(z) - F^{(2)}(x')\|_{\mathfrak{g}(\bar{x})^{-1}} = \| \partial_xH(z) - \partial_xH(z')\|_{\mathfrak{g}(\bar{x})^{-1}}$ . We have
$$
\begin{array}{l} \partial_ {x} H (z) - \partial_ {x} H \left(z ^ {\prime}\right) = \nabla V (x) - \nabla V \left(x ^ {\prime}\right) \tag {20} \\ + \frac {1}{2} D \mathfrak {g} (x ^ {\prime}) [ \mathfrak {g} (x ^ {\prime}) ^ {- 1} p ^ {\prime}, \mathfrak {g} (x ^ {\prime}) ^ {- 1} p ^ {\prime} ] - \frac {1}{2} D \mathfrak {g} (x) [ \mathfrak {g} (x) ^ {- 1} p, \mathfrak {g} (x) ^ {- 1} p ] \\ + \frac {1}{2} \mathfrak {g} (x) ^ {- 1}: D \mathfrak {g} (x) - \frac {1}{2} \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1}: D \mathfrak {g} \left(x ^ {\prime}\right) \\ \end{array}
$$
$$
\begin{array}{l} \left\| D \mathfrak {g} \left(x ^ {\prime}\right) \left[ \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1} p ^ {\prime}, \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1} p ^ {\prime} \right] - D \mathfrak {g} (x) \left[ \mathfrak {g} (x) ^ {- 1} p, \mathfrak {g} (x) ^ {- 1} p \right] \right\| _ {\mathfrak {g} (\bar {x}) ^ {- 1}} \\ \leq \left\| D \mathfrak {g} (x) [ \mathfrak {g} (x) ^ {- 1} p, \mathfrak {g} (x) ^ {- 1} p ] - D \mathfrak {g} (x) [ \mathfrak {g} (x ^ {\prime}) ^ {- 1} p ^ {\prime}, \mathfrak {g} (x ^ {\prime}) ^ {- 1} p ^ {\prime} ] \right\| _ {\mathfrak {g} (\bar {x}) ^ {- 1}} \\ + \left\| D \mathfrak {g} (x) \left[ \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1} p ^ {\prime}, \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1} p ^ {\prime} \right] - D \mathfrak {g} \left(x ^ {\prime}\right) \left[ \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1} p ^ {\prime}, \mathfrak {g} \left(x ^ {\prime}\right) ^ {- 1} p ^ {\prime} \right] \right\| _ {\mathfrak {g} (\bar {x}) ^ {- 1}} \tag {21} \\ \end{array}
$$
Note that we have  $\mathfrak{g}(x)^{-1}p = F^{(1)}(z)$  (resp.  $\mathfrak{g}(x')^{-1}p' = F^{(1)}(z'))$ . Using Lemma 9-(c), the first upper bound in (21) is bounded by
$$
\begin{array}{l} (1 - \bar {r}) ^ {- 1} \| D \mathfrak {g} (x) [ F ^ {(1)} (z), F ^ {(1)} (z) ] - D \mathfrak {g} (x) [ F ^ {(1)} (z ^ {\prime}), F ^ {(1)} (z ^ {\prime}) ] \| _ {\mathfrak {g} (x) ^ {- 1}} \\ \leq 2 (1 - \bar {r}) ^ {- 1} \| D \mathfrak {g} (x) [ F ^ {(1)} (z ^ {\prime}) - F ^ {(1)} (z), F ^ {(1)} (z ^ {\prime}) ] \| _ {\mathfrak {g} (x) ^ {- 1}} \\ + (1 - \bar {r}) ^ {- 1} \| D \mathfrak {g} (x) [ F ^ {(1)} \left(z ^ {\prime}\right) - F ^ {(1)} (z), F ^ {(1)} \left(z ^ {\prime}\right) - F ^ {(1)} (z) ] \| _ {\mathfrak {g} (x) ^ {- 1}} \\ \leq 4 (1 - \bar {r}) ^ {- 1} \| F ^ {(1)} \left(z ^ {\prime}\right) - F ^ {(1)} (z) \| _ {\mathfrak {g} (x)} \| F ^ {(1)} \left(z ^ {\prime}\right) \| _ {\mathfrak {g} (x)} \\ + 2 (1 - \bar {r}) ^ {- 1} \| F ^ {(1)} \left(z ^ {\prime}\right) - F ^ {(1)} (z) \| _ {\mathfrak {g} (x)} ^ {2} (Lemma 9-(a)) \\ \leq 4 (1 - \bar {r}) ^ {- 3} \| F ^ {(1)} (z ^ {\prime}) - F ^ {(1)} (z) \| _ {\mathfrak {g} (\bar {x})} \| F ^ {(1)} (z ^ {\prime}) \| _ {\mathfrak {g} (\bar {x})} \\ + 2 (1 - \bar {r}) ^ {- 3} \| F ^ {(1)} \left(z ^ {\prime}\right) - F ^ {(1)} (z) \| _ {\mathfrak {g} (\bar {x})} ^ {2} (Lemma 9-(c)) \\ \leq 4 (1 - \bar {r}) ^ {- 3} A _ {\bar {x}, 1} \| z - z ^ {\prime} \| _ {\bar {z}} \| F ^ {(1)} \left(z ^ {\prime}\right) \| _ {\mathfrak {g} (\bar {x})} + 2 (1 - \bar {r}) ^ {- 3} A _ {\bar {x}, 1} ^ {2} \| z - z ^ {\prime} \| _ {\bar {z}} ^ {2} \quad \text {(s e e (1 9))} \\ \leq \left\{4 A _ {\bar {x}, 1} (1 - \bar {r}) ^ {- 5} (\bar {r} + \| \bar {p} \| _ {\mathfrak {g} (\bar {x}) ^ {- 1}}) + 4 (1 - \bar {r}) ^ {- 3} A _ {\bar {x}, 1} ^ {2} \bar {r} \right\} \| z - z ^ {\prime} \| _ {\bar {z}}, \\ \end{array}
$$
where we used that  $\| F^{(1)}(z')\|_{\mathfrak{g}(\bar{x})}\leq (1 - \bar{r})^{-2}(\bar{r} +\| \bar{p}\|_{\mathfrak{g}(\bar{x})^{-1}})$  in the last inequality. Combining Lemma 9-(c) and Lemma 11-(c), the second upper bound in (21) is bounded by