SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[49] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015. Software available from tensorflow.org.
[50] J. Sohl-Dickstein. Two Equalities Expressing the Determinant of a Matrix in terms of Expectations over Matrix-Vector Products, 2020.
[51] K. P. Murphy. Probabilistic Machine Learning: Advanced Topics, page 811. MIT Press, 2023.
[52] M. Zhang, O. Key, P. Hayes, D. Barber, B. Paige, and F-X. Briol. Towards Healing the Blindness of Score Matching. Workshop on Score-Based Methods at Conf. on Neural Information Processing Systems (NeurIPS), 2022.
A.1 Derivations
In the following subsections, we provide theoretical derivations. In Section A.1.1, we discuss the asymptotic convergence properties as well as the assumptions of score-matching methods. In Section A.1.2, we elaborate on the formulation of EBFlow (i.e., Eqs. (8) and (9)), and provide a explanation of their interpretation. Finally, in Section A.1.3, we present a theoretical analysis of KL divergence and Fisher divergence, and discuss the underlying mechanism behind the proposed MaP technique.
A.1.1 Asymptotic Convergence Property of Score Matching
In this subsection, we provide a formal description of the consistency property of score matching. The description follows [16] and the notations are replaced with those used in this paper. The regularity conditions for  $p(\cdot ;\theta)$  are defined in Assumptions A.1~A.7. In the following paragraph, the parameter space is defined as  $\Theta$ . In addition,  $s(\pmb {x};\theta)\triangleq \frac{\partial}{\partial\pmb{x}}\log p(\pmb {x};\theta) = -\frac{\partial}{\partial\pmb{x}} E(\pmb {x};\theta)$  represents the score function.  $\hat{\mathcal{L}}_{\mathrm{SM}}(\theta)\triangleq \frac{1}{N}\sum_{k = 1}^{N}f(\mathbf{x}_k;\theta)$  denotes an unbiased estimator of  $\mathcal{L}_{\mathrm{SM}}(\theta)$ , where  $f(\pmb {x};\theta)\triangleq \frac{1}{2}\left\| \frac{\partial}{\partial\pmb{x}} E(\pmb {x};\theta)\right\| ^2 -\operatorname {Tr}\left(\frac{\partial^2}{\partial\pmb{x}^2} E(\pmb {x};\theta)\right) = \frac{1}{2}\left\| s(\pmb {x};\theta)\right\| ^2 +\operatorname {Tr}\left(\frac{\partial}{\partial\pmb{x}} s(\pmb {x};\theta)\right)$  and  $\{\mathbf{x}_1,\dots ,\mathbf{x}_N\}$  represents a collection of i.i.d. samples drawn from  $p_{\mathbf{x}}$ . For notational simplicity, we denote  $\partial h(\pmb {x};\theta)\triangleq \frac{\partial}{\partial\pmb{x}} h(\pmb {x};\theta)$  and  $\partial_ih_j(\pmb {x};\theta)\triangleq \frac{\partial}{\partial x_i} h_j(\pmb {x};\theta)$ , where  $h_j(\pmb {x};\theta)$  denotes the  $j$ -th element of  $h$ .
Assumption A.1. (Positiveness)  $p(\pmb{x};\theta) > 0$  and  $p_{\mathbf{x}}(\pmb{x}) > 0, \forall \theta \in \Theta, \forall \pmb{x} \in \mathbb{R}^D$ .
Assumption A.2. (Regularity of the score functions) The parameterized score function  $s(\pmb{x};\theta)$  and the true score function  $\frac{\partial}{\partial\pmb{x}}\log p_{\mathbf{x}}(\pmb{x})$  are both continuous and differentiable. In addition, their expectations  $\mathbb{E}_{p_{\mathbf{x}}(\pmb{x})}[s(\pmb{x};\theta)]$  and  $\mathbb{E}_{p_{\mathbf{x}}(\pmb{x})}\left[\frac{\partial}{\partial\pmb{x}}\log p_{\mathbf{x}}(\pmb{x})\right]$  are finite. (i.e.,  $\mathbb{E}_{p_{\mathbf{x}}(\pmb{x})}[s(\pmb{x};\theta)] < \infty$  and  $\mathbb{E}_{p_{\mathbf{x}}(\pmb{x})}\left[\frac{\partial}{\partial\pmb{x}}\log p_{\mathbf{x}}(\pmb{x})\right] < \infty$ )
Assumption A.3. (Boundary condition)  $\lim_{\| \pmb {x}\| \to \infty}p_{\mathbf{x}}(\pmb {x})s(\pmb {x};\theta) = 0,\forall \theta \in \Theta .$
Assumption A.4. (Compactness) The parameter space  $\Theta$  is compact.
Assumption A.5. (Identifiability) There exists a set of parameters  $\theta^{*}$  such that  $p_{\mathbf{x}}(\boldsymbol {x}) = p(\boldsymbol {x};\theta^{*})$  where  $\theta^{*}\in \Theta ,\forall \boldsymbol {x}\in \mathbb{R}^{D}$
Assumption A.6. (Uniqueness)  $\theta \neq \theta^{*} \Leftrightarrow p(\pmb{x};\theta) \neq p(\pmb{x};\theta^{*})$ , where  $\theta, \theta^{*} \in \Theta$ ,  $\pmb{x} \in \mathbb{R}^{D}$ .
Assumption A.7. (Lipschitzness of  $f$ ) The function  $f$  is Lipschitz continuous w.r.t.  $\theta$ , i.e.,  $|f(\pmb{x};\theta_1) - f(\pmb{x};\theta_2)| \leq L(\pmb{x})\|\theta_1 - \theta_2\|_2$ ,  $\forall \theta_1, \theta_2 \in \Theta$ , where  $L(\pmb{x})$  represents a Lipschitz constant satisfying  $\mathbb{E}_{p_{\pmb{x}}(\pmb{x})}[L(\pmb{x})] < \infty$ .
Theorem A.8. (Consistency of a score-matching estimator [16]) The score-matching estimator  $\theta_N \triangleq \operatorname{argmin}_{\theta \in \Theta} \hat{\mathcal{L}}_{\mathrm{SM}}$  is consistent, i.e.,
$$
\theta_ {N} \xrightarrow {p} \theta^ {*}, a s N \to \infty .
$$
Assumptions A.1~A.3 are the conditions that ensure  $\frac{\partial}{\partial\theta}\mathbb{D}_{\mathrm{F}}[p_{\mathbf{x}}(\boldsymbol {x})\| p(\boldsymbol {x};\theta)] = \frac{\partial}{\partial\theta}\mathcal{L}_{\mathrm{SM}}(\theta)$ . Assumptions A.4~A.7 lead to the uniform convergence property [16] of a score-matching estimator, which gives rise to the consistency property. The detailed derivation can be found in Corollary 1 in [16]. In the following Lemma A.9 and Proposition A.10, we examine the sufficient condition for  $g$  and  $p_{\mathbf{u}}$  to satisfy Assumption A.7.
Lemma A.9. (Sufficient condition for the Lipschitzness of  $f$ ) The function  $f(\pmb{x};\theta) = \frac{1}{2}\| s(\pmb{x};\theta)\|^2 + \mathrm{Tr}\left(\frac{\partial}{\partial\pmb{x}} s(\pmb{x};\theta)\right)$  is Lipschitz continuous if the score function  $s(\pmb{x};\theta)$  satisfies the following conditions:  $\forall \theta, \theta_1, \theta_2 \in \Theta, \forall i \in \{1,\dots ,D\}$ ,
$$
\left\| s (\boldsymbol {x}; \theta) \right\| _ {2} \leq L _ {1} (\boldsymbol {x}),
$$
$$
\left\| s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \leq L _ {2} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2},
$$
$$
\left\| \partial_ {i} s (\boldsymbol {x}; \theta_ {1}) - \partial_ {i} s (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \leq L _ {3} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2},
$$
where  $L_{1}$ ,  $L_{2}$ , and  $L_{3}$  are Lipschitz constants satisfying  $\mathbb{E}_{p_{\mathbf{x}}(\boldsymbol{x})}[L_1(\boldsymbol{x})] < \infty$ ,  $\mathbb{E}_{p_{\mathbf{x}}(\boldsymbol{x})}[L_2(\boldsymbol{x})] < \infty$ , and  $\mathbb{E}_{p_{\mathbf{x}}(\boldsymbol{x})}[L_3(\boldsymbol{x})] < \infty$ .
Proof. The Lipschitzness of  $f$  can be guaranteed by ensuring the Lipschitzness of  $\| s(\pmb{x};\theta)\|_2^2$  and  $\operatorname{Tr}(\partial s(\pmb{x};\theta))$ .
Step 1. (Lipschitzness of  $\| s(\pmb {x};\theta)\| _2^2$
$$
\begin{array}{l} \left| \left\| s (\boldsymbol {x}; \theta_ {1}) \right\| _ {2} ^ {2} - \left\| s (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} ^ {2} \right| \\ = \left| s (\boldsymbol {x}; \theta_ {1}) ^ {T} s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {2}) ^ {T} s (\boldsymbol {x}; \theta_ {2}) \right| \\ = \left| \left(s (\boldsymbol {x}; \theta_ {1}) ^ {T} s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {1}) ^ {T} s (\boldsymbol {x}; \theta_ {2})\right) + \left(s (\boldsymbol {x}; \theta_ {1}) ^ {T} s (\boldsymbol {x}; \theta_ {2}) - s (\boldsymbol {x}; \theta_ {2}) ^ {T} s (\boldsymbol {x}; \theta_ {2})\right) \right| \\ = \left| s (\boldsymbol {x}; \theta_ {1}) ^ {T} (s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {2})) + s (\boldsymbol {x}; \theta_ {2}) ^ {T} (s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {2})) \right| \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i)} {\leq} \left| s (\boldsymbol {x}; \theta_ {1}) ^ {T} (s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {2})) \right| + \left| s (\boldsymbol {x}; \theta_ {2}) ^ {T} (s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {2})) \right| \\ \stackrel {(i i)} {\leq} \| s (\boldsymbol {x}; \theta_ {1}) \| _ {2} \| s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {2}) \| _ {2} + \| s (\boldsymbol {x}; \theta_ {2}) \| _ {2} \| s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {2}) \| _ {2} \\ \stackrel {(i i i)} {\leq} L _ {1} (\boldsymbol {x}) \left\| s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} + L _ {1} (\boldsymbol {x}) \left\| s (\boldsymbol {x}; \theta_ {1}) - s (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \\ \stackrel {(i i i)} {\leq} 2 L _ {1} (\boldsymbol {x}) L _ {2} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2}, \\ \end{array}
$$
where  $(i)$  is based on triangle inequality,  $(ii)$  is due to Cauchy-Schwarz inequality, and  $(iii)$  follows from the listed assumptions.
Step 2. (Lipschitzness of  $\operatorname{Tr}(\partial s(x;\theta)))$
$$
\begin{array}{l} \left| \operatorname {T r} \left(\partial s (\boldsymbol {x}; \theta_ {1})\right) - \operatorname {T r} \left(\partial s (\boldsymbol {x}; \theta_ {2})\right) \right| = \left| \operatorname {T r} \left(\partial s (\boldsymbol {x}; \theta_ {1}) - \partial s (\boldsymbol {x}; \theta_ {2})\right) \right| \\ \stackrel {(i)} {\leq} D \left\| \partial s (\boldsymbol {x}; \theta_ {1}) - \partial s (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \\ \stackrel {(i i)} {\leq} D \sqrt {\sum_ {i} \| \partial_ {i} s (\boldsymbol {x} ; \theta_ {1}) - \partial_ {i} s (\theta_ {2}) \| _ {2} ^ {2}} \\ \stackrel {(i i i)} {\leq} D \sqrt {D L _ {3} ^ {2} (\boldsymbol {x}) \| \theta_ {1} - \theta_ {2} \| _ {2} ^ {2}} \\ = D \sqrt {D} L _ {3} (\boldsymbol {x}) \| \theta_ {1} - \theta_ {2} \| _ {2} \\ \end{array}
$$
where  $(i)$  holds by Von Neumann's trace inequality.  $(ii)$  is due to the property  $\| A\| _2\leq \sqrt{\sum_i\|a_i\|_2^2}$ , where  $\pmb{a}_i$  is the column vector of  $A$ .  $(iii)$  holds by the listed assumptions.
Based on Steps 1 and 2, the Lipschitzness of  $f$  is guaranteed, since
$$
\begin{array}{l} \left| f (\boldsymbol {x}; \theta_ {1}) - f (\boldsymbol {x}; \theta_ {2}) \right| = \left| \frac {1}{2} \| s (\boldsymbol {x}; \theta_ {1}) \| ^ {2} + \operatorname {T r} \left(\frac {\partial}{\partial \boldsymbol {x}} s (\boldsymbol {x}; \theta_ {1})\right) - \frac {1}{2} \| s (\boldsymbol {x}; \theta_ {2}) \| ^ {2} - \operatorname {T r} \left(\frac {\partial}{\partial \boldsymbol {x}} s (\boldsymbol {x}; \theta_ {2})\right) \right| \\ = \left| \frac {1}{2} \| s (\boldsymbol {x}; \theta_ {1}) \| ^ {2} - \frac {1}{2} \| s (\boldsymbol {x}; \theta_ {2}) \| ^ {2} + \operatorname {T r} \left(\frac {\partial}{\partial \boldsymbol {x}} s (\boldsymbol {x}; \theta_ {1})\right) - \operatorname {T r} \left(\frac {\partial}{\partial \boldsymbol {x}} s (\boldsymbol {x}; \theta_ {2})\right) \right| \\ \leq \frac {1}{2} \left| \| s (\boldsymbol {x}; \theta_ {1}) \| ^ {2} - \| s (\boldsymbol {x}; \theta_ {2}) \| ^ {2} \right| + \left| \operatorname {T r} \left(\frac {\partial}{\partial \boldsymbol {x}} s (\boldsymbol {x}; \theta_ {1})\right) - \operatorname {T r} \left(\frac {\partial}{\partial \boldsymbol {x}} s (\boldsymbol {x}; \theta_ {2})\right) \right| \\ \leq L _ {1} (\boldsymbol {x}) L _ {2} (\boldsymbol {x}) \| \theta_ {1} - \theta_ {2} \| _ {2} + D \sqrt {D} L _ {3} (\boldsymbol {x}) \| \theta_ {1} - \theta_ {2} \| _ {2} \\ = \left(L _ {1} (\boldsymbol {x}) L _ {2} (\boldsymbol {x}) + D \sqrt {D} L _ {3} (\boldsymbol {x})\right) \| \theta_ {1} - \theta_ {2} \| _ {2}. \\ \end{array}
$$
Proposition A.10. (Sufficient condition for the Lipschitzness of  $f$ ) The function  $f$  is Lipschitz continuous if  $g(\pmb{x};\theta)$  has bounded first, second, and third-order derivatives, i.e.,  $\forall i,j \in \{1,\dots ,D\}$ ,  $\forall \theta \in \Theta$ .
$$
\left\| \mathbf {J} _ {g} (\boldsymbol {x}; \theta) \right\| _ {2} \leq l _ {1} (\boldsymbol {x}), \left\| \partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x}; \theta) \right\| _ {2} \leq l _ {2} (\boldsymbol {x}), \left\| \partial_ {i} \partial_ {j} \mathbf {J} _ {g} (\boldsymbol {x}; \theta) \right\| _ {2} \leq l _ {3} (\boldsymbol {x}),
$$
and smooth enough on  $\Theta$ , i.e.,  $\theta_{1}, \theta_{2} \in \Theta$ :
$$
\left\| g (\boldsymbol {x}; \theta_ {1}) - g (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \leq r _ {0} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2},
$$
$$
\left\| \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {1}) - \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \leq r _ {1} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2},
$$
$$
\left\| \partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {1}) - \partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \leq r _ {2} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2}.
$$
$$
\left\| \partial_ {i} \partial_ {j} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {1}) - \partial_ {i} \partial_ {j} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \leq r _ {3} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2}.
$$
In addition, it satisfies the following conditions:
$$
\left\| \mathbf {J} _ {g} ^ {- 1} (\pmb {x}; \theta) \right\| _ {2} \leq l _ {1} ^ {\prime} (\pmb {x}), \left\| \partial_ {i} \mathbf {J} _ {g} ^ {- 1} (\pmb {x}; \theta) \right\| _ {2} \leq l _ {2} ^ {\prime} (\pmb {x}),
$$
$$
\left\| \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x}; \theta_ {1}) - \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \leq r _ {1} ^ {\prime} (\boldsymbol {x}) \| \theta_ {1} - \theta_ {2} \| _ {2},
$$
$$
\left\| \partial_ {i} \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x}; \theta_ {1}) - \partial_ {i} \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \leq r _ {2} ^ {\prime} (\boldsymbol {x}) \| \theta_ {1} - \theta_ {2} \| _ {2},
$$
where  $\mathbf{J}_g^{-1}$  represents the inverse matrix of  $\mathbf{J}_g$ . Furthermore, the prior distribution  $p_{\mathbf{u}}$  satisfies:
$$
\left\| s _ {\mathbf {u}} (\boldsymbol {u}) \right\| \leq t _ {1}, \left\| \partial_ {i} s _ {\mathbf {u}} (\boldsymbol {u}) \right\| \leq t _ {2}
$$
$$
\left\| s _ {\mathbf {u}} \left(\boldsymbol {u} _ {1}\right) - s _ {\mathbf {u}} \left(\boldsymbol {u} _ {2}\right) \right\| _ {2} \leq t _ {3} \left\| \boldsymbol {u} _ {1} - \boldsymbol {u} _ {2} \right\| _ {2},
$$
$$
\left\| \partial_ {i} s _ {\mathbf {u}} (\boldsymbol {u} _ {1}) - \partial_ {i} s _ {\mathbf {u}} (\boldsymbol {u} _ {2}) \right\| _ {2} \leq t _ {4} \left\| \boldsymbol {u} _ {1} - \boldsymbol {u} _ {2} \right\| _ {2},
$$
where  $s_{\mathbf{u}}(\pmb{u}) \triangleq \frac{\partial}{\partial \pmb{u}} \log p_{\mathbf{u}}(\pmb{u})$  is the score function of  $p_{\mathbf{u}}$ . The Lipschitz constants listed above (i.e.,  $l_1 \sim l_3$ ,  $r_0 \sim r_3$ ,  $l_1' \sim l_2'$ , and  $r_1' \sim r_2'$ ) have finite expectations.
Proof. We show that the sufficient conditions stated in Lemma A.9 can be satisfied using the conditions listed above.
Step 1. (Sufficient condition of  $\| s(\pmb {x};\theta)\| _2\leq L_1(\pmb {x}))$
Since  $\| s(\pmb{x};\theta)\|_2 = \left\|\frac{\partial}{\partial\pmb{x}}\log p_{\mathbf{u}}(g(\pmb{x};\theta)) + \frac{\partial}{\partial\pmb{x}}\log |\det \mathbf{J}_g(\pmb{x};\theta)|\right\|_2 \leq \left\|\frac{\partial}{\partial\pmb{x}}\log p_{\mathbf{u}}(g(\pmb{x};\theta))\right\|_2 + \left\|\frac{\partial}{\partial\pmb{x}}\log |\det \mathbf{J}_g(\pmb{x};\theta)|\right\|_2$  we first demonstrate that  $\left\|\frac{\partial}{\partial\pmb{x}}\log p_{\mathbf{u}}(g(\pmb{x};\theta))\right\|_2$  and  $\left\|\frac{\partial}{\partial\pmb{x}}\log |\det \mathbf{J}_g(\pmb{x};\theta)|\right\|_2$  are both bounded.
(1.1)  $\left\| \frac{\partial}{\partial \mathbf{x}} \log p_{\mathbf{u}}(g(\mathbf{x}; \theta)) \right\|_2$  is bounded:
$$
\left\| \frac {\partial}{\partial \boldsymbol {x}} \log p _ {\mathbf {u}} (g (\boldsymbol {x}; \theta)) \right\| _ {2} = \left\| (s _ {\mathbf {u}} (g (\boldsymbol {x}; \theta))) ^ {T} \mathbf {J} _ {g} (\boldsymbol {x}; \theta) \right\| _ {2} \leq \| s _ {\mathbf {u}} (g (\boldsymbol {x}; \theta)) \| _ {2} \| \mathbf {J} _ {g} (\boldsymbol {x}; \theta) \| _ {2} \leq t _ {1} l _ {1} (\boldsymbol {x}).
$$
(1.2)  $\left\| \frac{\partial}{\partial x} \log |\operatorname{det} \mathbf{J}_g(\boldsymbol{x}; \theta)| \right\|$  is bounded:
$$
\begin{array}{l} \left\| \frac {\partial}{\partial \boldsymbol {x}} \log | \det  \mathbf {J} _ {g} (\boldsymbol {x}; \theta) | \right\| = \left\| | \det  \mathbf {J} _ {g} (\boldsymbol {x}; \theta) | ^ {- 1} \frac {\partial}{\partial \boldsymbol {x}} | \det  \mathbf {J} _ {g} (\boldsymbol {x}; \theta) | \right\| \\ = \left\| (\det  \mathbf {J} _ {g} (\boldsymbol {x}; \theta)) ^ {- 1} \frac {\partial}{\partial \boldsymbol {x}} \det  \mathbf {J} _ {g} (\boldsymbol {x}; \theta) \right\| \\ \stackrel {(i)} {=} \left\| (\det  \mathbf {J} _ {g} (\boldsymbol {x}; \theta)) ^ {- 1} \det  \mathbf {J} _ {g} (\boldsymbol {x}; \theta) \boldsymbol {v} (\boldsymbol {x}; \theta) \right\| \\ = \| \boldsymbol {v} (\boldsymbol {x}; \theta) \|, \\ \end{array}
$$
where  $(i)$  is derived using Jacobi's formula, and  $\pmb{v}_i(\pmb{x};\theta) = \mathrm{Tr}\left(\mathbf{J}_g^{-1}(\pmb{x};\theta)\partial_i\mathbf{J}_g(\pmb{x};\theta)\right)$ .
$$
\begin{array}{l} \left\| \boldsymbol {v} (\boldsymbol {x}; \theta) \right\| = \sqrt {\sum_ {i} \left(\operatorname {T r} \left(\mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x} ; \theta) \partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x} ; \theta)\right)\right) ^ {2}} \\ \stackrel {(i)} {\leq} \sqrt {\sum_ {i} D ^ {2} \left\| \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x} ; \theta) \partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x} ; \theta) \right\| _ {2} ^ {2}} \\ \stackrel {(i i)} {\leq} \sqrt {\sum_ {i} D ^ {2} \left\| \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x} ; \theta) \right\| _ {2} ^ {2} \left\| \partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x} ; \theta) \right\| _ {2} ^ {2}} \\ \stackrel {(i i i)} {\leq} \sqrt {\sum_ {i} D ^ {2} l _ {1} ^ {\prime 2} (\boldsymbol {x}) l _ {2} ^ {2} (\boldsymbol {x})} \\ = \sqrt {D ^ {3}} l _ {1} ^ {\prime} (\boldsymbol {x}) l _ {2} (\boldsymbol {x}), \\ \end{array}
$$
where  $(i)$  holds by Von Neumann's trace inequality,  $(ii)$  is due to the property of matrix norm, and  $(iii)$  is follows from the listed assumptions.
Step 2. (Sufficient condition of the Lipschitzness of  $s(\pmb{x};\theta)$ )
Since  $s(\pmb{x};\theta) = \frac{\partial}{\partial\pmb{x}}\log p_{\mathbf{u}}(g(\pmb{x};\theta)) + \frac{\partial}{\partial\pmb{x}}\log |\operatorname*{det}\mathbf{J}_g(\pmb{x};\theta)|$ , we demonstrate that  $\frac{\partial}{\partial x}\log p_{\mathbf{u}}(g(\pmb{x};\theta))$  and  $\frac{\partial}{\partial x}\log |\operatorname*{det}\mathbf{J}_g(\pmb{x};\theta)|$  are both Lipschitz continuous on  $\Theta$ .
(2.1) Lipschitzness of  $\frac{\partial}{\partial\pmb{x}}\log p_{\mathbf{u}}(g(\pmb {x};\theta))$
$$
\begin{array}{l} \left\| \frac {\partial}{\partial \boldsymbol {x}} \log p _ {\mathbf {u}} (g (\boldsymbol {x}; \theta_ {1})) - \frac {\partial}{\partial \boldsymbol {x}} \log p _ {\mathbf {u}} (g (\boldsymbol {x}; \theta_ {2})) \right\| _ {2} \\ = \left\| \left(s _ {\mathbf {u}} \left(g (\boldsymbol {x}; \theta_ {1})\right)\right) ^ {T} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {1}) - \left(s _ {\mathbf {u}} \left(g (\boldsymbol {x}; \theta_ {2})\right)\right) ^ {T} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i)} {\leq} \| s _ {\mathbf {u}} (g (\boldsymbol {x}; \theta_ {1})) \| _ {2} \| \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {1}) - \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {2}) \| _ {2} + \| s _ {\mathbf {u}} (g (\boldsymbol {x}; \theta_ {1})) - s _ {\mathbf {u}} (g (\boldsymbol {x}; \theta_ {2})) \| _ {2} \| \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {2}) \| _ {2} \\ \stackrel {(i i)} {\leq} t _ {1} r _ {1} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2} + t _ {2} l _ {1} (\boldsymbol {x}) \left\| g (\boldsymbol {x}; \theta_ {1}) - g (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \\ \stackrel {(i i)} {\leq} t _ {1} r _ {1} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2} + t _ {2} l _ {1} (\boldsymbol {x}) r _ {0} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2} \\ = \left(t _ {1} r _ {1} (\boldsymbol {x}) + t _ {2} l _ {1} (\boldsymbol {x}) r _ {0} (\boldsymbol {x})\right) \| \theta_ {1} - \theta_ {2} \| _ {2}, \\ \end{array}
$$
where  $(i)$  is obtained using a similar derivation to Step 1 in Lemma A.9, while  $(ii)$  follows from the listed assumptions.
(2.2) Lipschitzness of  $\frac{\partial}{\partial x}\log |\operatorname*{det}\mathbf{J}_g(\boldsymbol {x};\theta)|$
Let  $\mathbf{M}(i,\pmb {x};\theta)\triangleq \mathbf{J}_g^{-1}(\pmb {x};\theta_1)\partial_i\mathbf{J}_g(\pmb {x};\theta)$ . We first demonstrate that  $\mathbf{M}$  is Lipschitz continuous:
$$
\begin{array}{l} \left\| \mathbf {M} (i, \boldsymbol {x}; \theta_ {1}) - \mathbf {M} (i, \boldsymbol {x}; \theta_ {2}) \right\| _ {2} \\ = \left\| \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x}; \theta_ {1}) \partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {1}) - \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x}; \theta_ {2}) \partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \\ \stackrel {(i)} {\leq} \left\| \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x}; \theta_ {1}) \right\| _ {2} \left\| \left(\partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {1}) - \partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {2})\right) \right\| _ {2} + \left\| \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x}; \theta_ {1}) - \mathbf {J} _ {g} ^ {- 1} (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \left\| \partial_ {i} \mathbf {J} _ {g} (\boldsymbol {x}; \theta_ {2}) \right\| _ {2} \\ \stackrel {(i i)} {\leq} l _ {1} ^ {\prime} (\boldsymbol {x}) r _ {2} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2} + l _ {2} (\boldsymbol {x}) r _ {1} ^ {\prime} (\boldsymbol {x}) \left\| \theta_ {1} - \theta_ {2} \right\| _ {2} \\ = \left(l _ {1} ^ {\prime} (\boldsymbol {x}) r _ {2} (\boldsymbol {x}) + l _ {2} (\boldsymbol {x}) r _ {1} ^ {\prime} (\boldsymbol {x})\right) \| \theta_ {1} - \theta_ {2} \| _ {2}, \\ \end{array}
$$
where  $(i)$  is obtained by an analogous derivation of the step 1 in Lemma A.9, and  $(ii)$  holds by the listed assumption.
The Lipschitzness of  $\mathbf{M}$  leads to the Lipschitzness of  $\frac{\partial}{\partial x} \log |\det \mathbf{J}_g(\boldsymbol{x}; \theta)|$ , since: