SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L _ {q _ {0}, \epsilon} = \sup  _ {\| y - x \| \leq \epsilon} \| y \| \leq \| x \| + \epsilon . \tag {12}
$$
The proof is then complete by directly applying Eqs. (11) and (12) to the error bound stated in Theorem 4.3.  $\square$
Corollary C.2. Let  $q$  be either HVAE or HamFlow of  $N$  layers with  $q_0 = \mathcal{N}(0, I_{2d})$ . Suppose the forward dynamics has the  $(\epsilon, \delta)$ -shadowing property, and  $\xi_0 = q_0$ . Then
$$
\left| \operatorname {E L B O} (q | | \bar {p}) - \mathbb {E} [ \hat {E} (X) ] \right| \leq \epsilon \cdot \left(\mathbb {E} \left[ L _ {\bar {p}, \epsilon} (\hat {X} _ {N}) \right] + \sqrt {2 d}\right) + \epsilon^ {2} \quad f o r X \sim q _ {0}.
$$
If additionally  $\log p$  is  $L$ -smooth (i.e.,  $\forall x, y, \| \nabla \log p(x) - \nabla \log p(y) \| \leq L \| x - y \|$ ), then
$$
\left| \operatorname {E L B O} (q | | \bar {p}) - \mathbb {E} [ \hat {E} (X) ] \right| \leq \epsilon \cdot \left(\mathbb {E} \left[ \| \nabla \log \bar {p} (\hat {X} _ {N}) \| \right] + \sqrt {2 d}\right) + (L \vee 1 + 1) \epsilon^ {2} \quad f o r X \sim q _ {0}.
$$
Proof of Corollary C.2. Similar to the proof of Fig. 2d, applying Eqs. (11) and (12) to the error bound stated in Theorem 4.4 yields that
$$
\left| \operatorname {E L B O} (q \| \bar {p}) - \mathbb {E} [ \hat {E} (X) ] \right| \leq \epsilon \cdot \mathbb {E} \left[ L _ {\bar {p}, \epsilon} (\hat {X} _ {N}) + \| X \| \right] + \epsilon^ {2} \quad \text {f o r} X \sim q _ {0}.
$$
The first result is then obtained by employing the following bound via Jensen's inequality:
$$
\mathbb {E} [ \| X \| ] \leq \sqrt {\mathbb {E} [ \| X \| ^ {2} ]} = \sqrt {2 d} \quad \text {f o r} X \sim q _ {0} = \mathcal {N} (0, I _ {2 d}).
$$
To arrive at the second result, we apply the following bound on  $L_{p,\epsilon}$  using the fact that  $\nabla \log p$  is  $L$ -Lipschitz continuous:
$$
L _ {\bar {p}, \epsilon} \leq \| \nabla \log \bar {p} (x) \| + \sup  _ {\| y - x \| \leq \epsilon} \| \nabla \log \bar {p} (x) - \nabla \log \bar {p} (y) \| \leq \| \nabla \log \bar {p} (x) \| + L \vee 1 \cdot \epsilon .
$$
C.2 Empirical results for Hamiltonian flow
In this section, we empirically validate our error bounds and the diagnostic procedure of HamFlow using two synthetic targets: the banana and cross distributions. We employ a warm-started HamFlow, as detailed in [6, Section 3.4]. This is achieved through a single pass of 10,000 samples from the reference distribution,  $q_{0}$ , without additional optimization. Specifically, the  $\mu$  and  $\Lambda$  values for each flow standardization layer are determined by the sample mean and the Cholesky factorization of the sample covariance, respectively, of the batched momentum samples processed through the layer. For the banana distribution, we set the leapfrog step  $K$  at 200 and a step size  $\epsilon$  of 0.01. For the cross distribution, we set  $K$  to 100 with an  $\epsilon$  value of 0.02. Figs. 8a
and 8b showcase a comparison between 1000 i.i.d. samples from the warm-started HamFlow with 400 layers and the true targets. The results illustrate that HamFlow generates samples of decent quality.
We start by examining the numerical orbit computaton error. As demonstrated in Fig. 9, the pointwise evaluation error grows exponentially with increasing flow depth, and quickly reaches the same order of magnitude as the scale of the target distributions. However, similar to what we have observed in the experiments of MixFlow (described in Section 6), the substantial numerical error of orbit computation does not impact the accuracy of Monte Carlo estimations of expectations and ELBO estimations significantly. Fig. 10 shows the relative numerical error of Monte Carlo estimates of the expectations of three test functions. The relative samples estimation error remains within  $5\%$  for the two bounded Lipschitz continuous test functions, aligning with our error bound in Proposition 4.2. Moreover, even for the unbounded test function, the error stays within reasonable limits:  $10\%$  for the banana and  $5\%$  for the cross examples. Fig. 11 then compares the numerical ELBO estimates and exact ELBO estimates for HamFlow over increasing flow length. In both examples, the numerical ELBO curve is almost identical to the exact one, notwithstanding the considerable sample computation error of the forward orbit.
However, unlike the small log-density evaluation error we observed in the MixFlow experiments, HamFlow's log-density shows significant issues because of the numerical errors of backward orbits computation. In Fig. 12, we measure the numerical error for the log-density evaluation of HamFlow at 100 i.i.d. points from the target distribution. As we add more flow layers, the error grows quickly. For the banana example, the log-density evaluation error can even go above 1000. This raises two main questions: (1) Why is the ELBO estimation error different from the log-density evaluation error in HamFlow? (2) Why is the scaling of the log-density evaluation error different between MixFlow and HamFlow? Our theory presented in Section 4 and Appendix C.1 offers explanation to these questions.
To address the first question, we compare the bounds for HamFlow on log-density evaluation (Corollary C.1) and ELBO estimation (Corollary C.2). The main difference between these bounds is in the local Lipschitz constants. The error in log-density evaluation is based on  $L_{q,\epsilon}$  while the error in ELBO estimation is based on  $\mathbb{E}[L_{\bar{p},\epsilon}]$ . These two quantities—  $L_{q,\epsilon}$  and  $\mathbb{E}[L_{\bar{p},\epsilon}]$ —relate to the smoothness of the learned log-density of HamFlow and log-density of the target. Figs. 8c to 8f display  $\log p$  and  $\log q$  side-by-side for both synthetic examples. Indeed, in both cases,  $\log q$  markedly deviates from  $\log p$  and presents drastic fluctuations, indicating that  $L_{q,\epsilon}$  can be substantially larger than  $\mathbb{E}[L_{\bar{p},\epsilon}]$ . Fig. 13 further provides a quantitative comparison on how  $L_{q,\epsilon}$  and  $\mathbb{E}[L_{\bar{p},\epsilon}]$  change as we increase the flow length. Note that since both  $L_{q,\epsilon}$  and  $L_{\bar{p},\epsilon}$  are intractable in general, we instead focus on the scaling of  $\| \nabla \log q \|$  (which is a lower bound of  $L_{q,\epsilon}$ ) and an upper bound of  $\mathbb{E}[L_{\bar{p},\epsilon}]$  (derived from case-by-case analysis for both the banana and cross distributions; see Appendix C.3 for detailed derivation). Results show that in both synthetic examples,  $L_{q,\epsilon}$  can increase exponentially over increasing flow length and is eventually larger than  $\mathbb{E}[L_{\bar{p},\epsilon}]$  by 30 orders of magnitude, while  $\mathbb{E}[L_{\bar{p},\epsilon}]$  does not grow and remains within a reasonable scale.
Similarly, the differences in log-density errors between MixFlow and HamFlow come from how smooth the learned log-density is in each method, according to the error bounds presented in Theorems 4.3 and 4.6. Fig. 2 visualizes the learned log-densities of MixFlow, which matches closely to the actual target distribution, and has a smoother output than HamFlow. This matches what was noted in Xu et al. [5, Figure 3], where MixFlow was shown to approximate the target density better than standard normalizing flows.
Finally, Fig. 14 presents the single step evaluation error  $\delta$  of HamFlow and the size of the shadowing window  $\epsilon$  as the flow length  $N$  increases. Results show that for both synthetic examples,  $\delta$  is approximately  $10^{-7}$ , hence we fix  $\delta$  to be this value when computing the shadowing window  $\epsilon$ . Compared to the significant orbit evaluation error when  $N$  is large, the size of  $\epsilon$  remains small for both the forward and backward orbits (in the scale of  $10^{-4}$  to  $10^{-3}$  when  $N = 400$ ). And more importantly, the size of  $\epsilon$  grows less drastically than the orbit computation error with the flow length—roughly quadratically in the banana example, and linearly in the cross example, which is similar to what is observed in the MixFlow experiments.
C.3 Upper bounds of  $L_{\bar{p},\epsilon}$  for synthetic examples
Recall that we denote  $\bar{p}(x) \propto p([x]_{1:2}) \cdot \exp\left(-\frac{1}{2}[x]_{3:4}^T[x]_{3:4}\right)$  the augmented target and  $p$  denotes the 2d-synthetic target we are interested.
$$
L _ {\bar {p}, \epsilon} (x) = \sup  _ {\| y - x \| \leq \epsilon} \| \nabla \log p ([ y ] _ {1: 2}) - [ y ] _ {3: 4} \| \leq L _ {p, \epsilon} ([ x ] _ {1: 2}) + \| [ x ] _ {3: 4} \| + \epsilon ,
$$
where  $L_{p,\epsilon}$  requires a case-by-case analysis. Therefore, in this section, we focus on bounding the local Lipschitz constant for  $\log p$ .
Proposition C.3. For the 2-dimensional banana distribution, i.e.,
$$
p (x) \propto \exp \left(- \frac {1}{2} \frac {x _ {1} ^ {2}}{\sigma^ {2}} - \frac {1}{2} \left(x _ {2} + b x _ {1} ^ {2} - \sigma^ {2} b\right)\right),
$$
we have that for all  $x\coloneqq \left[ \begin{array}{c}x_{1}\\ x_{2} \end{array} \right]\in \mathbb{R}^{2}$
$$
L _ {p, \epsilon} (x) \leq \| \nabla \log p (x) \| + \epsilon \cdot \left(2 b | x _ {1} | + \max  \left\{\left| 6 b ^ {2} x _ {1} ^ {2} + 2 b x _ {2} - 2 \sigma^ {2} b ^ {2} + \frac {1}{\sigma^ {2}} \right|, 1 \right\} + 2 b \max  \left\{6 b | x _ {1} | + 6 b (\epsilon + \epsilon^ {2}), 1 \epsilon \right\}\right)
$$
Proof of Proposition C.3. By the mean value theorem,
$$
\begin{array}{l} L _ {p, \epsilon} (x) \leq \| \nabla \log p (x) \| + \sup  _ {\| y - x \| \leq \epsilon} \| \nabla \log p (y) - \nabla \log p (x) \| \\ \leq \| \nabla \log p (x) \| + \sup  _ {\| y - x \| \leq \epsilon} \| \nabla^ {2} \log p (y) \| \cdot \epsilon . \\ \end{array}
$$
The proof will then complete by showing that
$$
\sup  _ {\| y - x \| \leq \epsilon} \| \nabla^ {2} \log p (y) \| \leq 2 b | x _ {1} | + \max  \left\{\left| 6 b ^ {2} x _ {1} ^ {2} + 2 b x _ {2} - 2 \sigma^ {2} b ^ {2} + \frac {1}{\sigma^ {2}} \right|, 1 \right\} + 2 b \max  \left\{6 b | x _ {1} | + 6 b (\epsilon + \epsilon^ {2}), \epsilon \right\}. \tag {13}
$$
To verify Eq. (13), note that
$$
\nabla^ {2} \log p (x) = - \left[ \begin{array}{c c} 6 b ^ {2} x _ {1} ^ {2} + 2 b x _ {2} - 2 \sigma^ {2} b ^ {2} + \frac {1}{\sigma^ {2}} & 2 b x _ {1} \\ 2 b x _ {1} & 1 \end{array} \right]
$$
The Gershgorin circle theorem then yields that
$$
\left\| \nabla^ {2} \log p (x) \right\| \leq 2 b | x _ {1} | + \max  \left\{\left| 6 b ^ {2} x _ {1} ^ {2} + 2 b x _ {2} - 2 \sigma^ {2} b ^ {2} + \frac {1}{\sigma^ {2}} \right|, 1 \right\}.
$$
Additionally, we have
$$
\sup_{\| y_{1} - x_{1}\| \leq \epsilon}\left|y_{1}\right|\leq \left|x_{1}\right| + \epsilon
$$
$$
\sup_{\| y - x\| \leq \epsilon}\left|6b^{2}y_{1}^{2} + 2by_{2} - 2\sigma^{2}b^{2} + \frac{1}{\sigma^{2}}\right|\leq \left|6b^{2}x_{1}^{2} + 2bx_{2} - 2\sigma^{2}b^{2} + \frac{1}{\sigma^{2}}\right| + 2b\max \left\{6b|x_{1}| + 6b(\epsilon +\epsilon^{2}),\epsilon \right\} .
$$
Proposition C.4. For the cross distribution, a 4-component Gaussian mixture of the form
$$
\begin{array}{l} p (x) = \frac {1}{4} \mathcal {N} \left(x | \left[ \begin{array}{c} 0 \\ 2 \end{array} \right], \left[ \begin{array}{c c} 0. 1 5 ^ {2} & 0 \\ 0 & 1 \end{array} \right]\right) + \frac {1}{4} \mathcal {N} \left(x | \left[ \begin{array}{c} - 2 \\ 0 \end{array} \right], \left[ \begin{array}{c c} 1 & 0 \\ 0 & 0. 1 5 ^ {2} \end{array} \right]\right) \\ + \frac {1}{4} \mathcal {N} \left(x | \left[ \begin{array}{c} 2 \\ 0 \end{array} \right], \left[ \begin{array}{c c} 1 & 0 \\ 0 & 0. 1 5 ^ {2} \end{array} \right]\right) + \frac {1}{4} \mathcal {N} \left(x | \left[ \begin{array}{c} 0 \\ - 2 \end{array} \right], \left[ \begin{array}{c c} 0. 1 5 ^ {2} & 0 \\ 0 & 1 \end{array} \right]\right), \\ \end{array}
$$
we have that for all  $x\in \mathbb{R}^2$
$$
L _ {p, \epsilon} (x) \leq \frac {1}{0 . 1 5 ^ {2}} \left(2 + \| x \| + \epsilon\right).
$$
Proof of Proposition C.4. For notational convenience, denote the density of 4 Gaussian component by  $p_1, p_2, p_3, p_4$ , hence
$$
\log p (x) = \log \left\{p _ {1} (x) + p _ {2} (x) + p _ {3} (x) + p _ {4} (x) \right\} - \log 4.
$$
Examining  $\nabla \log p(x)$  directly yields that for all  $x\in \mathbb{R}^2$
$$
\begin{array}{l} \nabla \log p (x) = \frac {1}{p _ {1} (x) + p _ {2} (x) + p _ {3} (x) + p _ {4} (x)} \sum_ {i = 1} ^ {4} p _ {i} (x) \Sigma_ {i} ^ {- 1} (\mu_ {i} - x) \\ \in \operatorname {c o n v} \left\{\Sigma_ {i} ^ {- 1} (\mu_ {i} - x): i = 1, 2, 3, 4 \right\}. \\ \end{array}
$$
Here  $\mu_{i}$  and  $\Sigma_{i}$  denotes the Gaussian mean and covariance of  $p_i$ . Then by Jensen's inequality,
$$
\begin{array}{l} \sup  _ {\| y - x \| \leq \epsilon} \| \nabla \log p (y) \| \leq \max  \left\{\sup  _ {\| y - x \| \leq \epsilon} \| \Sigma_ {i} ^ {- 1} (\mu_ {i} - y) \|: i = 1, 2, 3, 4 \right\} \\ \leq \max  _ {i \in [ 4 ]} \left\{\| \Sigma_ {i} ^ {- 1} \| \cdot \left(\| \mu_ {i} \| + \sup  _ {\| y - x \| \leq \epsilon} \| y \|\right) \right\} \\ \leq \frac {1}{0 . 1 5 ^ {2}} (2 + \| x \| + \epsilon). \\ \end{array}
$$
Trained HamFlow
Trained HamFlow
(b) Cross: HamFlow sample scatter
(c) True log-pdf
(a) Banana: HamFlow sample scatters
(d) HamFlow log-pdf
(e) True log-pdf
(f) HamFlow log-pdf
Figure 8: Visualization of sample scatters and density of HamFlow. Figure (a)-(b) each shows 1000 i.i.d. draws from HamFlow distribution (red) and target distribution (green). (c) and (e) show respectively the exact log-density of the Banana and the cross distribution, while (d) and (f) show respectively the sliced HamFlow log-density evaluation at  $\rho = (0,0)$  on those two targets. The HamFlow log-density evaluations are computed via 2048-big BigFloat representation.
(a) Banana
(b) Banana (log-scale)
(c) Cross
(d) Cross (log-scale)
Figure 9: HamFlow forward (fwd) and backward (bwd) orbit errors on synthetic examples. (a) and (c) show the median and upper/lower quartile forward error  $\| F^k x - \hat{F}^k x\|$  and backward error  $\| B^{k}y - \hat{B}^{k}y\|$  comparing  $k$  transformations of the forward exact/approximate maps  $F$ ,  $\hat{F}$  or backward exact/approximate maps  $B$ ,  $\hat{B}$ . And (b) and (d) respectively display (a) and (c) in a logarithmic scale, intended to provide a clearer illustration of the exponential growth of the error. The lines indicate the median, and error regions indicate  $25^{\mathrm{th}}$  to  $75^{\mathrm{th}}$  percentile from 100 independent runs, which take an i.i.d. draw of  $x$  from  $q_{0}$  and draw  $y$  from the actual target distribution  $p$ .
(a) Banana
Figure 10: HamFlow relative sample average computation error on three test functions:  $\sum_{i=1}^{d} [|x|]_i$ ,  $\sum_{i=1}^{d} [\sin(x) + 1]_i$  and  $\sum_{i=1}^{d} [\operatorname{sigmoid}(x)]_i$ . Each line denotes the numerical error computed from Monte Carlo average of the three test functions via 100 independent forward orbits starting at i.i.d. draws from  $q_0$ .
Figure 11: HamFlow exact and numerical ELBO estimation over increasing flow length. The lines indicate the averaged ELBO estimates over 100 independent forward orbits. The Monte Carlo error for the given estimates is sufficiently small so we omit the error bar.
(a) Banana
(a) Banana
Figure 13: Comparison of HamFlow's  $L_{q,\epsilon}$  and  $\mathbb{E}[L_{\bar{p},\epsilon}(\hat{X}_N)]$  on the banana and cross distributions. Figs. 13a and 13c plot  $\| \nabla \log q(x)\|$  against flow length  $N$ ; the lines indicate the median, and error regions indicate  $25^{\mathrm{th}}$  to  $75^{\mathrm{th}}$  percentile from i.i.d. 100 independent draws of  $x$  from  $\bar{p}$ . Figs. 13b and 13d plot estimated upper bounds of  $\mathbb{E}[L_{\bar{p},\epsilon}(\hat{X}_N)]$  (averaging over 100 independent draws from  $q$ ) against flow length  $N$ . The choice and detailed derivation of the upper bounds of  $L_{\bar{p},\epsilon}$  can be found in Appendix C.3
(c) Banana
(a) Cross
Figure 12: HamFlow log-density evaluation error on synthetic examples. Log-densities are assessed at positions of 100 independent samples from the target distribution. (b) and (d) respectively display (a) and (c) in a logarithmic scale for a better illustration of the growth of error over increasing flow length. The lines indicate the median, and error regions indicate  $25^{\text{th}}$  to  $75^{\text{th}}$  percentile from the 100 evaluations.
(c) Cross
D Additional experimental details
All experiments were conducted on a machine with an AMD Ryzen 9 3900X and 32G of RAM.
D.1 Model description
The two synthetic distributions tested in this experiment were
the banana distribution [51]:
$$
y = \left[ \begin{array}{c} y _ {1} \\ y _ {2} \end{array} \right] \sim \mathcal {N} \left(0, \left[ \begin{array}{c c} 1 0 0 & 0 \\ 0 & 1 \end{array} \right]\right), \quad x = \left[ \begin{array}{c} y _ {1} \\ y _ {2} + b y _ {1} ^ {2} - 1 0 0 b \end{array} \right], \quad b = 0. 1;
$$
(a) Banana
(d) Cross
(b) Banana
(e) Cross
Figure 14: HamFlow forward (fwd) and backward (bwd) single transformation error  $\delta$  on both synthetic examples (Figure (a) and (d)), shadowing window size  $\epsilon$  over increasing flow length (Figure (b) and (e)), and shadowing window computation time (Figure (c) and (f)). Figure. (a) and (d) show the distribution of forward error  $\left\{\| F^n x_0 - \hat{F} \circ F^{n-1} x_0\| : n = 1, \ldots, N\right\}$  (Fwd err.) and backward error  $\left\{\| F^n x_0 - \hat{B} \circ F^{n+1} x_0\| : n = 1, \ldots, N\right\}$  (Bwd err.). These errors are computed along precise forward trajectories that originate at  $x_0$ , which is sampled 50 times independently and identically from the reference distribution  $q_0$ .  $N$  here denotes the flow length for each example. Figure (b) and (e) show  $\epsilon$  over increasing flow length. Figure (c) and (f) show the wall time of computing shadowing window over increasing flow length. The lines in Figure (b)-(c) and (e)-(f) indicate the median, and error regions indicate  $25^{\text{th}}$  to  $75^{\text{th}}$  percentile from 50 runs.
(c) Banana
(f) Cross
- a cross-shaped distribution: a Gaussian mixture of the form
$$
\begin{array}{l} x \sim \frac {1}{4} \mathcal {N} \left(\left[ \begin{array}{c} 0 \\ 2 \end{array} \right], \left[ \begin{array}{c c} 0. 1 5 ^ {2} & 0 \\ 0 & 1 \end{array} \right]\right) + \frac {1}{4} \mathcal {N} \left(\left[ \begin{array}{c} - 2 \\ 0 \end{array} \right], \left[ \begin{array}{c c} 1 & 0 \\ 0 & 0. 1 5 ^ {2} \end{array} \right]\right) \\ + \frac {1}{4} \mathcal {N} \left(\left[ \begin{array}{c} 2 \\ 0 \end{array} \right], \left[ \begin{array}{c c} 1 & 0 \\ 0 & 0. 1 5 ^ {2} \end{array} \right]\right) + \frac {1}{4} \mathcal {N} \left(\left[ \begin{array}{c} 0 \\ - 2 \end{array} \right], \left[ \begin{array}{c c} 0. 1 5 ^ {2} & 0 \\ 0 & 1 \end{array} \right]\right); \\ \end{array}
$$
The two real-data experiments are described below.
Linear regression. We consider a Bayesian linear regression problem where the model takes the form
$$
\begin{array}{l} \text {L i n . R e g :} \beta \stackrel {{\text {i . i . d .}}} {{\sim}} \mathcal {N} (0, 1), \log \sigma^ {2} \stackrel {{\text {i . i . d .}}} {{\sim}} \mathcal {N} (0, 1), \quad y _ {j} \mid \beta , \sigma^ {2} \stackrel {{\text {i n d e p}}} {{\sim}} \mathcal {N} \left(x _ {j} ^ {T} \beta , \sigma^ {2}\right), \end{array}
$$
where  $y_{j}$  is the response and  $x_{j} \in \mathbb{R}^{p}$  is the feature vector for data point  $j$ . For this problem, we use the Oxford Parkinson's Disease Telemonitoring Dataset [52]. The original dataset is available at http://archive.ics.uci.edu/dataset/189/parkinsons+telemonitoring, composed of 16 biomedical voice measurements from 42 patients with early-stage Parkinson's disease. The goal is to use these voice measurements, as well as subject age and gender information, to predict the total UPDRS scores. We standardize all features and subsample the original dataset down to 500 data points. The posterior dimension of the linear regression inference problems is 21.
Logistic regression. We then consider a Bayesian hierarchical logistic regression:
$$
\textbf {L o g i s . R e g . :} \alpha \sim \operatorname {G a m} (1, 0. 0 1), \beta \mid \alpha \sim \mathcal {N} (0, \alpha^ {- 1} I), \quad y _ {j} \mid \beta \stackrel {\text {i n d e p}} {\sim} \operatorname {B e r n} \left(\frac {1}{1 + e ^ {- x _ {j} ^ {T} \beta}}\right),
$$
We use a bank marketing dataset [53] downsampled to 400 data points. The original dataset is available at https://archive.ics.uci.edu/ml/datasets/bank+marketing. The goal is to use client information to predict subscription to a term deposit. We include 8 features from dataset: client age, marital status, balance, housing loan status, duration of last contact, number of contacts during campaign, number of days since last contact, and number of contacts before the current campaign. For each of the binary variables (marital status and housing loan status), all unknown entries are removed. All included features are standardized. The resulting posterior dimension of the logistic regression problem is 9.
Figure 15: MixFlow inversion error when using 2048-bit BigFloat representation. Vertex axis shows the reconstruction error  $\| B^{N} \circ F^{N}(x_{0}) - x_{0}\|$  (Fwd) and  $\| F^{N} \circ B^{N}(x_{0}) - x_{0}\|$  (Bwd) for the two synthetic examples (Figs. 15a and 15b) and the two real data examples (Figs. 15c and 15d).  $F$  and  $B$  are implemented using 2048-bit BigFloat representation, and  $x_{0}$  is sampled from the reference distribution  $q_{0}$ . The lines indicate the median, and the upper and lower quantiles over 10 independent runs. It can be seen that in all four examples, the inversion error of MixFlow is ignorable when using 2048-bit computation.
(a) Banana
Figure 16: MixFlow forward (fwd) and backward (bwd) orbit errors on both synthetic examples and real data examples. (a)-(d) show the median and upper/ lower quartile forward error  $\| F^k x - \hat{F}^k x\|$  and backward error  $\| B^{k}x - \hat{B}^{k}x\|$  comparing  $k$  transformations of the forward exact/approximate maps  $F$ ,  $\hat{F}$  or backward exact/approximate maps  $B$ ,  $\hat{B}$ . Note that Figs. 1b, 1e, 1c and 1f respectively display the initial segments of (a)-(d) in a logarithmic scale, intended to provide a clearer illustration of the exponential growth of the error.
D.2 MixFlow parameter settings
For all four examples, we use the uncorrected Hamiltonian MixFlow map  $F$  composed of a discretized Hamiltonian dynamic simulation (via leapfrog integrator) followed by a sinusoidal momentum pseudorefreshment as used in Xu et al. [5] (see [5, Section 5, Appendices E.1 and E.2] for details). However, we use a Gaussian momentum distribution, which is standard in Hamiltonian Monte Carlo methods [54] as opposed to the Laplace momentum used previously in MixFlows [5]. In terms of the settings of leapfrog integrators for each target, we used 200 leapfrog steps of size 0.02 and 60 leapfrog steps of size 0.005 for the banana and cross target distributions respectively, and used 40 leapfrog steps of size 0.0006 and 50 leapfrog steps of size 0.002 for the linear regression and logistic regression examples, respectively. The reference distribution  $q_{0}$  for each target is chosen to be the mean-field Gaussian approximation as used in [5].
As for the NUTS used for assessing the log-density evaluation quality on two real data examples, we use the Julia package AdvancedHMC. j1 [55] with all default settings. NUTS is initialized with the learned mean of the mean-field Gaussian approximation  $q_{0}$ , and burns in the first 10,000 samples before collecting samples; the burn-in samples are used for adapting the hyperparameters of NUTS with target acceptance rate set to be 0.7.
D.3 Additional MixFlow results
(b) Cross
Figure 17: MixFlow log-density evaluation error. For synthetic examples (Figs. 17a and 17b), we assessed densities at 100 evenly distributed points within the target region (Figs. 2a and 2b). For real data examples (Figs. 17c and 17d), we evaluated densities at the locations of 100 NUTS samples. The lines indicate the median, and error regions indicate  $25^{\text{th}}$  to  $75^{\text{th}}$  percentile from 100 evaluations on different positions.