SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| \operatorname {E L B O} (q | | p) - \mathbb {E} [ \hat {E} (X) ] \right| \leq \epsilon \cdot \mathbb {E} \left[ L _ {p, \epsilon} (\hat {X} _ {N}) + L _ {q _ {0}, \epsilon} (X) + \sum_ {n = 1} ^ {N} L _ {J _ {n}, \epsilon} (\hat {X} _ {n}) \right] \quad f o r X \sim q _ {0}.
$$
4.3 Error analysis of MixFlows via shadowing
The error analysis for MixFlows for finite length  $N \in \mathbb{N}$  parallels that of normalizing flows, except that  $F_{n} = F$ ,  $B_{n} = B$ , and  $J_{n} = J$  for  $n = 1, \ldots, N$ . However, when  $F$  and  $B$  are ergodic and measure-preserving for the target  $\pi$ , we provide asymptotically simpler bounds in the large  $N$  limit that do not depend on the difficult-to-analyze details of the Lipschitz constants along a pseudo-orbit. These results show that the error of sampling tends to be constant in flow length, while the error of density evaluation and ELBO estimation grows at most linearly. We say the forward dynamics has the infinite  $(\epsilon, \delta)$ -shadowing property if  $(F_{n})_{n=1}^{\infty}$  has the  $(\epsilon, \delta)$ -shadowing property [32]. Analogous definitions hold for both the backward and joint forward/backward dynamics.
Sampling. Similar to Proposition 4.2 for normalizing flows, Proposition 4.5 bounds the error in exact draws  $Y$  and approximate draws  $\hat{Y}$  from the MixFlow. The result demonstrates that error does not directly depend on the flow length  $N$ , but rather on the shadowing window  $\epsilon$ . In addition, in the setting where the flow map  $F$  is  $\pi$ -ergodic and measure-preserving, we can (asymptotically) remove the total variation term between the initial shadowing distribution  $\xi_0$  and the reference distribution  $q_0$ . Note that in the asymptotic bound, the distributions of  $Y$  and  $\hat{Y}$  are functions of  $N$ .
Proposition 4.5. Suppose the forward dynamics has the  $(\epsilon, \delta)$ -shadowing property, and  $X \sim q_0$ . Let  $Y = X_K$ ,  $\hat{Y} = \hat{X}_K$  for  $K \sim \mathrm{Unif}\{0,1,\dots,N\}$ . Then
$$
\sup _ {f: | f | \leq U, \operatorname {L i p} (f) \leq \ell} \left| \mathbb {E} f (Y) - \mathbb {E} f (\hat {Y}) \right| \leq \ell \epsilon + 2 U \mathrm {D} _ {\mathrm {T V}} \left(\xi_ {0}, q _ {0}\right).
$$
In particular, if  $\ell = U = 1$ , we obtain that  $\mathrm{D_{BL}}(Y,\hat{Y})\leq \epsilon +2\mathrm{D_{TV}}(\xi_0,q_0)$ . If additionally the forward dynamics has the infinite  $(\epsilon ,\delta)$ -shadowing property,  $F$  is  $\pi$ -ergodic and measure-preserving, and  $\xi_0,q_0\ll \pi$ , then
$$
\limsup_{N\to \infty}\mathrm{D}_{\mathrm{BL}}\left(Y,\hat{Y}\right)\leq \epsilon .
$$
A direct corollary of the second result in Proposition 4.5 is that for  $W \sim \pi$ ,  $\lim_{N \to \infty} \mathrm{D}_{\mathrm{BL}}(W, \hat{Y}) \leq \epsilon$  which results from the fact that  $\mathrm{D}_{\mathrm{BL}}(W, Y) \to 0$  [5, Theorem 4.1]. In other words, the bounded Lipschitz distance of the approximated MixFlow and the target  $\pi$  is asymptotically controlled by the shadowing window  $\epsilon$ . This improves upon earlier theory governing approximate MixFlows, for which the best guarantee available had total variation error growing as  $O(N)$  [5, Theorem 4.3].
Density evaluation. The exact density follows Eq. (3); the numerical approximation is the same except that we use the backward pseudo-dynamics. We obtain a similar finite-  $N$  result as in Theorem 4.3. Further, we show that given infinite shadowing—where the shadowing window size  $\epsilon$  is independent of flow length  $N$  —the numerical approximation error asymptotically grows at most linearly in  $N$ , in proportion to  $\epsilon$ .
Theorem 4.6. Suppose the  $(\epsilon, \delta)$ -shadowing property holds for the backward dynamics. Then
$$
\forall x \in \mathcal {X}, \quad | \log \hat {q} (x) - \log q (x) | \leq \epsilon \cdot \left(L _ {q, \epsilon} (x) + \max  _ {0 \leq n \leq N} L _ {q _ {0}, \epsilon} (\hat {x} _ {- n}) + \sum_ {n = 1} ^ {N} L _ {J, \epsilon} (\hat {x} _ {- n})\right).
$$
If additionally the backward dynamics has the infinite  $(\epsilon, \delta)$ -shadowing property,  $F$  is  $\pi$ -ergodic and measure-preserving,  $L_{q,\epsilon}(x) = o(N)$  as  $N \to \infty$ , and  $\xi_0 \ll \pi$ , then for  $q_0$ -almost every  $x \in \mathcal{X}$ ,
$$
\limsup _ {N \to \infty} \frac {1}{N} | \log \hat {q} (x) - \log q (x) | \leq \epsilon \cdot \mathbb {E} \left[ L _ {q _ {0}, \epsilon} (X) + L _ {J, \epsilon} (X) \right], \quad X \sim \pi .
$$
ELBO estimation. The exact ELBO formula for the MixFlow is given in Eq. (4). Note that here we do not simply substitute the forward/backward pseudo-orbits as needed; the naive approximation of the terms  $q(F^n x)$  would involve  $n$  applications of  $\hat{F}$  followed by  $N$  applications of  $\hat{B}$ , which do not exactly invert one another. Instead, we analyze the method proposed in [5], which involves simulating a single forward pseudo-orbit  $\hat{x}_1, \dots, \hat{x}_N$  and backward pseudo-orbit  $\hat{x}_{-1}, \dots, \hat{x}_{-N}$  starting from  $x \in \mathcal{X}$ , and then caching these and using them as needed in the exact formula.
Theorem 4.7. Suppose the joint forward and backward dynamics has the  $(\epsilon, \delta)$ -shadowing property, and  $\xi_0 = q_0$ . Then for  $X \sim q_0$
$$
\left| \operatorname {E L B O} (q \| p) - \mathbb {E} \left[ \hat {E} (X) \right] \right| \leq \epsilon \cdot \mathbb {E} \left[ \frac {1}{N + 1} \sum_ {n = 0} ^ {N} L _ {p, \epsilon} (\hat {X} _ {n}) + \max  _ {- N \leq n \leq N} L _ {q _ {0}, \epsilon} (\hat {X} _ {n}) + \sum_ {n = - N} ^ {N} L _ {J, \epsilon} (\hat {X} _ {n}) \right].
$$
If additionally the joint forward and backward dynamics has the infinite  $(\epsilon, \delta)$ -shadowing property,  $F$  is  $\pi$ -ergodic and measure-preserving, and for some  $1 \leq m_1 < \infty$  and  $1/m_1 + 1/m_2 = 1$ $L_{p,\epsilon}, L_{q_0,\epsilon}, L_{J,\epsilon} \in L^{m_1}(\pi)$  and  $\frac{\mathrm{d}q_0}{\mathrm{d}\pi} \in L^{m_2}(\pi)$ , then
$$
\limsup_{N\to \infty}\frac{1}{N}\left|\operatorname {ELBO}\left(q\| p\right) - \mathbb{E}\left[\hat{E} (X)\right]\right|\leq 2\epsilon \cdot \mathbb{E}\left[L_{q_{0},\epsilon}(X) + L_{J,\epsilon}(X)\right],\quad X\sim \pi .
$$
5 Computation of the shadowing window size
We have so far assumed the  $(\epsilon, \delta)$ -shadowing property throughout. To make use of our results in practice, it is crucial to understand the shadowing window size  $\epsilon$ . Theorem 5.1 presents sufficient conditions for a finite dynamical system to have the shadowing property, and characterizes the size of the shadowing window  $\epsilon$ . Note that throughout this section we focus on the forward dynamics; the backward and joint dynamics can be treated identically. Let  $\|\cdot\|$  denote the spectral norm of a matrix.
Theorem 5.1 (Finite shadowing theorem). Suppose the dynamics  $(F_n)_{n=1}^N$  are  $C^2$  diffeomorphisms on  $\mathcal{X}$ , and the pseudo-dynamics  $(\hat{F}_n)_{n=1}^N$  satisfy Eq. (6). For a given  $x \in \mathcal{X}$ , define the operator  $A: \mathcal{X}^{N+1} \mapsto \mathcal{X}^N$  by
$$
(A u) _ {k} = u _ {k + 1} - \nabla F _ {k + 1} (\hat {x} _ {k}) u _ {k}, \quad f o r u = (u _ {0}, u _ {1}, \dots , u _ {N}) \in \mathcal {X} ^ {N}, \quad k = 0, 1, \dots , N - 1.
$$
$$
M := \max  \left\{\sup  _ {\| v \| \leq 2 \lambda \delta} \| \nabla^ {2} F _ {n + 1} (\hat {x} _ {n} + v) \|: n = 0, 1, \dots , N - 1 \right\} w h e r e \lambda = \lambda_ {\min } (A A ^ {T}) ^ {- 1 / 2}.
$$
If  $2M\lambda^2\delta \leq 1$ , then the pseudo-orbit starting at  $x$  is shadowed with  $\epsilon = 2\lambda \delta$ .
Proof. This result follows directly by following the proof of [18, Theorem 11.3] with nonconstant flow maps  $F_{n}$ , and then using the right-inverse norm from [33, Corollary 4.2].
In order to apply Theorem 5.1 we need to (1) estimate  $\delta$ , e.g., by examining numerical error of one step of the map in practice; (2) compute  $\lambda_{\min}(AA^T)$ ; and (3) estimate  $M$ , e.g., by bounding the third derivative of  $F_n$ . While estimating  $M$  and  $\delta$  are problem-specific, one can employ standard procedures for computing  $\lambda_{\min}(AA^T)$ . The matrix  $AA^T$  has a block-tridiagonal form,
$$
A A ^ {T} = \left[ \begin{array}{c c c c c} D _ {1} D _ {1} ^ {T} + I & - D _ {2} ^ {T} & & \\ - D _ {2} & D _ {2} D _ {2} ^ {T} + I & - D _ {3} ^ {T} & & \\ & \ddots & \ddots & \ddots & \\ & & - D _ {N - 1} & D _ {N - 1} D _ {N - 1} ^ {T} + I & - D _ {N} ^ {T} \\ & & & - D _ {N} & D _ {N} D _ {N} ^ {T} + I \end{array} \right],
$$
where  $D_{k} = \nabla F_{k}(\hat{x}_{k - 1})\in \mathbb{R}^{d\times d},\forall k\in [N]$ . Notice that  $AA^T$  is a symmetric positive definite sparse matrix with bandwidth  $d$  and so has  $O(Nd^{2})$  entries. The inherent structured sparsity can be leveraged to design efficient eigenvalue computation methods, e.g., the inverse power method [34], or tridiagonalization via Lanczos iterations [35] followed by divide-and-conquer algorithms [36]. However, in our experiments, directly calling the eigmin function provided in Julia suffices; as illustrated in Figs. 14 and 19, the shadowing window computation requires only a few seconds for low dimensional synthetic examples with hundreds flow layers, and several minutes for real data examples. Hence, we didn't pursue specialized methods, leaving that for future work. It is also noteworthy that practical computations of  $D_{k}$  can introduce floating-point errors, influencing the accuracy of  $\lambda$ . To address this, one might consider adjusting the shadowing window size. We explain how to manage this in Appendix B.1, and why such numerical discrepancies minimally impact results.
(a) Banana
(b) Cross
(c) Lin. Reg.
(d) Log. Reg.
Figure 4: MixFlow relative sample average computation error on three test functions:  $\sum_{i=1}^{d} [|x|]_i, \sum_{i=1}^{d} [\sin(x) + 1]_i$  and  $\sum_{i=1}^{d} [\operatorname{sigmoid}(x)]_i$ . The lines indicate the median, and error regions indicate  $25^{\text{th}}$  to  $75^{\text{th}}$  percentile from 20 runs.
(a) Banana
Figure 5: MixFlow relative log-density evaluation error. The lines indicate the median, and error regions indicate  $25^{\text{th}}$  to  $75^{\text{th}}$  percentile from 100 evaluations on different positions.
(a) Banana
Figure 6: MixFlow exact and numerical ELBO estimation over increasing flow length. The lines indicate the averaged ELBO estimates over 200 independent forward orbits. The Monte Carlo error for the given estimates is sufficiently small so we omit the error bar.
(a) Banana
Figure 7: MixFlow shadowing window size  $\epsilon$  over increasing flow length. The lines indicate the median, and error regions indicate  $25^{\text{th}}$  to  $75^{\text{th}}$  percentile from 10 runs.
(c) Linear regression
(d) Logistic regression
Finally, since  $\epsilon$  depends on  $\lambda_{\mathrm{min}}(AA^T)^{-1/2}$ , which itself depends on  $N$ , the shadowing window size may potentially increase with  $N$  in practice. Understanding this dependence accurately for a general dynamical system is challenging; there are examples where it remains constant, for instance (see Examples B.1 and B.2 in Appendix B.2), but  $\epsilon$  may grow with  $N$ . Our empirical results in next section show that on representative inferential examples with over 500 flow layers,  $\epsilon$  scales roughly linearly with  $N$  and is of a similar order of magnitude as the floating point representation error. For further discussion on this topic, we refer readers to Appendix B.2.
In this section, we verify our error bounds and diagnostic procedure of MixFlow on the banana, cross, and 2 real data targets—a Bayesian linear regression and logistic regression posterior; detailed
model and dataset descriptions can be found in Appendix D.1. We also provide a similar empirical investigation for Hamiltonian flow [6] on the same synthetic targets in Appendix C.2.
We begin by assessing the error of trajectory-averaged estimates based on approximate draws. Fig. 4 displays the relative numerical estimation error compared to the exact estimation based on the same initial draw. Although substantial numerical error was observed in orbit computation (see Fig. 16 in Appendix D.3), the relative sample estimate error was around  $1\%$  for all four examples, suggesting that the statistical properties of the forward orbits closely resemble those of the exact orbit.
We then focus on the density evaluation. For synthetic examples, we assessed densities at 100 evenly distributed points within the target region (Figs. 2a and 2b). For real data, we evaluated densities at the locations of 100 samples from MCMC method None-U-turn sampler (NUTS); detailed settings for NUTS is described in Appendix D.2. It is evident from the relative error shown in Fig. 5 that the numerical density closely matches the exact density evaluation, with the relative numerical error ranging between  $0.1\%$  and  $1\%$ , which is quite small. The absolute error can be found in Fig. 17 in Appendix D.3, showing that the density evaluation error does not grow as substantially as the orbit computation error (Fig. 16 in Appendix D.3), which aligns with the bound in Theorem 4.6.
Fig. 6 further demonstrates the numerical error for the ELBO estimations (Eq. (4)). In each example, both the averaged exact ELBO estimates and numerical ELBO estimates are plotted against an increasing flow length. Each ELBO curve is averaged over 200 independent forward orbits. Across all four examples, the numerical ELBO curve remains closely aligned with the exact ELBO curve, indicating that the numerical error is small in comparison to the scale of the ELBO value. Moreover, the error does not grow with an increasing flow length, and even presents a decreasing trend as  $N$  increases in the two synthetic examples and the Logistic regression example. This aligns well with the error bounds provided in Theorem 4.7.
Finally, Fig. 7 presents the size of the shadowing window  $\epsilon$  as the flow length  $N$  increases. As noted earlier,  $\epsilon$  depends on the flow map approximation error  $\delta$  and potentially  $N$ . We evaluated the size of  $\delta$  by calculating the approximation error of a single  $F$  or  $B$  for 1000 i.i.d. draws from the reference distribution. Boxplots of  $\delta$  for all examples can be found in Fig. 18 of Appendix D.3. These results show that in the four examples,  $\delta$  is close to the floating point representation error (approximately  $10^{-14}$ ). Thus, we fixed  $\delta$  at  $10^{-14}$  when computing  $\epsilon$ . As shown in Fig. 7,  $\epsilon$  for both the forward and backward orbits grows roughly linearly with the flow length. This growth is significantly less drastic than the exponential growth of the orbit computation error. And crucially, the shadowing window size is very small—smaller than  $10^{-10}$  for MixFlow of length over 1000, which justifies the validity of the downstream statistical procedures. In contrast, the orbit computation errors in the two synthetic examples rapidly reach a magnitude similar to the scale of the target distributions, as shown in Fig. 1.
This work delves into the numerical stability of variational flows, drawing insights from shadowing theory and introducing a diagnostic tool to assess the shadowing property. Experiments corroborate our theory and demonstrate the effectiveness of the diagnostic procedure. However, the scope of our current analysis is limited to downstream tasks post-training of discrete-time flows. Understanding the impact of numerical instability during the training phase or probing into recent architectures like continuous normalizing flows or neural ODEs [21, 37-40], remains an avenue for further exploration. Additionally, while our theory is centered around error bounds that are proportional to the shadowing window size  $\epsilon$ , we have recognized that  $\epsilon$  can grow with  $N$ . Further study on its theoretical growth rate is needed. Finally, in our experiments, we employed a basic method to calculate the minimum eigenvalue of  $AA^T$ . Given the sparsity of this matrix, a deeper exploration into a more efficient computational procedure is merited.
[1] Esteban Tabak and Cristina Turner. A family of non-parametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66(2):145-164, 2013.
[2] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, 2015.
[49] Tomas Geffner and Justin Domke. Langevin diffusion variational inference. In International Conference on Artificial Intelligence and Statistics, 2023.
[50] Achille Thin, Nikita Kotelevskii, Alain Durmus, Maxim Panov, Eric Moulines, and Arnaud Doucet. Monte Carlo variational auto-encoders. In International Conference on Machine Learning, 2021.
[51] Heikki Haario, Eero Saksman, and Johanna Tamminen. An adaptive Metropolis algorithm. Bernoulli, pages 223-242, 2001.
[52] Athanasios Tsanas, Max A. Little, Patrick E. McSharry, and Lorraine O. Ramig. Accurate telemonitoring of parkinson's disease progression by noninvasive speech tests. IEEE Transactions on Biomedical Engineering, 57:884-893, 2010. URL https://api.sementicscholar.org/CorpusID:7382779.
[53] Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62:22-31, 2014.
[54] Radford Neal. MCMC using Hamiltonian dynamics. In Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng, editors, Handbook of Markov chain Monte Carlo, chapter 5. CRC Press, 2011.
[55] Kai Xu, Hong Ge, Will Tebbutt, Mohamed Tarek, Martin Trapp, and Zoubin Ghahramani. AdvancedHMC.jl: A robust, modular and efficient implementation of advanced HMC algorithms. In Symposium on Advances in Approximate Bayesian Inference, 2020.
A.1 Layer-wise error analysis
The layer-wise error analysis follows by recursing the one-step bound on the difference between the exact and numerical flow:
$$
\begin{array}{l} \left\| F _ {N} \circ \dots \circ F _ {1} x - \hat {F} _ {N} \circ \dots \circ \hat {F} _ {1} x \right\| \\ \leq \| F _ {N} \circ \dots \circ F _ {1} x - F _ {N} \circ \hat {F} _ {N - 1} \circ \dots \hat {F} _ {1} x \| + \| F _ {N} \circ \hat {F} _ {N - 1} \circ \dots \hat {F} _ {1} x - \hat {F} _ {N} \circ \dots \circ \hat {F} _ {1} x \| \\ \leq \operatorname {L i p} \left(F _ {N}\right) \| F _ {N - 1} \circ \dots \circ F _ {1} x - \hat {F} _ {N - 1} \circ \dots \circ \hat {F} _ {1} x \| + \delta \\ \leq \dots \leq \delta \sum_ {n = 0} ^ {N - 1} \prod_ {j = n + 2} ^ {N} \operatorname {L i p} \left(F _ {j}\right). \\ \end{array}
$$
A.2 Error bounds for normalizing flows
Proof of Proposition 4.2. Let  $S$  be the random initial state of the orbit that shadows the pseudo-orbit of  $X \sim q_0$ . By triangle inequality,
$$
\left| \mathbb {E} f (X _ {N}) - \mathbb {E} f (\hat {X} _ {N}) \right| \leq | \mathbb {E} f (X _ {N}) - \mathbb {E} f (S _ {N}) | + \left| \mathbb {E} f (S _ {N}) - \mathbb {E} f (\hat {X} _ {N}) \right|.
$$
By  $\epsilon$ -shadowing and  $\ell$ -Lipshitz continuity of  $f$
$$
\left| \mathbb {E} f (S _ {N}) - \mathbb {E} f (\hat {X} _ {N}) \right| \leq \mathbb {E} \left| f (S _ {N}) - f (\hat {X} _ {N}) \right| \leq \ell \mathbb {E} \| S _ {N} - \hat {X} _ {N} \| \leq \ell \epsilon .
$$
$$
\begin{array}{l} \sup  _ {f: | f | \leq U, \operatorname {L i p} (f) \leq \ell} | \mathbb {E} f (X _ {N}) - \mathbb {E} f (S _ {N}) | \\ \leq \sup  _ {f: | f | \leq U} | \mathbb {E} f (X _ {N}) - \mathbb {E} f (S _ {N}) | = 2 U \mathrm {D} _ {\mathrm {T V}} \left(X _ {N}, S _ {N}\right) = 2 U \mathrm {D} _ {\mathrm {T V}} \left(X, S\right). \\ \end{array}
$$
The last equality is due to the fact that  $X_{N}$  and  $S_{N}$  are the map of  $X$  and  $S$ , respectively, under the bijection  $F_{N} \circ \dots \circ F_{1}$ , and the fact that the total variation is invariant under bijections [41, Theorem 1].
Proof of Theorem 4.3. Let  $s \in \mathcal{X}$  be the initial state of the backward shadowing orbit. By triangle inequality,
$$
| \log \hat {q} (x) - \log q (x) | \leq | \log q (x) - \log q (s) | + | \log \hat {q} (x) - \log q (s) |.
$$
For the first term on the right-hand side, note that for some  $y$  on the segment from  $x$  to  $s$ ,
$$
\left| \log q (x) - \log q (s) \right| = \left| \nabla \log q (y) ^ {T} (x - s) \right| \leq \epsilon \sup  _ {\| y - x \| \leq \epsilon} \| \nabla \log q (y) \|. \tag {7}
$$
For the second term,
$$
\log \hat {q} (x) - \log q (s) = | \log q _ {0} (\hat {x} _ {- N}) - \log q _ {0} (s _ {- N}) | + \sum_ {n = 1} ^ {N} \left| \log J _ {N - (n - 1)} (\hat {x} _ {- n}) - \log J _ {N - (n - 1)} (s _ {- n}) \right|.
$$
We apply the same technique to bound each term as in Eq. (7).
Proof of Theorem 4.4. By definition,  $\operatorname{ELBO}(q||p) = \mathbb{E}[E(X)]$ , and by hypothesis the distribution  $q_{0}$  of the initial flow state is equal to the distribution  $\xi_{0}$  of the shadowing orbit initial state, so  $\mathbb{E}[E(X)] = \mathbb{E}[E(S)]$ .
Hence  $\left|\operatorname{ELBO}(q||p) - \mathbb{E}\left[\hat{E}(X)\right]\right| \leq \mathbb{E}\left[\left|E(S) - \hat{E}(X)\right|\right]$ . Finally,
$$
\begin{array}{l} \left| E (s) - \hat {E} (x) \right| \\ = \left| \left(\log p (s _ {N}) - \log q _ {0} (s) + \sum_ {n = 1} ^ {N} \log J _ {n} (s _ {n})\right) - \left(\log p (\hat {x} _ {N}) - \log q _ {0} (x) + \sum_ {n = 1} ^ {N} \log J _ {n} (\hat {x} _ {n})\right) \right| \\ \leq \left| \log p \left(s _ {N}\right) - \log p \left(\hat {x} _ {N}\right) \right| + \left| \log q _ {0} (s) - \log q _ {0} (x) \right| + \sum_ {n = 1} ^ {N} \left| \log J _ {n} \left(s _ {n}\right) - \log J _ {n} \left(\hat {x} _ {n}\right) \right|. \\ \end{array}
$$
The proof is completed by bounding each difference with the local Lipschitz constant around the pseudo-orbit times the shadowing window size  $\epsilon$ , and applying the expectation.
A.3 Error bounds for MixFlows
Proof of Proposition 4.5. Let  $S$  be the initial point of the random shadowing orbit. By the definition of MixFlows,
$$
\begin{array}{l} \left| \mathbb {E} f (Y) - \mathbb {E} f (\hat {Y}) \right| = \left| \mathbb {E} \left[ \frac {1}{N + 1} \sum_ {n = 0} ^ {N} f \left(X _ {n}\right) - f \left(S _ {n}\right) + f \left(S _ {n}\right) - f \left(\hat {X} _ {n}\right) \right] \right| \\ \leq \left| \mathbb {E} \left[ \frac {1}{N + 1} \sum_ {n = 0} ^ {N} f (X _ {n}) - f (S _ {n}) \right] \right| + \mathbb {E} \left| \frac {1}{N + 1} \sum_ {n = 0} ^ {N} f (S _ {n}) - f (\hat {X} _ {n}) \right| \\ \leq \left| \mathbb {E} \left[ \frac {1}{N + 1} \sum_ {n = 0} ^ {N} f (X _ {n}) - f (S _ {n}) \right] \right| + \frac {1}{N + 1} \sum_ {n = 0} ^ {N} \mathbb {E} \left| f (S _ {n}) - f (\hat {X} _ {n}) \right|. \\ \end{array}
$$
To bound the second term, we apply shadowing and the Lipschitz continuity of  $f$  to show that each  $\left|f(S_n) - f(\hat{X}_n)\right| \leq \ell \epsilon$ , and hence their average has the same bound. For the first term, in the case where  $N$  is fixed and finite,
$$
\begin{array}{l} \sup  _ {f: | f | \leq U, \operatorname {L i p} (f) \leq \ell} \left| \mathbb {E} \left[ \frac {1}{N + 1} \sum_ {n = 0} ^ {N} f (X _ {n}) - f (S _ {n}) \right] \right| \leq \frac {1}{N + 1} \sum_ {n = 0} ^ {N} \sup  _ {f: | f | \leq U, \operatorname {L i p} (f) \leq \ell} | \mathbb {E} [ f (X _ {n}) - f (S _ {n}) ] | \\ \leq \frac {1}{N + 1} \sum_ {n = 0} ^ {N} 2 U \frac {1}{2} \sup  _ {f: | f | \leq 1} | \mathbb {E} [ f (X _ {n}) - f (S _ {n}) ] | \\ = \frac {1}{N + 1} \sum_ {n = 0} ^ {N} 2 U \mathrm {D} _ {\mathrm {T V}} \left(X _ {n}, S _ {n}\right) \\ = \frac {1}{N + 1} \sum_ {n = 0} ^ {N} 2 U \mathrm {D} _ {\mathrm {T V}} (X, S) \\ = 2 U D _ {\mathrm {T V}} \left(\xi_ {0}, q _ {0}\right). \\ \end{array}
$$
We now consider the large- $N$  limiting case when  $F$  is  $\pi$ -ergodic and measure-preserving,  $q_0 \ll \pi$  and  $\xi_0 \ll \pi$ , and  $U = \ell = 1$ . Let  $Z = S_K$ ,  $K \sim \mathrm{Unif}\{0, 1, \dots, N\}$ , and let  $W \sim \pi$ . In this case, by the triangle inequality,