SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zuheng Xu Trevor Campbell
[zuheng.xu | trevor]@stat.ubc.ca
In this paper, we investigate the impact of numerical instability on the reliability of sampling, density evaluation, and evidence lower bound (ELBO) estimation in variational flows. We first empirically demonstrate that common flows can exhibit a catastrophic accumulation of error: the numerical flow map deviates significantly from the exact map—which affects sampling—and the numerical inverse flow map does not accurately recover the initial input—which affects density and ELBO computations. Surprisingly though, we find that results produced by flows are often accurate enough for applications despite the presence of serious numerical instability. In this work, we treat variational flows as dynamical systems, and leverage shadowing theory to elucidate this behavior via theoretical guarantees on the error of sampling, density evaluation, and ELBO estimation. Finally, we develop and empirically test a diagnostic procedure that can be used to validate results produced by numerically unstable flows in practice.
Variational families of probability distributions play a prominent role in generative modelling and probabilistic inference. A standard construction of a variational family involves passing a simple reference distribution—such as a standard normal—through a sequence of parametrized invertible transformations, i.e., a flow [1-7]. Modern flow-based variational families have the representational capacity to adapt to highly complex target distributions while still providing computationally tractable i.i.d. draws and density evaluations, which in turn enables convenient training by minimizing the Kullback-Leibler (KL) divergence [8] via stochastic optimization methods [9-11].
In practice, creating a flexible variational flow often requires many flow layers, as the expressive power of a variational flow generally increases with the number of transformations [5, 12, 13]. Composing many transformations in this manner, however, can create numerical instability [5, 14-16], i.e., the tendency for numerical errors from imprecise floating-point computations to be quickly magnified along the flow. This accumulation of error can result in low-quality sample generation [14], numerical non-invertibility [14, 15], and unstable training [16]. But in this work, we demonstrate that this is not always the case with a counterintuitive empirical example: a flow that exhibits severe numerical instability—with error that grows exponentially in the length of the flow—but which surprisingly still returns accurate density values, i.i.d. draws, and evidence lower bound (ELBO) estimates.
Motivated by this example, the goal of this work is twofold: (1) to provide a theoretical understanding of how numerical instability in flows relates to error in downstream results; and (2) to provide a diagnostic procedure such that users of normalizing flows can check whether their particular results are reliable in practice. We develop a theoretical framework that investigates the influence of numerical instability using shadowing theory from dynamical systems [17-20]. Intuitively, shadowing theory asserts that although a numerical flow trajectory may diverge quickly from its exact counterpart,
there often exists another exact trajectory nearby that remains close to the numerical trajectory (the shadowing property). We provide theoretical error bounds for three core tasks given a flow exhibiting the shadowing property—sample generation, density evaluation, and ELBO estimation—that show that the error grows much more slowly than the error of the trajectory itself. Our results pertain to both normalizing flows [2, 3] and mixed variational flows [5]. We develop a diagnostic procedure for practical verification of the shadowing property of a given flow. Finally, we validate our theory and diagnostic procedure on MixFlow [5] on both synthetic and real data examples, and Hamiltonian variational flow [6] on synthetic examples.
Related work. Chang et al. [21] connected the numerical invertibility of deep residual networks (ResNets) and ODE stability analysis by interpreting ResNets as discretized differential equations, but did not provide a quantitative analysis of how stability affects the error of sample generation or density evaluation. This approach also does not apply to general flow transformations such as coupling layers. Behrmann et al. [14] analyzed the numerical inversion error of generic normalizing flows via bi-Lipschitz continuity for each flow layer, resulting in error bounds that grow exponentially with flow length. These bounds do not reflect empirical results in which the error in downstream statistical procedures (sampling, density evaluation, and ELBO estimation) tends to grow much more slowly. Beyond the variational literature, Bahsoun et al. [22] investigated statistical properties of numerical trajectories by modeling numerical round-off errors as small random perturbations, thus viewing the numerical trajectory as a sample path of a Markov chain. However, their analysis was constrained by the requirement that the exact trajectory follows a time-homogeneous, strongly mixing measure-preserving dynamical system, a condition that does not generally hold for variational flows. Tupper [23] established a relationship between shadowing and the weak distance between inexact and exact orbits in dynamical systems. Unlike our work, their results are limited to quantifying the quality of pushforward samples and do not extend to the evaluation of densities and ELBO estimation. In contrast, our error analysis provides a more comprehensive understanding of the numerical error of all three downstream statistical procedures.
2 Background: variational flows
A variational family is a parametrized set of probability distributions  $\{q_{\lambda}:\lambda \in \Lambda \}$  on some space  $\mathcal{X}$  that each enable tractable i.i.d. sampling and density evaluation. The ability to obtain draws and compute density values is crucial for fitting the family to data via maximum likelihood in generative modelling, as well as maximizing the ELBO in variational inference,
$$
\operatorname {E L B O} \left(q _ {\lambda} | | p\right) = \int q _ {\lambda} (x) \log \frac {p (x)}{q _ {\lambda} (x)} \mathrm {d} x,
$$
where  $p(x)$  is an unnormalized density corresponding to a target probability distribution  $\pi$ . In this work we focus on  $\mathcal{X} \subseteq \mathbb{R}^d$  endowed with its Borel  $\sigma$ -algebra and Euclidean norm  $\|\cdot\|$ , and parameter set  $\Lambda \subseteq \mathbb{R}^p$ . We assume all distributions have densities with respect to a common base measure on  $\mathcal{X}$ , and will use the same symbol to denote a distribution and its density. Finally, we will suppress the parameter subscript  $\lambda$ , since we consider a fixed member of a variational family throughout.
Normalizing flows. One approach to building a variational distribution  $q$  is to push a reference distribution,  $q_{0}$ , through a measurable bijection on  $\mathcal{X}$ . To ensure the function is flexible yet still enables tractable computation, it is often constructed via the composition of simpler measurable, invertible "layers"  $F_{1},\ldots ,F_{N}$ , referred together as a (normalizing) flow [1, 2, 4]. To generate a draw  $Y\sim q$ , we draw  $X\sim q_{0}$  and evaluate  $Y = F_{N}\circ F_{N - 1}\circ \dots \circ F_{1}(X)$ . When each  $F_{n}$  is a diffeomorphism, the density of  $q$  is
$$
\forall x \in \mathcal {X} \quad q (x) = \frac {q _ {0} \left(F _ {1} ^ {- 1} \circ \cdots \circ F _ {N} ^ {- 1} (x)\right)}{\prod_ {n = 1} ^ {N} J _ {n} \left(F _ {n} ^ {- 1} \circ \cdots \circ F _ {N} ^ {- 1} (x)\right)} \quad J _ {n} (x) = | \det  \nabla_ {x} F _ {n} (x) |. \tag {1}
$$
An unbiased ELBO estimate can be obtained using a draw from  $q_{0}$  via
$$
X \sim q _ {0}, E (X) = \log p \left(F _ {N} \circ \dots \circ F _ {1} (X)\right) - \log q _ {0} (X) + \sum_ {n = 1} ^ {N} \log J _ {n} \left(F _ {n} \circ \dots \circ F _ {1} (X)\right). \tag {2}
$$
<sup>1</sup>A differentiable map with a differentiable inverse; we assume all flow maps in this work are diffeomorphisms.
(a) Banana: num. and exact orbits MixFlow Fwd Orbit  $(N = 50)$
(b) Banana: flow error
(e) Cross: flow error
(c) Lin. Reg.: flow error
(f) Log. Reg.: flow error
Figure 1: MixFlow forward (fwd) and backward (bwd) orbit errors on the Banana and Cross distributions, and on two real data examples. The first column visualizes the exact and numerical orbits with the same starting point on the synthetic targets. For a better visualization, we display every  $2^{\mathrm{nd}}$  and  $4^{\mathrm{th}}$  states of presented orbits (instead of the complete orbits) in Figs. 1a and 1d, respectively. Figs. 1b, 1c, 1e and 1f show the median and upper/lower quartile forward error  $\| F^{k}x - \hat{F}^{k}x\|$  and backward error  $\| B^{k}x - \hat{B}^{k}x\|$  comparing  $k$  transformations of the forward exact/approximate maps  $F$ ,  $\hat{F}$  or backward exact/approximate maps  $B$ ,  $\hat{B}$ . Statistics are plotted over 100 initialization draws from the reference distribution  $q_{0}$ . For the exact maps we use a 2048-bit BigFloat representation, and for the numerical approximations we use 64-bit Float representation. The "exactness" of BigFloat representation is justified in Fig. 15 in Appendix D.3.
Common choices of  $F_{n}$  are RealNVP [24], neural spline [25], Sylvester [26], Hamiltonian-based [6, 7], and planar and radial [2] flows; see Papamakarios et al. [3] for a comprehensive overview.
MixFlows. Instead of using only the final pushforward distribution as the variational distribution  $q$ , Xu et al. [5] proposed averaging over all the pushforwards along a flow trajectory with identical flow layers, i.e.,  $F_{n} = F$  for some fixed  $F$ . When  $F$  is ergodic and measure-preserving for some target distribution  $\pi$ , Xu et al. [5] show that averaged flows guarantee total variation convergence to  $\pi$  in the limit of increasing flow length. To generate a draw  $Y\sim q$ , we first draw  $X\sim q_0$  and a flow length  $K\sim \mathrm{Unif}\{0,1,\ldots ,N\}$ , and then evaluate  $Y = F^{K}(X) = F\circ F\circ \dots \circ F(X)$  via  $K$  iterations of the map  $F$ . The density formula is given by the mixture of intermediate flow densities,
$$
\forall x \in \mathcal {X}, \quad q (x) = \frac {1}{N + 1} \sum_ {n = 0} ^ {N} \frac {q _ {0} \left(F ^ {- n} x\right)}{\prod_ {j = 1} ^ {n} J \left(F ^ {- j} x\right)}, \quad J (x) = | \det  \nabla_ {x} F (x) |. \tag {3}
$$
$$
X \sim q _ {0}, \quad E (X) = \frac {1}{N + 1} \sum_ {n = 0} ^ {N} \log \frac {p \left(F ^ {n} X\right)}{q \left(F ^ {n} X\right)}. \tag {4}
$$
3 Instability in variational flows and layer-wise error analysis
In practice, normalizing flows often involve tens of flow layers, with each layer coupled with deep neural network blocks [2, 4, 24] or discretized ODE simulations [6, 7]; past work on MixFlows requires thousands of flow transformations [5]. These flows are typically chaotic and sensitive to small perturbations, such as floating-point representation errors. As a result, the final output of a numerically computed flow can significantly deviate from its exact counterpart, which may cause downstream issues during training or evaluation of the flow [12, 14, 27]. To understand the accumulation of error,
(a) Banana sample scatter
(b) Cross sample scatter
(c) Banana log-densities
Figure 2: Figs. 2a and 2b respectively show sample scatters produced by the naive application of MixFlow formulae targeting at the banana and cross distributions, without accounting for numerical error. Figs. 2c and 2d display comparisons of log-densities on both synthetic examples. The true log-target is on the left, the exact MixFlow evaluation (computed via 2048-bit BigFloat representation) is in the middle, and the numerical MixFlow evaluation is on the right.
(d) Cross log-densities
Behrmann et al. [14] assume that each flow layer  $F_{n}$  and its inverse are Lipschitz and have bounded error,  $\sup_x\| F_n(x) - \hat{F}_n(x)\| \leq \delta$ , and analyze the error of each layer individually. This layer-wise analysis tends to yield error bounds that grow exponentially in the flow length  $N$  (see Appendix A.1):
$$
\left\| F _ {N} \circ \dots \circ F _ {1} (x) - \hat {F} _ {N} \circ \dots \circ \hat {F} _ {1} (x) \right\| \leq \delta \sum_ {n = 0} ^ {N - 1} \prod_ {j = n + 2} ^ {N} \operatorname {L i p} \left(F _ {j}\right). \tag {5}
$$
Given a constant flow map  $F_{j} = F$  with Lipschitz constant  $\ell$ , the bound is  $O(\delta \ell^{N})$ , which suggests that the error accumulates exponentially in the length of the flow. In Fig. 1, we test this hypothesis empirically using MixFlows on two synthetic targets (the banana and cross distribution) and two real data examples (Bayesian linear regression and logistic problems) taken from past work on MixFlows [5]. See Appendix D for the details of this test. Figs. 1b, 1c, 1e and 1f confirms that the exponential growth in the error bound Eq. (5) is reasonable; the error does indeed grow exponentially quickly in practice. And Fig. 16 (a)-(b) in Appendix D.3 further demonstrate that after fewer than 100 transformations both flows have error on the same order of magnitude as the scale of the target distribution. Naively, this implies that sampling, density evaluation, and ELBO estimation may be corrupted badly by numerical error.
But counterintuitively, in Fig. 2 we find that simply ignoring the issue of numerical error and using the exact formulae yield reasonable-looking density evaluations and sample scatters. These results are of course only qualitative in nature; we provide corresponding quantitative results later in Section 6. But Fig. 2 appears to violate the principle of "garbage in, garbage out;" the buildup of significant numerical error in the sample trajectories themselves does not seem to have a strong effect on the quality of downstream sampling and density evaluation. The remainder of this paper is dedicated to resolving this counterintuitive behavior using shadowing theory [28].
4 Global error analysis of variational flows via shadowing
4.1 The shadowing property
We analyze variational flows as finite discrete-time dynamical systems (see, e.g., [28]). In this work, we consider a dynamical system to be a sequence of diffeomorphisms on  $(\mathcal{X},\| \cdot \|)$ . We define the forward dynamics  $(F_{n})_{n = 1}^{N}$  to be the flow layer sequence, and the backward dynamics  $(B_{n})_{n = 1}^{N}$  to be comprised of the inverted flow layers  $B_{n} = F_{N - (n - 1)}^{-1}$  in reverse order. An orbit of a dynamical system starting at  $x\in \mathcal{X}$  is the sequence of states produced by the sequence of maps when initialized
Figure 3: A visualization of a pseudo-orbit and shadowing orbit. Solid arrows and filled dots indicate exact orbits, while dashed arrows and open dots indicate pseudo-orbits (e.g., via numerical computations). Red indicates the exact orbit  $(x_0,\dots ,x_4)$  that one intends to compute. Blue indicates the numerically computed  $\delta$ -pseudo-orbit  $(\hat{x}_0,\dots ,\hat{x}_4)$ . Grey indicates the corresponding  $\epsilon$ -shadowing orbit  $(s_1,\dots ,s_4)$ . At the top right of the figure,  $\| \hat{x}_4 - F_4(\hat{x}_3)\| \leq \delta$  illustrates the  $\delta$  numerical error at each step of the pseudo-orbit. The grey dashed circles demonstrate the  $\epsilon$ -shadowing window.
at  $x$ . Therefore, the forward and backward orbits initialized at  $x \in \mathcal{X}$  are
$$
\text {F o r w a r d O r b i t :} \quad x = x _ {0} \xrightarrow {F _ {1}} x _ {1} \xrightarrow {F _ {2}} x _ {2} \rightarrow \dots \xrightarrow {F _ {N}} x _ {N}
$$
$$
x _ {- N} \stackrel {{B _ {N} = F _ {1} ^ {- 1}}} {{\leftarrow}} \dots \leftarrow x _ {- 2} \stackrel {{B _ {2} = F _ {N - 1} ^ {- 1}}} {{\leftarrow}} x _ {- 1} \stackrel {{B _ {1} = F _ {N} ^ {- 1}}} {{\leftarrow}} x _ {0} = x: \text {B a c k w a r d O r b i t}.
$$
Given numerical implementations  $\hat{F}_n\approx F_n$  and  $\hat{B}_n\approx B_n$  with tolerance  $\delta >0$ , i.e.,
$$
\forall x \in \mathcal {X}, \| F _ {n} (x) - \hat {F} _ {n} (x) \| \leq \delta , \| B _ {n} (x) - \hat {B} _ {n} (x) \| \leq \delta , \tag {6}
$$
we define the forward and backward pseudo-dynamics to be  $(\hat{F}_n)_{n=1}^N$  and  $(\hat{B}_n)_{n=1}^N$ , respectively, along with their forward and backward pseudo-orbits initialized at  $x \in \mathcal{X}$ :
$$
\text {F o r w a r d P s e u d o - O r b i t :} \quad x = \hat {x} _ {0} \stackrel {{\hat {F} _ {1} \approx F _ {1}}} {\rightarrow} \hat {x} _ {1} \stackrel {{\hat {F} _ {2} \approx F _ {2}}} {\rightarrow} \hat {x} _ {2} \rightarrow \dots \stackrel {{\hat {F} _ {N} \approx F _ {N}}} {\rightarrow} \hat {x} _ {N}
$$
$$
\hat {x} _ {- N} \stackrel {{\hat {B} _ {N} \approx F _ {1} ^ {- 1}}} {{\leftarrow}} \dots \leftarrow \hat {x} _ {- 2} \stackrel {{\hat {B} _ {2} \approx F _ {N - 1} ^ {- 1}}} {{\leftarrow}} \hat {x} _ {- 1} \stackrel {{\hat {B} _ {1} \approx F _ {N} ^ {- 1}}} {{\leftarrow}} \hat {x} _ {0} = x: \text {B a c k w a r d P s e u d o - O r b i t}.
$$
For notational brevity, we use subscripts on elements of  $\mathcal{X}$  throughout to denote forward/backward orbit states. For example, given a random element  $Z\in \mathcal{X}$  drawn from some distribution,  $Z_{k}$  is the  $k^{\mathrm{th}}$  state generated by the forward dynamics  $(F_n)^N_{n = 1}$  initialized at  $Z$ , and  $Z_{-k}$  is the  $k^{\mathrm{th}}$  state generated by the backward dynamics  $(B_{n})_{n = 1}^{N}$  initialized at  $Z$ . Hat accents denote the same for the pseudo-dynamics: for example,  $\hat{Z}_k$  is the  $k^{\mathrm{th}}$  state generated by  $(\hat{F}_n)^N_{n = 1}$  when initialized at  $Z$ .
The forward/backward orbits satisfy  $x_{k + 1} = F_k(x_k)$  and  $x_{-(k + 1)} = B_k(x_{-k})$  exactly at each step. On the other hand, the forward/backward pseudo-orbits incur a small amount of error,
$$
\left\| \hat {x} _ {k + 1} - F _ {k} \left(\hat {x} _ {k}\right) \right\| \leq \delta \quad \text {a n d} \quad \left\| \hat {x} _ {- (k + 1)} - B _ {k} \left(\hat {x} _ {- k}\right) \right\| \leq \delta ,
$$
which can be magnified quickly along the orbit. However, there often exists another exact orbit starting at some other point  $s \in \mathcal{X}$  that remains in a close neighbourhood of the numerical orbit. This property, illustrated in Fig. 3, is referred to as the shadowing property [19, 28-31].
Definition 4.1 (Shadowing property [28]). The forward dynamics  $(F_{n})_{n = 1}^{N}$  has the  $(\epsilon, \delta)$ -shadowing property if for all  $x \in \mathcal{X}$  and all  $(\hat{F}_n)_{n = 1}^N$  satisfying Eq. (6), there exists an  $s \in \mathcal{X}$  such that
$$
\forall k = 0, 1, \dots , N, \| s _ {k} - \hat {x} _ {k} \| <   \epsilon .
$$
An analogous definition holds for the backward dynamics  $(B_n)_{n=1}^N$  — where there is a shadowing orbit  $s_{-k}$  that is nearby the pseudo-orbit  $\hat{x}_{-k}$ —and for the joint forward and backward dynamics, where there is a shadowing orbit nearby both the backward and forward pseudo-orbits simultaneously. The key idea in this paper is that, intuitively, if the numerically computed pseudo-orbit is close to
some exact orbit (the shadowing orbit), statistical computations based on the pseudo-orbit—e.g., sampling, density evaluation, and ELBO estimation—should be close to those obtained via that exact orbit. We will defer the examination of when (and to what extent) shadowing holds to Section 5; in this section, we will use it as an assumption when analyzing statistical computations with numerically implemented variational flows.
4.2 Error analysis of normalizing flows via shadowing
Sampling. Our first goal is to relate the marginal distributions of  $X_N$  and  $\hat{X}_N$  for  $X \sim q_0$ , i.e., to quantify the error in sampling due to numerical approximation. Assume the normalizing flow has the  $(\epsilon, \delta)$ -shadowing property, and let  $\xi_0$  be the marginal distribution of the shadowing orbit start point. We suspect that  $\xi_0 \approx q_0$ , in some sense; for example, we know that the bounded Lipschitz distance  $\mathrm{D}_{\mathrm{BL}}(\xi_0, q_0)$  is at most  $\epsilon$  due to shadowing. And  $\xi_0$  is indeed an implicit function of  $q_0$ ; it is a fixed point of a twice differentiable function involving the whole orbit starting at  $q_0$  [17, Page 176]. But the distribution  $\xi_0$  is generally hard to describe more completely, and thus it is common to impose additional assumptions. For example, past work shows that the Lévy-Prokhorov metric  $\mathrm{D}_{\mathrm{LP}}(\hat{X}_N, X_N)$  is bounded by  $\epsilon$ , under the assumption that  $\xi_0 = q_0$  [23]. We provide a more general result (Proposition 4.2) without distributional assumptions on  $\xi_0$ . We control  $\mathrm{D}_{\mathrm{BL}}(\cdot, \cdot)$  rather than  $\mathrm{D}_{\mathrm{LP}}(\cdot, \cdot)$ , as its analysis is simpler and both metrize weak distance.
Proposition 4.2. Suppose the forward dynamics has the  $(\epsilon, \delta)$ -shadowing property, and  $X \sim q_0$ . Then
$$
\sup  _ {f: | f | \leq U, \operatorname {L i p} (f) \leq \ell} \left| \mathbb {E} f \left(X _ {N}\right) - \mathbb {E} f \left(\hat {X} _ {N}\right) \right| \leq \ell \epsilon + 2 U \mathrm {D} _ {\mathrm {T V}} \left(\xi_ {0}, q _ {0}\right).
$$
In particular, with  $\ell = U = 1$ , we obtain that  $\mathrm{D}_{\mathrm{BL}}(X_N,\hat{X}_N)\leq \epsilon +2\mathrm{D}_{\mathrm{TV}}(\xi_0,q_0)$ .
Recall the layerwise error bound from Eq. (5)—which suggests that the difference in orbit and pseudo-orbit grows exponentially in  $N$ —and compare to Proposition 4.2, which asserts that the error is controlled by the shadowing window size  $\epsilon$ . This window size may depend on  $N$ , but we find in Section 6 it is usually not much larger than  $\delta$  in practice, which itself is typically near the precision of the relevant numerical representation. We will show how to estimate  $\epsilon$  later in Section 5.
Density evaluation. The exact density  $q(x)$  follows Eq. (1), while the approximation  $\hat{q}(x)$  is the same except that we use the backward pseudo-dynamics. For  $x \in \mathcal{X}$ , a differentiable function  $g: \mathcal{X} \mapsto \mathbb{R}^+$ , define the local Lipschitz constant for the logarithm of  $g$  around  $x$  as:
$$
L _ {g, \epsilon} (x) = \sup  _ {\| y - x \| \leq \epsilon} \| \nabla \log g (y) \|.
$$
Theorem 4.3 shows that, given the shadowing property, the numerical error is controlled by the shadowing window size  $\epsilon$  and the sum of the Lipschitz constants along the pseudo-orbit. The constant  $L_{q,\epsilon}$  occurs because we are essentially evaluating  $q(s)$  rather than  $q(x)$ , where  $s \in \mathcal{X}$  is the backward shadowing orbit initialization. The remaining constants occur because of the approximation of the shadowing orbit with the nearby, numerically-computed pseudo-orbit in the density formula.
Theorem 4.3. Suppose the backward dynamics has the  $(\epsilon, \delta)$ -shadowing property. Then
$$
\forall x \in \mathcal {X}, \quad | \log \hat {q} (x) - \log q (x) | \leq \epsilon \cdot \left(L _ {q, \epsilon} (x) + L _ {q _ {0}, \epsilon} (\hat {x} _ {- N}) + \sum_ {n = 1} ^ {N} L _ {J _ {N - n + 1}, \epsilon} (\hat {x} _ {- n})\right).
$$
ELBO estimation. The exact ELBO estimation function is given in Eq. (2); the numerical ELBO estimation function  $\hat{E}(x)$  is the same except that we use the forward pseudo-dynamics. The quantity  $E(X), X \sim q_0$  is an unbiased estimate of the exact ELBO, while  $\hat{E}(X)$  is biased by numerical error; Theorem 4.4 quantifies this error. Note that for simplicity we assume that the initial state distributions  $q_0, \xi_0$  described earlier are identical. It is possible to obtain a bound including a  $\mathrm{D}_{\mathrm{TV}}(q_0, \xi_0)$  term, but this would require the assumption that  $\log p(x) / q(x)$  is uniformly bounded on  $\mathcal{X}$ .
Theorem 4.4. Suppose the forward dynamics has the  $(\epsilon, \delta)$ -shadowing property, and  $\xi_0 = q_0$ . Then