Title: Embracing the chaos: analysis and diagnosis of numerical instability in variational flows
Venue: NeurIPS 2023
Accepted: yes
Keywords: variational flow

numerical instability

shadowing property
Abstract: In this paper, we investigate the impact of numerical instability on the reliability of sampling, density evaluation, and evidence lower bound (ELBO) estimation in variational flows. We first empirically demonstrate that common flows can exhibit a catastrophic accumulation of error: the numerical flow map deviates significantly from the exact map---which affects sampling---and the numerical inverse flow map does not accurately recover the initial input---which affects density and ELBO computations. Surprisingly though, we find that results produced by flows are often accurate enough for applications despite the presence of serious numerical instability. In this work, we treat variational flows as chaotic dynamical systems, and leverage shadowing theory to elucidate this behavior via theoretical guarantees on the error of sampling, density evaluation, and ELBO estimation. Finally, we develop and empirically test a diagnostic procedure that can be used to validate results produced by numerically unstable flows in practice.

Embracing the chaos: analysis and diagnosis of numerical instability in variational flows

Abstract

In this paper, we investigate the impact of numerical instability on the reliability of sampling, density evaluation, and evidence lower bound (ELBO) estimation in variational flows. We first empirically demonstrate that common flows can exhibit a catastrophic accumulation of error: the numerical flow map deviates significantly from the exact map-which affects sampling-and the numerical inverse flow map does not accurately recover the initial input-which affects density and ELBO computations. Surprisingly though, we find that results produced by flows are often accurate enough for applications despite the presence of serious numerical instability. In this work, we treat variational flows as dynamical systems, and leverage shadowing theory to elucidate this behavior via theoretical guarantees on the error of sampling, density evaluation, and ELBO estimation. Finally, we develop and empirically test a diagnostic procedure that can be used to validate results produced by numerically unstable flows in practice.

1 Introduction

Variational families of probability distributions play a prominent role in generative modelling and probabilistic inference. A standard construction of a variational family involves passing a simple reference distribution-such as a standard normal-through a sequence of parametrized invertible transformations, i.e., a flow [1-7]. Modern flow-based variational families have the representational capacity to adapt to highly complex target distributions while still providing computationally tractable i.i.d. draws and density evaluations, which in turn enables convenient training by minimizing the Kullback-Leibler (KL) divergence [8] via stochastic optimization methods [9-11].

In practice, creating a flexible variational flow often requires many flow layers, as the expressive power of a variational flow generally increases with the number of transformations [5, 12, 13]. Composing many transformations in this manner, however, can create numerical instability [5, 14-16], i.e., the tendency for numerical errors from imprecise floating-point computations to be quickly magnified along the flow. This accumulation of error can result in low-quality sample generation [14], numerical non-invertibility [14, 15], and unstable training [16]. But in this work, we demonstrate that this is not always the case with a counterintuitive empirical example: a flow that exhibits severe numerical instability-with error that grows exponentially in the length of the flow-but which surprisingly still returns accurate density values, i.i.d. draws, and evidence lower bound (ELBO) estimates.

Motivated by this example, the goal of this work is twofold: (1) to provide a theoretical understanding of how numerical instability in flows relates to error in downstream results; and (2) to provide a diagnostic procedure such that users of normalizing flows can check whether their particular results are reliable in practice. We develop a theoretical framework that investigates the influence of numerical instability using shadowing theory from dynamical systems [17-20]. Intuitively, shadowing theory asserts that although a numerical flow trajectory may diverge quickly from its exact counterpart,

there often exists another exact trajectory nearby that remains close to the numerical trajectory (the shadowing property ). We provide theoretical error bounds for three core tasks given a flow exhibiting the shadowing property-sample generation, density evaluation, and ELBO estimation-that show that the error grows much more slowly than the error of the trajectory itself. Our results pertain to both normalizing flows [2, 3] and mixed variational flows [5]. We develop a diagnostic procedure for practical verification of the shadowing property of a given flow. Finally, we validate our theory and diagnostic procedure on MixFlow [5] on both synthetic and real data examples, and Hamiltonian variational flow [6] on synthetic examples.

Related work. Chang et al. [21] connected the numerical invertibility of deep residual networks (ResNets) and ODE stability analysis by interpreting ResNets as discretized differential equations, but did not provide a quantitative analysis of how stability affects the error of sample generation or density evaluation. This approach also does not apply to general flow transformations such as coupling layers. Behrmann et al. [14] analyzed the numerical inversion error of generic normalizing flows via bi-Lipschitz continuity for each flow layer, resulting in error bounds that grow exponentially with flow length. These bounds do not reflect empirical results in which the error in downstream statistical procedures (sampling, density evaluation, and ELBO estimation) tends to grow much more slowly. Beyond the variational literature, Bahsoun et al. [22] investigated statistical properties of numerical trajectories by modeling numerical round-off errors as small random perturbations, thus viewing the numerical trajectory as a sample path of a Markov chain. However, their analysis was constrained by the requirement that the exact trajectory follows a time-homogeneous, strongly mixing measure-preserving dynamical system, a condition that does not generally hold for variational flows. Tupper [23] established a relationship between shadowing and the weak distance between inexact and exact orbits in dynamical systems. Unlike our work, their results are limited to quantifying the quality of pushforward samples and do not extend to the evaluation of densities and ELBO estimation. In contrast, our error analysis provides a more comprehensive understanding of the numerical error of all three downstream statistical procedures.

2 Background: variational flows

A variational family is a parametrized set of probability distributions { q λ : λ ∈ Λ } on some space X that each enable tractable i.i.d. sampling and density evaluation. The ability to obtain draws and compute density values is crucial for fitting the family to data via maximum likelihood in generative modelling, as well as maximizing the ELBO in variational inference,

where p ( x ) is an unnormalized density corresponding to a target probability distribution π . In this work we focus on X ⊆ R d endowed with its Borel σ -algebra and Euclidean norm ∥·∥ , and parameter set Λ ⊆ R p . We assume all distributions have densities with respect to a common base measure on X , and will use the same symbol to denote a distribution and its density. Finally, we will suppress the parameter subscript λ , since we consider a fixed member of a variational family throughout.

Normalizing flows. One approach to building a variational distribution q is to push a reference distribution, q 0 , through a measurable bijection on X . To ensure the function is flexible yet still enables tractable computation, it is often constructed via the composition of simpler measurable, invertible 'layers' F 1 , . . . , F N , referred together as a (normalizing) flow [1, 2, 4]. To generate a draw Y ∼ q , we draw X ∼ q 0 and evaluate Y = F N ◦ F N -1 ◦ · · · ◦ F 1 ( X ) . When each F n is a diffeomorphism, 1 the density of q is

An unbiased ELBO estimate can be obtained using a draw from q 0 via

(a) Banana: num. and exact orbits

(b) Banana: flow error

(c) Lin. Reg.: flow error

(d) Cross: num. and exact orbits

(e) Cross: flow error

(f) Log. Reg.: flow error

MixFlownumericalerror

MixFlownumericalerror

30

10°

exact

10°

20

numerical

rror

rror

10

0

E

10~10

10~10

Fwd

Fwd

-10

0

10

20

30

0

20

40

608

80100

-20

-10

0

10

20

#transformations

#transformations

MixFlownumericalerror

MixFlownumericalerror

4

exact

10°

Fwd

10°

2

numerical

rror

rror

0

E

10~10

-2

Fwd

-4

0

20

40

6080100

0

2040

6080100

-4

-2

0

2

4

#transformations

#transformations

Common choices of F n are RealNVP [24], neural spline [25], Sylvester [26], Hamiltonian-based [6, 7], and planar and radial [2] flows; see Papamakarios et al. [3] for a comprehensive overview.

MixFlows. Instead of using only the final pushforward distribution as the variational distribution q , Xu et al. [5] proposed averaging over all the pushforwards along a flow trajectory with identical flow layers, i.e., F n = F for some fixed F . When F is ergodic and measure-preserving for some target distribution π , Xu et al. [5] show that averaged flows guarantee total variation convergence to π in the limit of increasing flow length. To generate a draw Y ∼ q , we first draw X ∼ q 0 and a flow length K ∼ Unif { 0 , 1 , . . . , N } , and then evaluate Y = F K ( X ) = F ◦ F ◦ · · · ◦ F ( X ) via K iterations of the map F . The density formula is given by the mixture of intermediate flow densities,

An unbiased ELBO estimate can be obtained using a draw from q 0 via

3 Instability in variational flows and layer-wise error analysis

In practice, normalizing flows often involve tens of flow layers, with each layer coupled with deep neural network blocks [2, 4, 24] or discretized ODE simulations [6, 7]; past work on MixFlows requires thousands of flow transformations [5]. These flows are typically chaotic and sensitive to small perturbations, such as floating-point representation errors. As a result, the final output of a numerically computed flow can significantly deviate from its exact counterpart, which may cause downstream issues during training or evaluation of the flow [12, 14, 27]. To understand the accumulation of error,

(a) Banana sample scatter

(b) Cross sample scatter

(c) Banana log-densities

(d) Cross log-densities

MixFlowsamplescatters

30

20

10

0

-10

-20

-10

0

10

20

MixFlowsamplescatters

MixFlowsamplescatters

30

4

20

2

10

0

0

-2

-10

-4

-20

-10

0

10

20

-4

4-2

2

4

Behrmann et al. [14] assume that each flow layer F n and its inverse are Lipschitz and have bounded error, sup x ∥ F n ( x ) -ˆ F n ( x ) ∥ ≤ δ , and analyze the error of each layer individually. This layer-wise analysis tends to yield error bounds that grow exponentially in the flow length N (see Appendix A.1):

Given a constant flow map F j = F with Lipschitz constant ℓ , the bound is O ( δℓ N ) , which suggests that the error accumulates exponentially in the length of the flow. In Fig. 1, we test this hypothesis empirically using MixFlows on two synthetic targets (the banana and cross distribution) and two real data examples (Bayesian linear regression and logistic problems) taken from past work on MixFlows [5]. See Appendix D for the details of this test. Figs. 1b, 1c, 1e and 1f confirms that the exponential growth in the error bound Eq. (5) is reasonable; the error does indeed grow exponentially quickly in practice. And Fig. 16 (a)-(b) in Appendix D.3 further demonstrate that after fewer than 100 transformations both flows have error on the same order of magnitude as the scale of the target distribution. Naïvely, this implies that sampling, density evaluation, and ELBO estimation may be corrupted badly by numerical error.

But counterintuitively, in Fig. 2 we find that simply ignoring the issue of numerical error and using the exact formulae yield reasonable-looking density evaluations and sample scatters. These results are of course only qualitative in nature; we provide corresponding quantitative results later in Section 6. But Fig. 2 appears to violate the principle of 'garbage in, garbage out;' the buildup of significant numerical error in the sample trajectories themselves does not seem to have a strong effect on the quality of downstream sampling and density evaluation. The remainder of this paper is dedicated to resolving this counterintuitive behavior using shadowing theory [28].

4 Global error analysis of variational flows via shadowing

4.1 The shadowing property

We analyze variational flows as finite discrete-time dynamical systems (see, e.g., [28]). In this work, we consider a dynamical system to be a sequence of diffeomorphisms on ( X , ∥ · ∥ ) . We define the forward dynamics ( F n ) N n =1 to be the flow layer sequence, and the backward dynamics ( B n ) N n =1 to be comprised of the inverted flow layers B n = F -1 N -( n -1) in reverse order. An orbit of a dynamical system starting at x ∈ X is the sequence of states produced by the sequence of maps when initialized

C4

S4iy

1

S0

S2

F4(c3)

S1

S0

E

ic0

c0

C1

C2

4 = F4(c3)

at x . Therefore, the forward and backward orbits initialized at x ∈ X are

Given numerical implementations ˆ F n ≈ F n and ˆ B n ≈ B n with tolerance δ > 0 , i.e.,

we define the forward and backward pseudo-dynamics to be ( ˆ F n ) N n =1 and ( ˆ B n ) N n =1 , respectively, along with their forward and backward pseudo-orbits initialized at x ∈ X :

For notational brevity, we use subscripts on elements of X throughout to denote forward/backward orbit states. For example, given a random element Z ∈ X drawn from some distribution, Z k is the k th state generated by the forward dynamics ( F n ) N n =1 initialized at Z , and Z -k is the k th state generated by the backward dynamics ( B n ) N n =1 initialized at Z . Hat accents denote the same for the pseudo-dynamics: for example, ˆ Z k is the k th state generated by ( ˆ F n ) N n =1 when initialized at Z .

The forward/backward orbits satisfy x k +1 = F k ( x k ) and x -( k +1) = B k ( x -k ) exactly at each step. On the other hand, the forward/backward pseudo-orbits incur a small amount of error,

which can be magnified quickly along the orbit. However, there often exists another exact orbit starting at some other point s ∈ X that remains in a close neighbourhood of the numerical orbit. This property, illustrated in Fig. 3, is referred to as the shadowing property [19, 28-31].

Definition 4.1 (Shadowing property [28]) . The forward dynamics ( F n ) N n =1 has the ( ϵ, δ ) -shadowing property if for all x ∈ X and all ( ˆ F n ) N n =1 satisfying Eq. (6) , there exists an s ∈ X such that

An analogous definition holds for the backward dynamics ( B n ) N n =1 -where there is a shadowing orbit s -k that is nearby the pseudo-orbit ˆ x -k -and for the joint forward and backward dynamics, where there is a shadowing orbit nearby both the backward and forward pseudo-orbits simultaneously. The key idea in this paper is that, intuitively, if the numerically computed pseudo-orbit is close to

some exact orbit (the shadowing orbit), statistical computations based on the pseudo-orbit-e.g., sampling, density evaluation, and ELBO estimation-should be close to those obtained via that exact orbit. We will defer the examination of when (and to what extent) shadowing holds to Section 5; in this section, we will use it as an assumption when analyzing statistical computations with numerically implemented variational flows.

4.2 Error analysis of normalizing flows via shadowing

Sampling. Our first goal is to relate the marginal distributions of X N and ˆ X N for X ∼ q 0 , i.e., to quantify the error in sampling due to numerical approximation. Assume the normalizing flow has the ( ϵ, δ ) -shadowing property, and let ξ 0 be the marginal distribution of the shadowing orbit start point. We suspect that ξ 0 ≈ q 0 , in some sense; for example, we know that the bounded Lipschitz distance D BL ( ξ 0 , q 0 ) is at most ϵ due to shadowing. And ξ 0 is indeed an implicit function of q 0 ; it is a fixed point of a twice differentiable function involving the whole orbit starting at q 0 [17, Page. 176]. But the distribution ξ 0 is generally hard to describe more completely, and thus it is common to impose additional assumptions. For example, past work shows that the Lévy-Prokhorov metric D LP ( ˆ X N , X N ) is bounded by ϵ , under the assumption that ξ 0 = q 0 [23]. We provide a more general result (Proposition 4.2) without distributional assumptions on ξ 0 . We control D BL ( · , · ) rather than D LP ( · , · ) , as its analysis is simpler and both metrize weak distance.

Proposition 4.2. Suppose the forward dynamics has the ( ϵ, δ ) -shadowing property, and X ∼ q 0 . Then

In particular, with ℓ = U = 1 , we obtain that D BL ( X N , ˆ X N ) ≤ ϵ +2D TV ( ξ 0 , q 0 ) .

Recall the layerwise error bound from Eq. (5)-which suggests that the difference in orbit and pseudo-orbit grows exponentially in N -and compare to Proposition 4.2, which asserts that the error is controlled by the shadowing window size ϵ . This window size may depend on N , but we find in Section 6 it is usually not much larger than δ in practice, which itself is typically near the precision of the relevant numerical representation. We will show how to estimate ϵ later in Section 5.

Density evaluation. The exact density q ( x ) follows Eq. (1), while the approximation ˆ q ( x ) is the same except that we use the backward pseudo -dynamics. For x ∈ X , a differentiable function g : X ↦→ R + , define the local Lipschitz constant for the logarithm of g around x as:

Theorem 4.3 shows that, given the shadowing property, the numerical error is controlled by the shadowing window size ϵ and the sum of the Lipschitz constants along the pseudo-orbit. The constant L q,ϵ occurs because we are essentially evaluating q ( s ) rather than q ( x ) , where s ∈ X is the backward shadowing orbit initialization. The remaining constants occur because of the approximation of the shadowing orbit with the nearby, numerically-computed pseudo-orbit in the density formula.

Theorem 4.3. Suppose the backward dynamics has the ( ϵ, δ ) -shadowing property. Then

ELBO estimation. The exact ELBO estimation function is given in Eq. (2); the numerical ELBO estimation function ˆ E ( x ) is the same except that we use the forward pseudo -dynamics. The quantity E ( X ) , X ∼ q 0 is an unbiased estimate of the exact ELBO, while ˆ E ( X ) is biased by numerical error; Theorem 4.4 quantifies this error. Note that for simplicity we assume that the initial state distributions q 0 , ξ 0 described earlier are identical. It is possible to obtain a bound including a D TV ( q 0 , ξ 0 ) term, but this would require the assumption that log p ( x ) /q ( x ) is uniformly bounded on X .

Theorem 4.4. Suppose the forward dynamics has the ( ϵ, δ ) -shadowing property, and ξ 0 = q 0 . Then

4.3 Error analysis of MixFlows via shadowing

The error analysis for MixFlows for finite length N ∈ N parallels that of normalizing flows, except that F n = F , B n = B , and J n = J for n = 1 , . . . , N . However, when F and B are ergodic and measure-preserving for the target π , we provide asymptotically simpler bounds in the large N limit that do not depend on the difficult-to-analyze details of the Lipschitz constants along a pseudo-orbit. These results show that the error of sampling tends to be constant in flow length, while the error of density evaluation and ELBO estimation grows at most linearly. We say the forward dynamics has the infinite ( ϵ, δ ) -shadowing property if ( F n ) ∞ n =1 has the ( ϵ, δ ) -shadowing property [32]. Analogous definitions hold for both the backward and joint forward/backward dynamics.

Sampling. Similar to Proposition 4.2 for normalizing flows, Proposition 4.5 bounds the error in exact draws Y and approximate draws ˆ Y from the MixFlow. The result demonstrates that error does not directly depend on the flow length N , but rather on the shadowing window ϵ . In addition, in the setting where the flow map F is π -ergodic and measure-preserving, we can (asymptotically) remove the total variation term between the initial shadowing distribution ξ 0 and the reference distribution q 0 . Note that in the asymptotic bound, the distributions of Y and ˆ Y are functions of N .

Proposition 4.5. Suppose the forward dynamics has the ( ϵ, δ ) -shadowing property, and X ∼ q 0 . Let Y = X K , ˆ Y = ˆ X K for K ∼ Unif { 0 , 1 , . . . , N } . Then

In particular, if ℓ = U = 1 , we obtain that D BL ( Y, ˆ Y ) ≤ ϵ + 2D TV ( ξ 0 , q 0 ) . If additionally the forward dynamics has the infinite ( ϵ, δ ) -shadowing property, F is π -ergodic and measure-preserving, and ξ 0 , q 0 ≪ π , then

Adirect corollary of the second result in Proposition 4.5 is that for W ∼ π , lim N →∞ D BL ( W, ˆ Y ) ≤ ϵ , which results from the fact that D BL ( W,Y ) → 0 [5, Theorem 4.1]. In other words, the bounded Lipschitz distance of the approximated MixFlow and the target π is asymptotically controlled by the shadowing window ϵ . This improves upon earlier theory governing approximate MixFlows, for which the best guarantee available had total variation error growing as O ( N ) [5, Theorem 4.3].

Density evaluation. The exact density follows Eq. (3); the numerical approximation is the same except that we use the backward pseudo -dynamics. We obtain a similar finiteN result as in Theorem 4.3. Further, we show that given infinite shadowing-where the shadowing window size ϵ is independent of flow length N -the numerical approximation error asymptotically grows at most linearly in N , in proportion to ϵ .

Theorem 4.6. Suppose the ( ϵ, δ ) -shadowing property holds for the backward dynamics. Then

If additionally the backward dynamics has the infinite ( ϵ, δ ) -shadowing property, F is π -ergodic and measure-preserving, L q,ϵ ( x ) = o ( N ) as N →∞ , and ξ 0 ≪ π , then for q 0 -almost every x ∈ X ,

ELBO estimation. The exact ELBO formula for the MixFlow is given in Eq. (4). Note that here we do not simply substitute the forward/backward pseudo-orbits as needed; the naïve approximation of the terms q ( F n x ) would involve n applications of ˆ F followed by N applications of ˆ B , which do not exactly invert one another. Instead, we analyze the method proposed in [5], which involves simulating a single forward pseudo-orbit ˆ x 1 , . . . , ˆ x N and backward pseudo-orbit ˆ x -1 , . . . , ˆ x -N starting from x ∈ X , and then caching these and using them as needed in the exact formula.

Theorem 4.7. Suppose the joint forward and backward dynamics has the ( ϵ, δ ) -shadowing property, and ξ 0 = q 0 . Then for X ∼ q 0 ,

If additionally the joint forward and backward dynamics has the infinite ( ϵ, δ ) -shadowing property, F is π -ergodic and measure-preserving, and for some 1 ≤ m 1 < ∞ and 1 /m 1 + 1 /m 2 = 1 L p,ϵ , L q 0 ,ϵ , L J,ϵ ∈ L m 1 ( π ) and d q 0 d π ∈ L m 2 ( π ) , then

5 Computation of the shadowing window size

We have so far assumed the ( ϵ, δ ) -shadowing property throughout. To make use of our results in practice, it is crucial to understand the shadowing window size ϵ . Theorem 5.1 presents sufficient conditions for a finite dynamical system to have the shadowing property, and characterizes the size of the shadowing window ϵ . Note that throughout this section we focus on the forward dynamics; the backward and joint dynamics can be treated identically. Let ∥ · ∥ denote the spectral norm of a matrix.

Theorem 5.1 (Finite shadowing theorem) . Suppose the dynamics ( F n ) N n =1 are C 2 diffeomorphisms on X , and the pseudo-dynamics ( ˆ F n ) N n =1 satisfy Eq. (6) . For a given x ∈ X , define the operator A : X N +1 ↦→X N by

Let

If 2 Mλ 2 δ ≤ 1 , then the pseudo-orbit starting at x is shadowed with ϵ = 2 λδ .

Proof. This result follows directly by following the proof of [18, Theorem 11.3] with nonconstant flow maps F n , and then using the right-inverse norm from [33, Corollary 4.2].

In order to apply Theorem 5.1 we need to (1) estimate δ , e.g., by examining numerical error of one step of the map in practice; (2) compute λ min ( AA T ) ; and (3) estimate M , e.g., by bounding the third derivative of F n . While estimating M and δ are problem-specific, one can employ standard procedures for computing λ min ( AA T ) . The matrix AA T has a block-tridiagonal form,

where D k = ∇ F k (ˆ x k -1 ) ∈ R d × d , ∀ k ∈ [ N ] . Notice that AA T is a symmetric positive definite sparse matrix with bandwidth d and so has O ( Nd 2 ) entries. The inherent structured sparsity can be leveraged to design efficient eigenvalue computation methods, e.g., the inverse power method [34], or tridiagonalization via Lanczos iterations [35] followed by divide-and-conquer algorithms [36]. However, in our experiments, directly calling the eigmin function provided in Julia suffices; as illustrated in Figs. 14 and 19, the shadowing window computation requires only a few seconds for low dimensional synthetic examples with hundreds flow layers, and several minutes for real data examples. Hence, we didn't pursue specialized methods, leaving that for future work. It is also noteworthy that practical computations of D k can introduce floating-point errors, influencing the accuracy of λ . To address this, one might consider adjusting the shadowing window size. We explain how to manage this in Appendix B.1, and why such numerical discrepancies minimally impact results.

(b) Cross

(c) Lin. Reg.

(d) Log. Reg.

MixFlowsamplingerror

MixFlowsamplingerror

MixFlowsamplingerror

MixFlowsamplingerror

Rel. err.

1

0

101

23

err.

10

10

2345６

err.

10

10

5

err.

10

10

5

2

10

1

456

7

|x

sinx+1

sigmoid

Rel.

10

10

1

10

7

sinx+1

sigmoid

10~14

10~11

10

8

sinx+1

sigmoid

10~

10

10

-14

11

8

sinx+1

sigmoid

200

400

600

800

0020030040000600

410020030040000600

#transformations

#transformations

#transformations

#transformations

(b) Cross

(c) Lin. Reg.

(d) Log. Reg.

MixFlowlog-densityerror

MixFlowlog-densityerror

MixFlowlog-densityerror

MixFlowlog-densityerror

R

err.

10~12

10

10-8

10

10

15

5

8

2

Rel.

err.

10~12

10

10

10

10

15

8

err.

10

10

10

10

10

10

12

15

18

3

6

R

err.

10~12

10

10

10

10

-15

4

6

2

0020030040000600

200

400

600

800

0100200300400500600

002003004000600

#transformations

#transformations

#transformations

#transformations

(b) Cross

(c) Lin. Reg.

(d) Log. Reg.

MixFlowELBO est.

MixFlowELBOest.

MixFlowELBO est.

MixFlowELBOest.

B

-0.1

-0.3

-0.4

-0.5

-0.2

-0.6

exact

numerical

E

B

-0.5

-0.3

-0.4

-0.6

-0.2

-0.7

0.1

numerical

exact

O

B

E

-2210.5

-2210.0

-2209.0

-2209.5

-2211.0

numerical

exact

B

-250.6

-250.4

-250.5

-250.7

exact

numerical

0

100200300400

500

0

200

400

600

800

#transformations

#transformations

#transformations

#transformations

(b) Cross

(c) Linear regression

(d) Logistic regression

MixFlowEsize

MixFlow E size

MixFlowEsize

MixFlowEsize

6.0×10-12

4.0×10-12

2.0×10-12

Fwd

8.0×10-13

6.0×10-13

4.0×10

2.0×10

-13

13

Fwd

1.20×10

1.00×10

8.00×10

6.00×10

4.00×10

2.00×10

10

10

11

11

11

11

Fwd

3.0×10-11

2.0×10-11

1.0×10-11

4.0×10-11

Fwd

0

0

0

0

0

200

400

600

800

10020030040000600

#transformations

#transformations

#transformations

#transformations

Finally, since ϵ depends on λ min ( AA T ) -1 / 2 , which itself depends on N , the shadowing window size may potentially increase with N in practice. Understanding this dependence accurately for a general dynamical system is challenging; there are examples where it remains constant, for instance (see Examples B.1 and B.2 in Appendix B.2), but ϵ may grow with N . Our empirical results in next section show that on representative inferential examples with over 500 flow layers, ϵ scales roughly linearly with N and is of a similar order of magnitude as the floating point representation error. For further discussion on this topic, we refer readers to Appendix B.2.

6 Experiments

In this section, we verify our error bounds and diagnostic procedure of MixFlow on the banana, cross, and 2 real data targets-a Bayesian linear regression and logistic regression posterior; detailed

model and dataset descriptions can be found in Appendix D.1. We also provide a similar empirical investigation for Hamiltonian flow [6] on the same synthetic targets in Appendix C.2.

We begin by assessing the error of trajectory-averaged estimates based on approximate draws. Fig. 4 displays the relative numerical estimation error compared to the exact estimation based on the same initial draw. Although substantial numerical error was observed in orbit computation (see Fig. 16 in Appendix D.3), the relative sample estimate error was around 1% for all four examples, suggesting that the statistical properties of the forward orbits closely resemble those of the exact orbit.

We then focus on the density evaluation. For synthetic examples, we assessed densities at 100 evenly distributed points within the target region (Figs. 2a and 2b). For real data, we evaluated densities at the locations of 100 samples from MCMC method None-U-turn sampler ( NUTS ); detailed settings for NUTS is described in Appendix D.2. It is evident from the relative error shown in Fig. 5 that the numerical density closely matches the exact density evaluation, with the relative numerical error ranging between 0 . 1% and 1% , which is quite small. The absolute error can be found in Fig. 17 in Appendix D.3, showing that the density evaluation error does not grow as substantially as the orbit computation error (Fig. 16 in Appendix D.3), which aligns with the bound in Theorem 4.6.

Fig. 6 further demonstrates the numerical error for the ELBO estimations (Eq. (4)). In each example, both the averaged exact ELBO estimates and numerical ELBO estimates are plotted against an increasing flow length. Each ELBO curve is averaged over 200 independent forward orbits. Across all four examples, the numerical ELBO curve remain closely aligned with the exact ELBO curve, indicating that the numerical error is small in comparison to the scale of the ELBO value. Moreover, the error does not grow with an increasing flow length, and even presents a decreasing trend as N increases in the two synthetic examples and the Logistic regression example. This aligns well with the error bounds provided in Theorem 4.7.

Finally, Fig. 7 presents the size of the shadowing window ϵ as the flow length N increases. As noted earlier, ϵ depends on the flow map approximation error δ and potentially N . We evaluated the size of δ by calculating the approximation error of a single F or B for 1000 i.i.d. draws from the reference distribution. Boxplots of δ for all examples can be found in Fig. 18 of Appendix D.3. These results show that in the four examples, δ is close to the floating point representation error (approximately 10 -14 ). Thus, we fixed δ at 10 -14 when computing ϵ . As shown in Fig. 7, ϵ for both the forward and backward orbits grows roughly linearly with the flow length. This growth is significantly less drastic than the exponential growth of the orbit computation error. And crucially, the shadowing window size is very small-smaller than 10 -10 for MixFlow of length over 1000 , which justifies the validity of the downstream statistical procedures. In contrast, the orbit computation errors in the two synthetic examples rapidly reach a magnitude similar to the scale of the target distributions, as shown in Fig. 1.

7 Conclusion

This work delves into the numerical stability of variational flows, drawing insights from shadowing theory and introducing a diagnostic tool to assess the shadowing property. Experiments corroborate our theory and demonstrate the effectiveness of the diagnostic procedure. However, the scope of our current analysis is limited to downstream tasks post-training of discrete-time flows. Understanding the impact of numerical instability during the training phase or probing into recent architectures like continuous normalizing flows or neural ODEs [21, 37-40], remains an avenue for further exploration. Additionally, while our theory is centered around error bounds that are proportional to the shadowing window size ϵ , we have recognized that ϵ can grow with N . Further study on its theoretical growth rate is needed. Finally, in our experiments, we employed a basic method to calculate the minimum eigenvalue of AA T . Given the sparsity of this matrix, a deeper exploration into a more efficient computational procedure is merited.

References

Esteban Tabak and Cristina Turner. A family of non-parametric density estimation algorithms. Communications on Pure and Applied Mathematics , 66(2):145-164, 2013.

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning , 2015.

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research , 22:1-64, 2021.

Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: an introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(11):3964-3979, 2021.

Zuheng Xu, Naitong Chen, and Trevor Campbell. MixFlows: principled variational inference via mixed flows. arXiv:2205.07475 , 2022.

Naitong Chen, Zuheng Xu, and Trevor Campbell. Bayesian inference via sparse Hamiltonian flows. In Advances in Neural Information Processing Systems , 2022.

Anthony Caterini, Arnaud Doucet, and Dino Sejdinovic. Hamiltonian variational auto-encoder. In Advances in Neural Information Processing Systems , 2018.

Solomon Kullback and Richard Leibler. On information and sufficiency. The Annals of Mathematical Statistics , 22(1):79-86, 1951.

Matthew Hoffmann, David Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research , 14:1303-1347, 2013.

Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, and David Blei. Automatic Differentiation Variational Inference. Journal of Machine Learning Research , 18(14), 2017.

Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In International Conference on Artificial Intelligence and Statistics , 2014.

Jonas Köhler, Andreas Krämer, and Frank Noe. Smooth normalizing flows. In Advances in Neural Information Processing Systems , 2021.

Zhifeng Kong and Kamalika Chaudhuri. The expressive power of a class of normalizing flow models. In International Conference on Artificial Intelligence and Statistics , 2020.

Jens Behrmann, Paul Vicol, Kuan-Chieh Wang, Roger Grosse, and Jörn-Henrik Jacobsen. Understanding and mitigating exploding inverses in invertible neural networks. In International Conference on Artificial Intelligence and Statistics , 2021.

Shifeng Zhang, Chen Zhang, Ning Kang, and Zhenguo Li. ivpf: Numerical invertible volume preserving flow for efficient lossless compression. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Advances in Neural Information Processing Systems , 2017.

Brian Coomes, Hüseyin Koçak, and Kenneth Palmer. Shadowing in discrete dynamical systems. In Six lectures on dynamical systems , pages 163-211. World Scientific, 1996.

Kenneth Palmer. Shadowing in dynamical systems: theory and applications , volume 501. Springer Science &Business Media, 2000.

Sergei Yu Pilyugin. Shadowing in dynamical systems . Springer, 2006.

Lucas Backes and Davor Dragiˇ cevi´ c. Shadowing for nonautonomous dynamics. Advanced Nonlinear Studies , 19(2):425-436, 2019.

Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible architectures for arbitrarily deep residual neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , 2018.

Wael Bahsoun, Huyi Hu, and Sandro Vaienti. Pseudo-orbits, stationary measures and metastability. dynamical systems , 29(3):322-336, 2014.

Paul Tupper. The relation between approximation in distribution and shadowing in molecular dynamics. SIAM Journal on Applied Dynamical Systems , 8(2):734-755, 2009.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In International Conference on Learning Representations , 2017.

Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems , 2019.

Rianne van den Berg, Leonard Hasenclever, Jakub Tomczak, and Max Welling. Sylvester normalizing flows for variational inference. In Conference on Uncertainty in Artificial Intelligence , 2018.

Sameera Ramasinghe, Kasun Fernando, Salman Khan, and Nick Barnes. Robust normalizing flows using Bernstein-type polynomials. arXiv:2102.03509 , 2021.

Sergei Yu Pilyugin. Theory of pseudo-orbit shadowing in dynamical systems. Differential Equations , 47 (13):1929-1938, 2011.

Abbas Fakhari and Helen Ghane. On shadowing: ordinary and ergodic. Journal of Mathematical Analysis and Applications , 364(1):151-155, 2010.

Robert Corless and Sergei Yu Pilyugin. Approximate and real trajectories for generic dynamical systems. Journal of mathematical analysis and applications , 189(2):409-423, 1995.

Jonathan Meddaugh. Shadowing as a structural property of the space of dynamical systems. arXiv:2106.01957 , 2021.

Sergei Yu Pilyugin and O.B. Plamenevskaya. Shadowing is generic. Topology and its Applications , 97(3): 253-266, 1999.

Ivan Dokmani´ c and Rémi Gribonval. Beyond moore-penrose part i: generalized inverses that minimize matrix norms. arXiv:1706.08349 , 2017.

Michelle Schatzman. Numerical Analysis: a Mathematical Introduction . Oxford University Press, 2002.

Cornelius Lanczos. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. 1950.

Ed S. Coakley and Vladimir Rokhlin. A fast divide-and-conquer algorithm for computing the spectra of real symmetric tridiagonal matrices. Applied and Computational Harmonic Analysis , 2013.

Juntang Zhuang, Nicha C Dvornek, sekhar tatikonda, and James s Duncan. MALI: A memory efficient and reverse accurate integrator for neural ODEs. In International Conference on Learning Representations , 2021.

Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ODE. In International Conference on Machine Learning , 2020.

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems , 2018.

Patrick Kidger. On neural differential equations. arXiv:2202.02435 , 2022.

Yu Qiao and Nobuaki Minematsu. A study on invariance of f -divergence and its application to speech recognition. IEEE Transactions on Signal Processing , 58(7):3884-3890, 2010.

George Birkhoff. Proof of the ergodic theorem. Proceedings of the National Academy of Sciences , 17(12): 656-660, 1931.

Tanja Eisner, Bálint Farkas, Markus Haase, and Rainer Nagel. Operator Theoretic Aspects of Ergodic Theory . Graduate Texts in Mathematics. Springer, 2015.

Brian Coomes, Hüseyin Koçak, and Kenneth Palmer. Rigorous computational shadowing of orbits of ordinary differential equations. Numerische Mathematik , 69:401-421, 1995.

Joseph Frederick Elliott. The characteristic roots of certain real symmetric matrices. Master's thesis, University of Tennessee - Knoxville, 1953.

Sergey Yu Pilyugin. Spaces of dynamical systems . de Gruyter, 2 nd edition, 2019.

Tomas Geffner and Justin Domke. MCMC variational inference via uncorrected Hamiltonian annealing. In Advances in Neural Information Processing Systems , 2021.

Guodong Zhang, Kyle Hsu, Jianing Li, Chelsea Finn, and Roger Grosse. Differentiable annealed importance sampling and the perils of gradient noise. In Advances in Neural Information Processing Systems , 2021.

Tomas Geffner and Justin Domke. Langevin diffusion variational inference. In International Conference on Artificial Intelligence and Statistics , 2023.

Achille Thin, Nikita Kotelevskii, Alain Durmus, Maxim Panov, Eric Moulines, and Arnaud Doucet. Monte Carlo variational auto-encoders. In International Conference on Machine Learning , 2021.

Heikki Haario, Eero Saksman, and Johanna Tamminen. An adaptive Metropolis algorithm. Bernoulli , pages 223-242, 2001.

Athanasios Tsanas, Max A. Little, Patrick E. McSharry, and Lorraine O. Ramig. Accurate telemonitoring of parkinson's disease progression by noninvasive speech tests. IEEE Transactions on Biomedical Engineering , 57:884-893, 2010. URL https://api.semanticscholar.org/CorpusID:7382779 .

Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems , 62:22-31, 2014.

Radford Neal. MCMC using Hamiltonian dynamics. In Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng, editors, Handbook of Markov chain Monte Carlo , chapter 5. CRC Press, 2011.

Kai Xu, Hong Ge, Will Tebbutt, Mohamed Tarek, Martin Trapp, and Zoubin Ghahramani. AdvancedHMC.jl: A robust, modular and efficient implementation of advanced HMC algorithms. In Symposium on Advances in Approximate Bayesian Inference , 2020.

A Proofs

A.1 Layer-wise error analysis

The layer-wise error analysis follows by recursing the one-step bound on the difference between the exact and numerical flow:

A.2 Error bounds for normalizing flows

Proof of Proposition 4.2. Let S be the random initial state of the orbit that shadows the pseudo-orbit of X ∼ q 0 . By triangle inequality,

By ϵ -shadowing and ℓ -Lipshitz continuity of f ,

Next

The last equality is due to the fact that X N and S N are the map of X and S , respectively, under the bijection F N ◦ · · · ◦ F 1 , and the fact that the total variation is invariant under bijections [41, Theorem 1].

Proof of Theorem 4.3. Let s ∈ X be the initial state of the backward shadowing orbit. By triangle inequality,

For the first term on the right-hand side, note that for some y on the segment from x to s ,

For the second term,

We apply the same technique to bound each term as in Eq. (7).

Proof of Theorem 4.4. By definition, ELBO( q || p ) = E [ E ( X )] , and by hypothesis the distribution q 0 of the initial flow state is equal to the distribution ξ 0 of the shadowing orbit initial state, so E [ E ( X )] = E [ E ( S )] .

The proof is completed by bounding each difference with the local Lipschitz constant around the pseudo-orbit times the shadowing window size ϵ , and applying the expectation.

A.3 Error bounds for MixFlows

Proof of Proposition 4.5. Let S be the initial point of the random shadowing orbit. By the definition of MixFlows,

To bound the second term, we apply shadowing and the Lipschitz continuity of f to show that each ∣ ∣ ∣ f ( S n ) -f ( ˆ X n ) ∣ ∣ ∣ ≤ ℓϵ , and hence their average has the same bound. For the first term, in the case where N is fixed and finite,

We now consider the largeN limiting case when F is π -ergodic and measure-preserving, q 0 ≪ π and ξ 0 ≪ π , and U = ℓ = 1 . Let Z = S K , K ∼ Unif { 0 , 1 , . . . , N } , and let W ∼ π . In this case, by the triangle inequality,

By [5, Theorem 4.1], both terms on the right-hand side converge to 0 as N →∞ .

Lemma A.1. If ∀ i ∈ [ n ] , a i , b i > 0 , then

Proof. Define A := ∑ n i =1 a i ,

Jensen's inequality yields that

Applying the same technique to the other direction yields the result.

Proof of Theorem 4.6. First, shadowing and the triangle inequality yields

where q n is the density of the lengthn normalizing flow with constant forward flow map F , and ˆ q n is its approximation via the pseudo-orbit. Using the same technique as in the proof of Theorem 4.3,

Combining this with the earlier bound yields the first stated result. To obtain the second result in the infinite shadowing setting, we repeat the process of bounding max 0 ≤ n ≤ N | log ˆ q n ( x ) -log q n ( s ) | , but rather than expressing each supremum around the pseudo-orbit ˆ x -n , we express it around the shadowing orbit s -n to find that

We then bound max 0 ≤ n ≤ N with a sum and merge this with the first bound to find that

Since L q,ϵ ( x ) = o ( N ) as N →∞ , the first term decays to 0. The latter two terms are ergodic averages under the backward dynamics initialized at s ; if the pointwise ergodic theorem [42], [43, p. 212] holds at s , then the result follows. However, the pointwise ergodic theorem holds only π -almost surely for each of L q 0 ,ϵ and L J,ϵ . Denote Z to be the set of π -measure zero for which the theorem does not apply. Suppose there is a set A ⊆ X such that q 0 ( A ) > 0 , and x ∈ A implies that s ∈ Z . But then ξ 0 ( Z ) > 0 , which contradicts ξ 0 ≪ π . Therefore the pointwise ergodic theorem applies to s for q 0 -almost every x ∈ X .

Proof of Theorem 4.7. By definition, ELBO( q || p ) = E [ E ( X )] , and by hypothesis the distribution q 0 of the initial flow state is equal to the distribution ξ 0 of the joint shadowing orbit initial state, so E [ E ( X )] = E [ E ( S )]

. Hence ∣ ∣ ∣ ELBO( q || p ) -E [ ˆ E ( X ) ]∣ ∣ ∣ ≤ E [∣ ∣ ∣ E ( S ) -ˆ E ( X ) ∣ ∣ ∣ ] . So applying the triangle inequality, we find that

For the first sum, each term is bounded using the local Lipschitz constant,

For the second, we apply Lemma A.1 and then use the local Lipschitz constants,

Combining this with the previous bound and taking the expectation yields the first result,

Once again to arrive at the second result, we redo the analysis but center the supremum Lipschitz constants around the shadowing orbit points, and replace the maximum with a sum, yielding the bound

Now since ξ 0 = q 0 by assumption, S n d = X n for each n ∈ N , so

We divide by N and take the limit supremum:

Consider just the term

[43, Theorem 8.10 (vi)] asserts that as long as L J,ϵ ∈ L m 1 ( π ) and d q 0 d π ∈ L m 2 ( π ) for 1 /m 1 +1 /m 2 = 1 , 1 ≤ m 1 < ∞ , then

Applying this result to each term above yields the stated result.

A.4 Norm of matrix right inverse

Notice that the operator A in Theorem 5.1 has a following matrix representation:

Proposition A.2 ([33, Corollary 4.2]) . Let A be as defined in Eq. (8) . Suppose ( F n ) N n =1 are all differentiable. Then

and ∥ A † ∥ = σ min ( A ) -1 , where σ min ( A ) denotes the smallest singular value of A .

Proof of Proposition A.2. Note that A has full row rank due to the identity blocks. In this case, AA T is invertible, thus A T ( AA T ) -1 is a valid right inverse of A . Eq. (9) is then obtained by a direct application of [33, Corollary 4.2].

To obtain the norm of A † , since A has full row rank,

Then

Notice that since AA T is an invertible Hermitian matrix, σ min ( A ) is strictly positive.

B Discussion about the shadowing window ϵ

B.1 Numerical error when computing ϵ

As mentioned in Section 5, due to the floating-point error δ involved in evaluating ∇ F k , the evaluation of A is perturbed by the numerical error. This consequntely leads to an inaccuracy of the estimation of λ = ∥ A -1 r ∥ . Here A -1 r denotes the right inverse of A . In this section, we explain how to calibrate the calculation of λ to take into account of this error.

This computational challenge, rooted in numerical inaccuracies, was previously acknowledged in Coomes et al. [17, Section 3.4], with a provided remedy. Suppose we compute terms in ∇ F k (ˆ x k -1 ) with a floating-point error δ . This introduces O ( √ N δ ) Frobenius norm error into the practically calculated A . Here we denote the digital computed A by ˜ A , ˜ λ := ∥ ˜ A -1 r ∥ , and ˜ δ := ∥ A -˜ A ∥ . According to Coomes et al. [17, Eq. 45], we can employ the following upper bound on λ :

Substituting values of δ , N , and computed ˜ λ from our experiments (refer to Fig. 7), this amounts to an inconsequential relative error of around (1 -10 -9 ) -1 ≈ 1 on λ .

It is also worth discussing the the shadowing window computation methods proposed in Coomes et al. [17, Section 3.4]. As opposed to our generic eigenvalue computation introduced in Section 5, the computation procedure presented in Coomes et al. [17, Section 3.4] can potentially offer better scalability. However, the methodology in Coomes et al. [17] operates under the heuristic that a dynamical system with shadowing is intrinsically hyperbolic, or nearly so. Specifically, the 'hyperbolic splitting' discussed in Coomes et al. [44, P.413, Appendix B] (i.e., the choice of ℓ ), and the 'hyperbolic threshold' choice in Coomes et al. [44, P.415, Appendix B] (i.e., the choice of p ), both implicitly assume that the dynamics either exhibit hyperbolicity or tend closely to it. Such presuppositions might be overly restrictive for generic variational flows, given that the underlying dynamics are general time-inhomogeneous systems. When the underlying dynamics are not (nearly) hyperbolic, applying strategies from Coomes et al. [17] could potentially lead to a poor estimation of the shadowing window.

B.2 Dependence of ϵ on N

One may notice that ϵ depends on N implicitly through λ min ( AA T ) -1 / 2 . Recall that A can be represented as a matrix

And the shadowing window is given by ϵ = 2 δλ min ( AA T ) -1 / 2 . If λ min ( AA T ) -1 / 2 scales badly with N , our O ( Nϵ ) error bounds become vacuous. Therefore, in this section, we collect some insights about the scaling of ϵ over increasing N . It is important to note that understanding the explicit quantitative relationship between ϵ and N for general nonautonomous dynamical systems is quite challenging. Although current literature of dynamical systems focuses on studying conditions so that ϵ does not scale with N , these conditions are typically difficult to verify in practice or are only applicable in restrictive settings such as time-homogeneous linear dynamical systems. For a more systematic study, see [17, 20, 28]. The examples provided below merely serve as synthetic illustrations for intuition and should not be taken as definitive statements for all cases.

Example B.1 (Scaling map in R ) . Consider the time-homogeneous dynamics formed by T : R ↦→ R such that ∀ x ∈ R , T ( x ) = Cx for some C > 0 . Then the corresponding shadowing window of a δ -pseudo-orbit is

̸

Therefore, for all C = 1 , as N →∞ , ϵ = O ( δ ) . For C = 1 , ϵ = O ( Nδ ) . The formula for λ min ( AA T ) above arises because in this case, AA T has the following symmetric tridiagonal form,

whose eigenvalues have the following closed form [45, Page. 19-20]:

◁

Example B.2 (Uniform hyperbolic linear map[46, Theorem 12.3 and p. 171]) . Consider the time-homogeneous linear dynamical system in R d , T : x ↦→ Ax , where A ∈ R d × d is similar to a block-diagonal matrix diag( A 1 , A 2 ) such that ∥ A 1 ∥ ≤ λ and ∥ A -1 2 ∥ ≤ λ for some λ ∈ (0 , 1) . Then for all N ∈ N , any δ -pseudoorbit is ϵ -shadowed, with ϵ = 1+ λ 1 -λ δ . ◁

As illustrated in Examples B.1 and B.2, it is possible for the ϵ -shadowing property to hold in a way that ϵ does not depend on the flow length N and remains in a similar magnitude as δ . This phenomenon is referred to as the infinite shadowing property [17, 19, 32] as mentioned in the beginning of Section 4.3. The infinite shadowing property is shown to be generic in the space of all dynamical systems [32], meaning that dynamical systems exhibiting this property form a dense subset of the collection of all dynamical systems. However, while genericity offers valuable insights into the typical behavior of dynamical systems, it does not imply that all systems possess the property. As previously explained, only a small class of systems has been shown to have the infinite shadowing property, and verifying this property for a specific given system remains a challenging task.

C Normalizing flow via Hamiltonian dynamics

While past work on normalizing flows [2, 3, 24-26] typically builds a flexible flow family by composing numerous parametric flow maps that are agnostic to the target distribution, recent target-informed variational flows focus on designing architectures that take into account the structures of the target distribution. A common approach for constructing such target-informed flows is to use Langevin and Hamiltonian dynamics as part of the flow transformations [6, 7, 47-50]. In this section, we begin with a concise overview of two Hamiltonian-based normalizing flows: Hamiltonian variational auto-encoder (HVAE) [7] and Hamiltonian flow (HamFlow) [6]. Subsequently, we present corollaries derived from Theorems 4.3 and 4.4, specifically applying these general results to HV AE and HamFlow. Finally, we offer empirical evaluations of our theory for HamFlow on the same synthetic examples as those examined for MixFlow.

Hamiltonian dynamics (Eq. (10)) describes the evolution of a particle's position θ t and momentum ρ t within a physical system. This movement is driven by a differentiable negative potential energy log p ( θ t ) and kinetic energy 1 2 ρ T t ρ t :

For t ∈ R , we define the mappings H t : R 2 d → R 2 d , transforming ( θ s , ρ s ) ↦→ ( θ t + s , ρ t + s ) according to the dynamics of Eq. (10). Intuitively, traversing the trajectory of H t , t ∈ R allows particles to efficiently explore the distribution of interest's support, thereby motivating the use of H t as a flow map. In practice, the exact Hamiltonian flow H t is replaced by its numerical counterpart-leapfrog integration-which involves K iterations of the subsequent steps:

Here ϵ denotes the step size of the leapfrog integrator. We use T K,ϵ : R 2 d → R 2 d to define the map by sequencing the above three steps K times. There are two key properties of T K,ϵ that make it suitable for normalizing flow constructions. T K,ϵ is invertible (i.e., T -1 K,ϵ = T K, -ϵ ) and has a simple Jacobian determinant of the form: | det ∇ T K,ϵ | = 1 . This enables a straightforward density transformation formula: for any density q 0 ( θ, ρ ) on R 2 d , pushforward of q under T K,ϵ has the density q 0 ( T K, -ϵ ( θ, ρ )) .

Given the above properties, Caterini et al. [7] and Chen et al. [6] developed flexible normalizing flowsHVAE and HamFlow, respectively-utilizing flow transformations through alternating compositions of multiple T K,ϵ and linear transformations. Notably, the momentum variable ρ introduced in the Hamiltonian dynamics Eq. (10) necessitates that both HV AE and HamFlow focus on approximating the augmented target distribution

¯ π ( θ, ρ ) ∝ π ( θ ) · exp( -1 2 ρ T ρ ) on R 2 d . This represents the joint distribution of the target π and an independent momentum distribution N (0 , I d ) .

Specifically, flow layers of HV AE alternate between the leapfrog transformation T K,ϵ and a momentum scaling layer:

And those of HamFlow alternate between T K,ϵ and a momentum standardization layer:

( θ, ρ ) ↦→ ( θ, Λ( ρ -µ )) , Λ ∈ R d × d is lower triangular with positive diagonals , µ ∈ R d .

C.1 Error analysis for Hamiltonian-based flows

We have provided error analysis for generic normalizing flows in Section 4.2. In this section, we specialize the error bounds for density evaluation (Theorem 4.3) and ELBO estimation (Theorem 4.4) for the two Hamiltonianbased flows. π denotes the target distribution of interest on R d , and ¯ p denotes the augmented target distribution on R 2 d . p and ¯ p respectively denote the unnormalized density of π and ¯ π .

Corollary C.1. Let q be either HVAE or HamFlow of N layers with q 0 = N (0 , I 2 d ) . Suppose the backward dynamics has the ( ϵ, δ ) -shadowing property. Then

Proof of Corollary C.1. Since the Jacobian determinant of each HVAE/HamFlow layer is constant,

And since q 0 is a standard normal distribution,

The proof is then complete by directly applying Eqs. (11) and (12) to the error bound stated in Theorem 4.3.

Corollary C.2. Let q be either HVAE or HamFlow of N layers with q 0 = N (0 , I 2 d ) . Suppose the forward dynamics has the ( ϵ, δ ) -shadowing property, and ξ 0 = q 0 . Then

If additionally log p is L -smooth (i.e., ∀ x, y, ∥∇ log p ( x ) -∇ log p ( y ) ∥ ≤ L ∥ x -y ∥ ), then

Proof of Corollary C.2. Similar to the proof of Fig. 2d, applying Eqs. (11) and (12) to the error bound stated in Theorem 4.4 yields that

The first result is then obtained by employing the following bound via Jensen's inequality:

To arrive at the second result, we apply the following bound on L p,ϵ using the fact that ∇ log p is L -Lipschitz continuous:

C.2 Empirical results for Hamiltonian flow

In this section, we empirically validate our error bounds and the diagnostic procedure of HamFlow using two synthetic targets: the banana and cross distributions. We employ a warm-started HamFlow, as detailed in [6, Section 3.4]. This is achieved through a single pass of 10,000 samples from the reference distribution, q 0 , without additional optimization. Specifically, the µ and Λ values for each flow standardization layer are determined by the sample mean and the Cholesky factorization of the sample covariance, respectively, of the batched momentum samples processed through the layer. For the banana distribution, we set the leapfrog step K at 200 and a step size ϵ of 0 . 01 . For the cross distribution, we set K to 100 with an ϵ value of 0 . 02 . Figs. 8a

and 8b showcase a comparison between 1000 i.i.d. samples from the warm-started HamFlow with 400 layers and the true targets. The results illustrate that HamFlow generates samples of decent quality.

We start by examining the numerical orbit computaiton error. As demonstrated in Fig. 9, the pointwise evaluation error grows exponentially with increasing flow depth, and quickly reaches the same order of magnitude as the scale of the target distributions. However, similar to what we have observed in the experiments of MixFlow (described in Section 6), the substantial numerical error of orbit computation does not impact the accuracy of Monte Carlo estimations of expectations and ELBO estimations significantly. Fig. 10 shows the relative numerical error of Monte Carlo estimates of the expectations of three test functions. The relative samples estimation error remains within 5% for the two bounded Lipschitz continuous test functions, aligning with our error bound in Proposition 4.2. Moreover, even for the unbounded test function, the error stays within reasonable limits: 10% for the banana and 5% for the cross examples. Fig. 11 then compares the numerical ELBO estimates and exact ELBO estimates for HamFlow over increasing flow length. In both examples, the numerical ELBO curve is almost identical to the exact one, notwithstanding the considerable sample computation error of the forward orbit.

However, unlike the small log-density evaluation error we observed in the MixFlow experiments, HamFlow's log-density shows significant issues because of the numerical errors of backward orbits computation. In Fig. 12, we measure the numerical error for the log-density evaluation of HamFlow at 100 i.i.d. points from the target distribution. As we add more flow layers, the error grows quickly. For the banana example, the log-density evaluation error can even go above 1000 . This raises two main questions: (1) Why is the ELBO estimation error different from the log-density evaluation error in HamFlow? (2) Why is the scaling of the log-density evaluation error different between MixFlow and HamFlow? Our theory presented in Section 4 and Appendix C.1 offers explanation to these questions.

To address the first question, we compare the bounds for HamFlow on log-density evaluation (Corollary C.1) and ELBO estimation (Corollary C.2). The main difference between these bounds is in the local Lipschitz constants. The error in log-density evaluation is based on L q,ϵ while the error in ELBO estimation is based on E [ L ¯ p,ϵ ] . These two quantitiesL q,ϵ and E [ L ¯ p,ϵ ] -relate to the smoothness of the learned log-density of HamFlow and log-density of the target. Figs. 8c to 8f display log p and log q side-by-side for both synthetic examples. Indeed, in both cases, log q markedly deviates from log p and presents drastic fluctuations, indicating that L q,ϵ can be substantially larger than E [ L ¯ p,ϵ ] . Fig. 13 further provides a quantitative comparison on how L q,ϵ and E [ L ¯ p,ϵ ] change as we increase the flow length. Note that since both L q,ϵ and L ¯ p,ϵ are intractable in general, we instead focus on the scaling of ∥∇ log q ∥ (which is a lower bound of L q,ϵ ) and an upper bound of E [ L ¯ p,ϵ ] (derived from case-by-case analysis for both the banana and cross distributions; see Appendix C.3 for detailed derivation). Results show that in both synthetic examples, L q,ϵ can increase exponentially over increasing flow length and is eventually larger than E [ L ¯ p,ϵ ] by 30 orders of magnitude, while E [ L ¯ p,ϵ ] does not grow and remains within a reasonable scale.

Similarly, the differences in log-density errors between MixFlow and HamFlow come from how smooth the learned log-density is in each method, according to the error bounds presented in Theorems 4.3 and 4.6. Fig. 2 visualizes the learned log-densities of MixFlow, which matches closely to the actual target distribution, and has a smoother output than HamFlow. This matches what was noted in Xu et al. [5, Figure 3], where MixFlow was shown to approximate the target density better than standard normalizing flows.

Finally, Fig. 14 presents the single step evaluation error δ of HamFlow and the size of the shadowing window ϵ as the flow length N increases. Results show that for both synthetic examples, δ is approximately 10 -7 , hence we fix δ to be this value when computing the shadowing window ϵ . Compared to the significant orbit evaluation error when N is large, the size of ϵ remains small for both the forward and backward orbits (in the scale of 10 -4 to 10 -3 when N = 400 ). And more importantly, the size of ϵ grows less drastically than the orbit computation error with the flow length-roughly quadratically in the banana example, and linearly in the cross example, which is similar to what is observed in the MixFlow experiments.

C.3 Upper bounds of L ¯ p,ϵ for synthetic examples

Recall that we denote ¯ p ( x ) ∝ p ([ x ] 1:2 ) · exp( -1 2 [ x ] T 3:4 [ x ] 3:4 ) the augmented target and p denotes the 2dsynthetic target we are interested.

Notice that

where L p,ϵ requires a case-by-case analysis. Therefore, in this section, we focus on bounding the local Lipschitz constant for log p .

Proposition C.3. For the 2-dimensional banana distribution, i.e.,

Proof of Proposition C.3. By the mean value theorem,

The proof will then complete by showing that

To verify Eq. (13), note that

The Gershgorin circle theorem then yields that

Additionally, we have and

The proof is then completed.

Proposition C.4. For the cross distribution, a 4-component Gaussian mixture of the form

we have that for all x ∈ R 2 ,

Proof of Proposition C.4. For notational convenience, denote the density of 4 Gaussian component by p 1 , p 2 , p 3 , p 4 , hence

Examining ∇ log p ( x ) directly yields that for all x ∈ R 2 ,

Here µ i and Σ i denotes the Gaussian mean and covariance of p i . Then by Jensen's inequality,

HamFlow

sample scatters

(b) Cross:

HamFlow

sample scatter

(c) True log-pdf

(d)

HamFlow

log-pdf

(e) True log-pdf

(f)

HamFlow

log-pdf

Trained HamFlow

Trained HamFlow

0

4

-50

2

0

-100

-2

-150

True Distribution

TrueDistribution

Trained Flow

-4

Trained Flow

0

20

40

-4

-2

0

2

4

(b) Banana (log-scale)

(c) Cross

(d) Cross (log-scale)

HamFlownumericalerror

HamFlownumericalerror

HamFlownumericalerror

HamFlownumericalerror

50

Fwd

10%

10.0

Fwd

10°

Fwd

Error

40

30

Error

10~5

rror

7.5

rror

20

10

10~10

Fwd

E

5.0

2.5

E

10

-10

0

0.0

0

100

200

300

400

0

100200300

400

0

100200

300

400

0

100200300

400

#transformations

#transformations

#transformations

#transformations

(b) Banana (log-scale)

(c) Cross

(d) Cross (log-scale)

HamFlowsamplingerror

HamFlowsamplingerror

HamFlowsamplingerror

HamFlowsamplingerror

0.100

0.04

err.

0.075

sin（x）+1

sigmoid

10~5

err.

0.03

sigmoid

sin(x)+1

10

0.050

0.025

10~10

sin（x）+1

0.02

0.01

R

D

10~10

sin（x）+1

0.000

10~15

sigmoid

0.00

10~15

sigmoid

0

100

200300

400

0

100200300

400

0

100200

300

400

0

100200300

400

#transformations

#transformations

#transformations

#transformations

(b) Cross

HamFlow ELBO est.

hamflowelboest.

-10

一

3

-20

4

O

o

5

B

-30

b

６

e

一

-40

numerical

一

7

numerical

exact

-8

exact

-50

0

100

200

300

400

0

100

200

300

400

#transformations

#transformations

(b) Banana (log-scale)

(c) Cross

(d) Cross (log-scale)

HamFlowlog-densityerror

HamFlowlog-densityerror

HamFlowlog-densityerror

HamFlowlog-densityerror

100

1500

10°

15

error

Error

1000

Error

10

Error

10

500

10~10

5

Rel.

10~10

0

0

0

100200300

400

0

100200300

400

0

100

200

300

400

0

100200300

400

#transformations

#transformations

#transformations

#transformations

(a) Cross

(c) Cross

HamFlow|/Vlogqll

HamFlow upper bound of Lpe

HamFlow|/Vlogql

HamFlow upper bound of Lpe

1030

10

280

8

260

1020

6

10°

30

240

1020

4

220

2

10°

200

0

100

200

300

400

0

100

200

300

400

0

100

200

300

400

0

100

200

300

400

#transformations

#transformations

#transformations

#transformations

D Additional experimental details

All experiments were conducted on a machine with an AMD Ryzen 9 3900X and 32G of RAM.

D.1 Model description

The two synthetic distributions tested in this experiment were

the banana distribution [51]:

(d) Cross

(e) Cross

(f) Cross

HamFlow single map err.

HamFlowwindowsize

Ecomputationtime

0.15

0.0015

Fwd

Fwd

Error

10~6

0.0010

0.10

0.0005

0.05

-9

10

0.0000

0.00

0

100200300400

0

100

200300

400

Fwd err.

#transformations

#transformations

HamFlow single map err.

HamFlow window size

E computationtime

10%/

!

0.000100

Fwd

0.06

Fwd

0.000075

Error

0.04

0.000050

10~5

0.000025

0.02

0.000000！

0.00

10~10

0

100200300400

0

100

200

300

400

Fwd err.

#transformations

#transformations

backward error { ∥ F n x 0 -ˆ B ◦ F n +1 x 0 ∥ : n = 1 , . . . , N } ( Bwd err. ). These errors are computed along precise forward trajectories that originate at x 0 , which is sampled 50 times independently and identically from the reference distribution q 0 . N here denotes the flow length for each example. Figure (b) and (e) show ϵ over increasing flow length. Figure (c) and (f) show the wall time of computing shadowing window over increasing flow length. The lines in Figure (b)-(c) and (e)-(f) indicate the median, and error regions indicate 25 th to 75 th percentile from 50 runs.

a cross-shaped distribution: a Gaussian mixture of the form

The two real-data experiments are described below.

Linear regression. We consider a Bayesian linear regression problem where the model takes the form

where y j is the response and x j ∈ R p is the feature vector for data point j . For this problem, we use the Oxford Parkinson's Disease Telemonitoring Dataset [52]. The original dataset is available at http://archive.ics.uci. edu/dataset/189/parkinsons+telemonitoring , composed of 16 biomedical voice measurements from 42 patients with early-stage Parkinson's disease. The goal is to use these voice measurements, as well as subject age and gender information, to predict the total UPDRS scores. We standardize all features and subsample the original dataset down to 500 data points. The posterior dimension of the linear regression inference problems is 21 .

Logistic regression. We then consider a Bayesian hierarchical logistic regression:

We use a bank marketing dataset [53] downsampled to 400 data points. The original dataset is available at https://archive.ics.uci.edu/ml/datasets/bank+marketing . The goal is to use client information to predict subscription to a term deposit. We include 8 features from dataset: client age, marital status, balance, housing loan status, duration of last contact, number of contacts during campaign, number of days since last contact, and number of contacts before the current campaign. For each of the binary variables (marital status and housing loan status), all unknown entries are removed. All included features are standardized. The resulting posterior dimension of the logistic regression problem is 9 .

(b) Cross

(c) Linear regression

(d) Logistic regression

MixFlowinversionerror

MixFlowinversionerror

MixFlowinversionerror

MixFlowinversionerror

100

100

100

Error

10~49.8

Fwd err.

Error

Bwderr.

Fwd err.

Error

Fwd err.

Error

Fwd err.

10

-49.9

10~30

10~30

10~30

10~50.0

0

200

400

600

0

200

400

600800

0

200

400

600

0

200

400

600

N

N

N

N

(b) Cross

(c) Linear regression

(d) Logistic regression

MixFlownumericalerror

MixFlownumericalerror

MixFlownumericalerror

MixFlownumericalerror

Error

30

20

Fwd

Error

５4３２

Fwd

Error

８６４２

Error

10

Fwd

Fwd

1

0

0

0

0

0

100200300400500

0

100200300400

500

0

1002003004

400

500

0

1002003004

400

500

#transformations

#transformations

#transformations

#transformations

D.2 MixFlow parameter settings

For all four examples, we use the uncorrected Hamiltonian MixFlow map F composed of a discretized Hamiltonian dynamic simulation (via leapfrog integrator) followed by a sinusoidal momentum pseudorefreshment as used in Xu et al. [5] (see [5, Section 5, Appendices E.1 and E.2] for details). However, we use a Gaussian momentum distribution, which is standard in Hamiltonian Monte Carlo methods [54] as opposed to the Laplace momentum used previously in MixFlows [5]. In terms of the settings of leapfrog integrators for each target, we used 200 leapfrog steps of size 0.02 and 60 leapfrog steps of size 0.005 for the banana and cross target distributions respectively, and used 40 leapfrog steps of size 0 . 0006 and 50 leapfrog steps of size 0 . 002 for the linear regression and logistic regression examples, respectively. The reference distribution q 0 for each target is chosen to be the mean-field Gaussian approximation as used in [5].

As for the NUTS used for assessing the log-density evaluation quality on two real data examples, we use the Julia package AdvancedHMC.jl [55] with all default settings. NUTS is initialized with the learned mean of the mean-field Gaussian approximation q 0 , and burns in the first 10 , 000 samples before collecting samples; the burn-in samples are used for adapting the hyperparameters of NUTS with target acceptance rate set to be 0 . 7 .

D.3 Additional MixFlow results

(b) Cross

(c) Lin. Reg.

(d) Log. Reg.

MixFlowlog-densityerror

MixFlowlog-densityerror

MixFlowlog-densityerror

MixFlowlog-densityerror

Error

0.3

0.4

0.2

0.1

E

Error

1.00

0.75

0.50

0.25

Error

1.0

0.5

1.5

Error

0.4

0.3

0.2

0.1

0.0

0.00

0.0

0.0

0100200300400500600

200

400

600

800

0100200300400500600

#transformations

#transformations

#transformations

#transformations

(b) Cross

(c) Linear regression

(d) Logistic regression

MixFlowsinglemaperr.

MixFlowsinglemaperr.

MixFlowsinglemaperr.

MixFlowsinglemaperr.

10

11

10~11

Error

10

13

Error

10~12.5

10

-12

rror

10~12

10~13

10~14

10~15

10~15.0

10~14

10~15

Fwd err.

Bwderr.

Fwd err.

Bwderr.

Fwderr.

Bwderr.

Fwd err.

Bwderr.

(b) Cross

(c) Linear regression

(d) Logistic regression

Ecomputationtime

Ecomputationtime

Ecomputationtime

Ecomputationtime

Wall time in sec

0.3

insec

sec

800

Wall time insec

0.2

Fwd

Wall time

0.6

0.4

Fwd

Walltime

=

600

400

Fwd

60

40

Fwd

0.1

0.2

200

20

0.0

0.0

0

0

0

200

400

600

800

#transformations

#transformations

#transformations

#transformations

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an analytical framework to quantify the effects of numerical errors on sampling, density and ELBO estimation in variational flows. The framework leverages the shadowing theory to derive error bounds based on the shadowing window sizes and the local Lipschitz constants of the dynamical operators. A procedure to verify whether the shadowing property holds in a given system is provided.\n\nSTRENGTHS:\n* [Significance] Numerical analysis is an important and under-studied aspect in flow-based models. This work takes a step forward in this direction and offers a plausible explanation for why fast accumulation of numerical errors do not necessarily lead to bad results.\n* [Quality] The derivations of the numerical bounds seem to make sense intuitively - although I did not check the math in detail.\n* [Clarity] The material is presented in a clear and logical manner. I find it easy to follow in general.\n* [Novelty] There exists prior work that is based on shadowing theory but this work extends the analysis to the computation of density and ELBO estimations.\n\nWEAKNESSES:\n* My primary concern about this work is mainly on the usefulness of the presented framework in practice. All results are predicated on the basis that the shadowing property is already established. The approach presented in section 5 looks cumbersome and does not seem to scale particularly well.\n* I do not find the numerical experiments to support the claims very well. They only demonstrate some qualitative traits - sampling, density and ELBO errors are much lower compared to trajectory errors, which coincides with the observation that the derived shadowing window sizes are relatively low. There is not much numerical evidence showing how tight the derived bounds are.\n\nQUESTIONS:\n* The analysis in section 4.3 is predicated on the basis that (ln 181) “when $F$ and $B$ are ergodic and measure-preserving for the target $\\pi$”, which is a pretty strong assumption. Do you have ways to establish this?\n* The horizontal axis in Figure 7 does not have the same range as those in Figures 4 to 6. Is this due to computational constraints, i.e. the proportionality to $N$? \n* It appears that $\\epsilon$ is still growing at the end while the sampling, density and ELBO errors all seem to have converged. Do you have any comments on the qualitative behaviors when extrapolating further? This would imply that the bounds eventually become very loose.\n\nLIMITATIONS:\nThere is not much explicit discussion on the limitations. The computational cost for establishing and calculating the shadow window size is an obvious one.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper investigated the impact of accumulated numerical error on the sampling, density evaluation, and evidence lower bound (ELBO) estimation in variational flows. It demonstrated that the results produced by flows are not destroyed by the serious numerical instability. To explain this phenomenon, it leveraged shadowing theory to theoretically bound the error of sampling, density evaluation, and ELBO estimation. It also developed a diagnostic procedure that can be used to validate results produced by numerically unstable flows in practice.

STRENGTHS:
This paper proposed an interesting question that why the accumulated numerical error does not influence the results of sampling, ELBO, etc. The technique is sound, and the analyses are solid. The derived theoretical results could support its claim.

WEAKNESSES:
The presentation could be improved, and the notations are hard to follow, see details below.

QUESTIONS:
1. What is the definition of "s_k" in theorem 4.1.
2. What is the range of mathcal{X}, is it belong to R or R^N or R\times[ 0,T]? 
3. I cannot follow the information in Figure 1(b)(d), e.g., what do you want to show through the dots in these figures?

LIMITATIONS:
The limitations are not fully discussed in this paper, e.g., the theory only provide explanations while it does not inspire new algorithm for generative learning. It will narrow the impact of the theory that practitioners may ignore.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Embracing the chaos: analysis and diagnosis of numerical instability in variational flows\" investigates the impact of numerical instability on the reliability of sampling, density evaluation, and evidence lower bound (ELBO) estimation in variational flows. The authors treat variational flows as dynamical systems and leverage shadowing theory to elucidate the abnormal behavior: common flows can exhibit a catastrophic accumulation of error, but surprisingly, results produced by flows are often accurate enough for applications despite the presence of serious numerical instability.\n\nSTRENGTHS:\nIn terms of originality, the paper presents a new approach to understanding the behavior of variational flows in the presence of numerical instability. By treating variational flows as dynamical systems and leveraging shadowing theory, the authors provide theoretical guarantees on the error of sampling, density evaluation, and ELBO estimation. The paper also presents a diagnostic procedure to empirically validate results produced by numerically unstable flows in practice.\n\nIn terms of quality, the paper is well-written. The authors provide a clear motivation for their work and present their results in a coherent manner. The paper also includes several theorems, which adds to the quality of the work.\n\nIn terms of clarity, the paper is easy to follow and understand. The authors provide clear explanations of their methods and results, and the paper includes several figures and examples that help to illustrate the concepts presented.\n\nIn terms of significance, the paper makes mainly two contributions to the field of variational inference. The paper provides a theoretical framework to understand the behavior of variational flows in the presence of numerical instability, which is an important problem in the field. The paper also presents a diagnostic procedure to validate results produced by numerically unstable flows in practice.\n\nWEAKNESSES:\nOne potential weakness of the paper \"Embracing the chaos: analysis and diagnosis of numerical instability in variational flows\" is that it focuses primarily on theoretical analysis and does not provide as much empirical validation of the proposed diagnostic procedure as would be desirable. While the authors do provide some empirical results to support their claims, it would be beneficial to see more extensive experiments that demonstrate the effectiveness of the proposed diagnostic procedure on more complex problems datasets like Cifar 10.\n\nAnother potential weakness of the paper is that the theoretical contribution with the concepts of dynamical systems and shadowing theory has an overlap with [1] , which is the main theoretical insight of the paper. Moreover, the theorems in ELBO esitimation and density estimation follows diretcly from the shadowing property.\n\nReference:\n1. Paul Tupper. The relation between approximation and shaowing in molecular dynamics.\n\nQUESTIONS:\n1. Does the shaowing property holds for continuous normalizing flow (empirically or theoretically) ?\n\n2. On theorem 5.1, the important constant $M$ depends on $x \\in \\mathcal{X}$, right?  If so, then we cannot derivative the shaowing property (which is a global property independent of $x$) from theorem 5.1.\n\nLIMITATIONS:\nThe paper could benefit from a more detailed discussion of the limitations of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the (non)impact of numerical stability concerns when implementing variational flows, specifically, the robustness of sampling, density evaluation or ELBO computation against the chaotic behaviour of numerical implementations of variational flows.\n\nThe results are the following: while small perturbations of the input imply large perturbations of the output of variational flows (Section 3), the quality of samples, density evaluations, and ELBOs are rarely affected. \nThe paper explains this phenomenon with \"shadowing\":\nMore specifically (Section 4), under the assumption of ($\\epsilon$, $\\delta$)-shadowing, the errors of sampling, density evaluation, and ELBO computation behave like $\\epsilon + d_\\text{TV}(q_0, \\xi_0)$, where $\\xi_0$ is the initial condition that shadows $q_0$. \nWhether or not a flow admits shadowing (and how $\\epsilon$ relates to $\\delta$) can be quantified by estimating (and bounding) the operator norm of the inverse of a specific operator that depends on the gradient(s) of the flow (Section 5). Numerical experiments show how the shadowing windows are minimal in typical applications of variational flows, even though numerical inaccuracy in implementing the flow is significant.\n\nSTRENGTHS:\nAnalysing the numerical implementation of variational flows through the lens of dynamical systems is exciting because it logically explains a seemingly unlogical phenomenon. On top of that, the paper is very well written: Although some components are rather technical (e.g. shadowing), the manuscript remains easy to follow.\n\nWEAKNESSES:\nIn general, I like the paper, but Section 5 (\"Computation of the shadowing window size\") leaves some open questions, especially in comparison to the finite-time shadowing Theorem in [33]. In my view, the following points should be addressed before publication:\n\n1. The algorithm for computing $\\lambda = \\|A^{-1}\\|$ is inaccurate: The implementation explained in Appendix B pretends that one can compute $\\nabla F_k$ exactly, but since evaluating $F_k$ suffers from approximation errors, $\\nabla F_k$ must as well. For context, compare Appendix B to Section 3.4 (especially, Equation 45) in [33] (reference in the paper). The inaccuracy is unlikely to be crucial but exists. It can (and should) be corrected. Or have I missed something?\n2. Why does the paper estimate the operator norm the way it does? In other words, why is a call to the default `eigmin` function superior to the algorithm discussed in Section 3.4 in [33]? The paper mentions that the complexity of `eigmin` is $O(d^3N^3)$, whereas the tailored algorithm on p. 187 in Section 3.4 in [33] appears to cost $O(N d^3)$. Please discuss the numerical cost of estimating the shadowing windows in the experiments (for example, by providing estimates of the run-/wall time).\n3. Could the authors please elaborate on how to estimate $M$? Why is estimating $M$ and $\\delta$ problem-specific, and estimating $\\lambda$ not (line 238)? If estimating $M$ is problem-specific, please discuss estimating $M$ for the problems in Section 6. (I have not found this information in Section 6 and Appendix D.)\n4. Sections 5 and 6 seem to focus on the computation of the shadowing window size $\\epsilon$. Does this mean all flows considered in this paper automatically have the shadowing property? Is this true for all variational flows (e.g., those discussed by Papamakarios et al.: neural spline, planar, radial, Hamiltonian, ...)? It would be helpful to discuss the existence of shadowing before computing the window size in the experiments (e.g. by estimating $2M\\lambda^2\\delta$).\n\nAs mentioned above, these points should be addressed before publication. But I expect all questions to be straightforward to answer, and I will increase my score once this is done.\n\nQUESTIONS:\n- This is not a question but more of a non-technical suggestion for the author(s). There have been discussions about potential ethical problems of the \"Boston Housing\" dataset: https://scikit-learn.org/1.0/modules/generated/sklearn.datasets.load_boston.html. If the corresponding experiment in the paper can be rerun with a different dataset, the longevity of this work may benefit.\n- Out of curiosity: Why are the sampling bounds propositions but the ELBO and density bounds theorems?\n- Please correct the complexity statements as commented in Appendix B.\n\nLIMITATIONS:\nAs mentioned under \"weaknesses\" above, one potential weakness of the algorithm derived from Theorem 5.1 could be computational feasibility. From the manuscript, it is unclear how much computational power it takes to implement this algorithm. A reader will benefit from learning about this limitation if the algorithm is prohibitively costly for some applications.\nOther than that, limitations seem to be discussed wherever relevant.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the impact of numerical instability on the reliability of sampling, density evaluation, and evidence lower bound (ELBO) estimation in variational flows. It demonstrates through empirical examples that numerical instability can lead to deviations in the flow map affecting sampling, density, and ELBO computations. However, the paper also finds that despite numerical instability, results from flows can still be accurate enough for practical applications. The paper treats variational flows as dynamical systems and uses shadowing theory to provide theoretical guarantees on the error of downstream tasks. It also develops a diagnostic procedure to validate results produced by numerically unstable flows in practice. The paper concludes by validating its theory and diagnostic procedure on MixFlow with both synthetic and real data examples.\n\nSTRENGTHS:\n* The paper addresses an important and practical problem in the context of variational flows, which are widely used in generative modeling and probabilistic inference.\n* The empirical examples and experimental validation provide concrete evidence for the theoretical claims made in the paper.\n* The use of shadowing theory from dynamical systems provides a rigorous framework to understand the behavior of numerically unstable flows.\n* The diagnostic procedure developed in the paper could be valuable for practitioners using variational flows, allowing them to assess the reliability of their results.\n\nWEAKNESSES:\n* Due to rather strong assumptions, the paper's claims may not be applicable in all scenarios. The experiments focus only on a few datasets as well as on a specific type of flow (MixFlow). Thus, they may not fully capture the behavior of other types of variational flows or other datasets, which limits the generalizability of the findings.\n\n* It should be explained in more detail what novelty the current submission offers compared to the work of [23] (except for an extension to the evaluation of densities and ELBO estimations).\n\nQUESTIONS:\n* It appears that the exponential scaling in the density evaluation is circumvented by looking at *log*-densities?\n\n* Can one motivate the assumptions (in particular, $\\xi_0=q_0$) for the ELBO result?\n\nLIMITATIONS:\nDetails on limitations are lacking.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-29"}