Whitening Convergence Rate of Coupling-based Normalizing Flows
Authors: Felix Draxler
Christoph Schnoerr
Ullrich Koethe
Venue: NeurIPS 2022 Accept (2022)
Keywords: normalizing flows
generative modeling
RealNVP
theory
maximum likelihood
kullback leibler divergence
invertible neural network
information theory
convergence
coupling block

Abstract: Coupling-based normalizing flows (e.g. RealNVP) are a popular family of normalizing flow architectures that work surprisingly well in practice. This calls for theoretical understanding. Existing work shows that such flows weakly converge to arbitrary data distributions. However, they make no statement about the stricter convergence criterion used in practice, the maximum likelihood loss. For the first time, we make a quantitative statement about this kind of convergence: We prove that all coupling-based normalizing flows perform whitening of the data distribution (i.e. diagonalize the covariance matrix) and derive corresponding convergence bounds that show a linear convergence rate in the depth of the flow. Numerical experiments demonstrate the implications of our theory and point at open questions.

Whitening Convergence Rate of
Coupling-based Normalizing Flows
Felix Draxler
Christoph Schnörr
Heidelberg University
Heidelberg University
felix. draxlerriwr. uni uni-heidelberg.cdo de
schnoerromath, sid-hiddelberg.de
Ullrich Köthe
Heidelberg University
ullrich.ketheeiiw. koethe@iwr. www.uniieiiiie mmii-hidelbber.coo
Abstract
Coupling-basee normalizing flows (e.g. RealNVP) are a popular family of nor- malizing flow architectures that work surprisingly well in practice. This calls for theoretical understanding. Existing work shows that such flows weakly converge arbitrary data distributions ( However, they make statement about the stricter convergence criterion used in practice, the maximum likelihood loss. For the first time, we make a quantitative statement about this kind of convergence: We prove that all coupling- based normalizing flows perform whitening of the data distribution (i.e. diagonalize the covariance matrix) and derive corresponding convergence bounds that show linear convergence rate in the depth of the flow. Numerical experiments demonstrate the implications of our theory and point open questions.
Introduction
Normalizing flows and have already demonstrated convincing performance in a wide variety of practical applications, to astrophysics
among the most promising approaches generative machine learning
mechanical engincering [0O, causality [II. computational biology 12 and medicine [13] As the name suggests, normalizing flows represent complex data distributions bijective transformations (also known as flows or push- forwards) of standard normal or other well-understood distributions.
ranging from image analysis [ 1S
In this paper, focus on theoretical underpinning of coupling- based normalizing flows, a par- ticularly effective class of normalizing flows in terms of invertible neural networks. All of the above applications actually implemented using coupling-based normalizing flows. Their central building blocks are coupling layers, which decompose the space into two subspaces called active and the passive dimensions, which makes the mapping computationally easy to invert. In order to vary the assignment of dimensions the active and passive subspaces, coupling layers are combined with preceding orthonormal transformation layers into coupling blocks. These blocks are arranged into deep networks such that the orthonormal nansformations are sampled uniformly at random from the orthogonal matrices and the coupling layers are trained with the maximum likelihood objective, see Equation ( Upon convergence of the training, the sequence of coupling blocks gradually transforms the probability density that generated the given training data, into standard normal distribution and vice versa.
passive subspace (see Section 3}. Only the active dimensions transformed conditioned
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

nors Gaussiannyy Plp X))
non-Gaussianity G
Son-Standerdress
Theorctical Upper Bound 615)
p(z)
Nia 22)
Contrbbutors
2.0
L C
1.5
non-Standardness = D(Vis EJIL F))
Nommlizing Flow Loss == D(pla) 13)
1.0
0.5
0.0
N(0.))
Coupling Blocks
Figure 1: (Left) The Maximum Likelihood Loss C (blue) can be split into the non-Gaussianity (orange) 25 and the non-Standardnes S (green) of the latent code = fo(): L G + (Proposition For the latter, we give explicit guarantees one more coupling block is added in Theorems and and show a global convergence rate Theorem g
(Right) Typical fit EMNIST digits by standard affine coupling flow for various depths. Our theory (Theorem D upper bounds the average for L + 1 coupling blocks given trained model with L coupling blocks (dotted green). We observe that our bound is predictive for how much end-to-end training reduces S.
Since the resulting normalizing flows deviate significantly from optimal transport flows [4] and the bulk of the mathematical literature is focusing on optimal transport, an analysis tailored to coupling architectures is lacking. In a landmark paper, M proved that sufficiently large affine coupling flows weakly converge to arbitrary data densities. The notion of weak convergence is critical here, as it does not imply convergence in maximum likelihood [15] Remark 3]. Maximum likelihood (or, equivalently, the Bullback-Leiblee (KL) divergence) is the loss that is actually used in practice. It can be used for gradient descent and it guarantees not only convergence
samples ("xr ~ q(x) + x ~ p(x)") but also in density estimates ("y(x: + p(x)""). It is strong in the sense that the square root of the KL divergence upper bounds (up factor 2) the total variation metric, and hence also the Wasserstein metric if the underlying space is bounded [16]. Moreover, convergence under the KI. divergence implies weak convergence which is fundamental for robust statistics
We take a first step towards showing that coupling blocks also converge in terms of maximum likeli- hood. To the best of our knowledge, our paper presents for the first time a quantitative convergence analysis of coupling based normalizing flows based on this strong notion of convergence. Specifically, make the following contributions towards this goal:
We utilize that the loss of normalizing flow can be decomposed into two parts (Figure) Tp: The divergence to the nearest Gaussian (non-Gaussianity) plus the divergence of that Gaussian to the standard normal (non-Standardness)..
The contribution of a single coupling layer on the non-Standardness is analyzed in terms of matrix operations (Schur complement and scaling).
Explicit bounds for the non-Standardness after single coupling block in expectation all orthonormal transformations are derived.
We use these results prove that sequence of coupling blocks whitens the data covariance and to derive linear convergence rates for this process.
Our results hold for all coupling architectures we are aware of (Appendix Cp including: NICE 4
RealNVP 5. and GLOW 3 Flow++ [8]: nonlinear-squared flow 19 linear, quadratic 20
cubic 1 and rational quadratic splines [22]: neural autoregressive flows 23 and unconstrained
monotonic neural networks (24 We confirm our theoretical findings experimentally and identify
directions for further improvement.

---

2
Related work
Analyzing which distributions coupling-based normalizing flows can approximate is an active area of research. A general statement shows that a coupling-basee normalizing flow which can approximate an arbitrary invertible function can learn any probability density weakly [.. This applies to affine coupling flows 4 [ G Flow++ [8]. neural autoregressive flows [26], and SOS polynomial flows 127 Affine coupling flows converge arbitrary densities in Wasserstein distance [TS]. Both universality results, however, require that the couplings become ill-conditioned (i.e. the learnt functions become increasingly discontinuous as the error decreases, whereas in practice one observes that functions remain smooth). Also, they consider only finite subspace of the data space. Even more importantly; the convergence criterion employed their proofs (weak convergence resp. convergence under Wasserstein metric) is critical: Those criteria do not imply convergence in the loss that is employed in practice [15 Remark 3]. the Kullback-Leblerr divergence (equivalent to maximum likelihood). An arbitrarily small distance in any of the above metrics can even result in an infinite KL divergence. In contrast to previous work on affine coupling flows, we work directly on the KL divergence. We decompose it in two contributions and show the flow's convergence for one of the parts.
Regarding when ill-conditioned flows need to arise to fit distribution, 28 showed that well- conditioned affine couplings can approximate log-concave padded distributions, again terms of Wasserstein distance. Lipschitz flows on the other hand cannot model arbitrary tail behavior, but this can be fixed by adapting the latent distribution [29].
which also does not imply convergence in KL divergence; zero-padded affine coupling flows converge weakly 23] and so do Neural ODEs 31 32). Normalizing Flows with diagonal Jacobian in each layer, known as Gaussianization (Flows), also converge weakly (33]
SOS polynomial flows converge in total variation to arbitrary probability densities 30
Closely related to our work, 48 linear affine coupling blocks can represent any invertible linear function Ax + b with (e) 0 [15] Theorem 2]. This also allows mapping any Gaussian distribution Mmm, Σ) to the standard normal (0, 7). We put this statement into context in terms of the KL divergence: The loss is exactly composed of the divergence to the nearest Gaussian and of that Gaussian to the standard normal. We then make strong statements about the convergence of the latter, concluding that for typical flows a smaller number of layers is required for accurate approximation than predicted by [15].
3 Coupling-based normalizing flows
Normalizing flows learn an invertible function fo(x) that maps samples x from some unknown distribution p(x) given by samples to latent variables 2 = fo(x) so that 2 follow a simple distribution, typically the standard normal. The function fa then yields an estimate q(x) for the true data distribution p(x) via the change of variables formula (e.g. (5)):
g(x) = [N)((<x) 0, 0,1)) det
(I)
where = = fo(x)) is the Jacobian of fo().. We can train normalizing flow via the maximum likelihood loss, which is equivalent to minimizing the Kullback-Leibler divergence between the distribution of the latent code q(2), given by fo(x) when p(z). and the standard normal:
L = (x<)(((=)]) 1)) = Eu-p(s) []] - log
| det det J1] + const.
The invertible architecture that makes up So has to (i) be computationally easy to invert, (ii) be able to represent complex transformations, and (iii) have a tractable Jacobian determinant det J| M Building such an architecture is an active area of research, see c.g. 2 for a review. In this work, we focus on the family of coupling-base architecture [4. It is a deep architecture that consists of several blocks, each containing a rotation, a coupling and ActNorm layer 6
normalizing flows, first presented in the form of the NICE
(x)) (face fept fzo)
The coupling Sept splits an incoming vector zo in two parts along the coordinate axis: The first part Po. which we call passive, is left unchanged. The second part ap, which we call active, is modified as
3

---

function of the passive dimensions:
fop1(z0) = Scpl
14
pa c(aa:
P1
(4)
Here, the coupling function o: IR x D/2 RD/22 has to be a function that is easy to invert when Po given, i.e. it is easy to compute ag e (a1; (o) given po- This makes the coupling easy to invert: Call 11 = (p1: a1) the output of the layer, then P1. Use this to invert a1 c(ag: Po) For example, RealNVP [5] proposes a simple affine transformation for e: aj c(ao: po) ag @ s(P)) +
t(po) (O means element-wiss multiplication). s(po) E IR422 and t(po) P/2 are computed by geed-forward neural network. The coupling functions of other architectures our theory applies to are listed in Appendix [
An Activation Normalization (ActNorm) layer 6 helps stabilize training and is implemented in practice like in the popular INN framework FrEIA
35 rescales and shifts each dimension:
(aa(c) = x +
(5)
given parameters r E R and u E R.P. We include it as it simplifies our mathematical arguments.
If we were to concatenate several coupling layers, the entire network would never change the passive dimensions apart from the element-wise affine transformation in the ActNorm layer. Here, the rotation layers (x) Qz come into play 6 They multiply orthogonal matrix Q to the data, changing which subspaces are passive respectively active. This matrix typically fixed at random at initialization and then left unchanged during training.
4 Coupling layers as whitening transformation
The central mathematical question we answer in this work the following: How can deep coupling- based normalizing flow whiten the data? As the latent distribution is standard normal, whitening is a necessary condition for the flow to converge. This is direct property of the loss:
Proposition (Pythagorean Identity, Proof in Appendix B.I. Given data with distribution p(z) with mean m and covariance E. Then, the Kullback-Leible divergence to a standard normal distribution decomposes as follows:
Dxu(o())IIW(I11) (xL( + (xa(N(m (0),,
non- Goussianity G(p)
non- non-Sunndarbwess S(p)
and the con-Standardness again decomposes:
S(p) (D, E)] (m, Di)) + DRL(N(m, Diag (ON()
Correlation C(p)
Dingonal non Sunndordness
This splits the transport from the data distribution the latent standard normal into three parts: (i) From the data to the nearest Gaussian distribution (m,, E), measured by G. This measure of non-Gaussianity often denoted as the negentropy, e.g. [36] (ii) From that nearest Gaussian the corresponding uncorrelated Gaussian N (mm, Diag(E), measured by C. (iii) From the uncorrelated Gaussian standard normal.
We do not make explicit use of the fact that the non-Standardness can again be decomposed, but we show it nevertheless relate our result to the literature: The Pythagorean identity DxI. (x<)) (W Diag(E))) G(p) + C(p) has been shown before by (25] Section 2.3]. Both their and our result are specific applications of the general 337 Theorem 3.8] from information geometry. Our proof is given in Appendix B.I
Proposition M is visualized in Figure I In experiment, we fit set of Glow [ coupling flows of increasing depths to the EMNIST digit dataset 38) using maximum likelihood loss and measure the capability of each flow in decreasing & and S (Details in Appendix A.Ip. The form of the non- Standardness S is given by the well-known KL divergence between the involved normal distributions, Equation 30 Appendix B.1 It is invariant under rotations and only depends on the first two
moments m, 3:
1 (11m] + tr D log det Σ)) S(Qm, QEQ").
S(m, Σ) = S(p) =
(8)

---

The -on-Standardness S will be our measure on how far the covariance and mean have approached the standard normal in the latent space. We give explicit loss guarantees for S for a single coupling block in Theorems I and and imply a linear convergence rate for deep network in Theorem R
Deep Normalizing Flows typically trained end-to-end, i.e. the entire stack of blocks is trained jointly. In this work, our amsatz is to consider the effect of a single coupling block on the non- Standardness S. Then, we combine the effect of many isolated blocks, disregarding potential further improvements to S due to joint, cooperative learning of all blocks. This simplifies the theoretical analysis of the network, but it is not a restriction on the model: Any function that is achieved in blick-wise training could also be the solution of end-to-end training.
We aim to strongly reduce S while leaving room for a complementary theory explaining how non- Gaussianity 9 is reduced in practice. Note that affine-linear functions Ar + b can never change G. because they jointly transform the distribution p(x)) hand and correspondingly the closest Gaussian to it (see Lemma | in Appendix B..2). Thus, we restrict our coupling layers to be affine-linear functions, we are able reduce S without increasing G in turn. This motivates considering affine- linear couplings of the following form, spelled out together with ActNorm as given by Equation ( The results in this work apply to all coupling architectures, as they all can represent this coupling, see Appendix C
(Sac o (Qx)) = r (+ ())
P1 (1
PD 00
+
For future work considering G, we propose to lift the restriction to affine-linear layers while making sure that S behaves described in what follows. As the convergence of & however will strongly depend on the coupling architecture and data p(z) at hand, this is beyond the scope of this work.
Our first result shows which mean my and covariance D1 a single affine-linear coupling as in Equation D yields to minimize S(m1, Ei) given data with mean m and covariance M, rotated by Q: Proposition 2 (Proof in Appendix B.2 Given D-dimensional data with mean and covariance E and a rotation matrix Q. Split the covariance of the rotated data into four blocks, corresponding to the passive and active dimensions of the coupling layer:
Eo.pp Eo,ap
Eo.pa
QUQ = =co
(10)
L0,aa
Then, the moments m1, En that can be reached by a coupling as in Equation are:
/M (E
mi O, Σ
ME - Eo, Eo,pal)
(11)
This minimizes S as given in Equation and G does not increase.
The function M takes matrix and rescales the diagonal to as follows. It is a well-known operation in numerics called Diagonal scaling or Jacobi preconditioning so that M()a = 1:
M(A), VAnAj AG (Diag( 1/22 -1/2 Diag(A)
(12)
Proposition 2 shows how the covariance can be brought closer to the identity.
The new covariance has passive and active dimensions uncorrelated. In the active subspace, the covariance is the Schur complement X0, Σo. "o.1 "0. -o.pa- This coincides with the covariance of the Gaussian (0, )) as it conditioned on any passive value p. Afterwards, the diagonal is rescaled one, matching the standard deviations of all dimensions with the desired latent code. The proof is based on a more general result how a single layer maximally reduces the Maximum Likelihood Loss for arbitrary data [14, which we apply the non-Standardsess S (see Appendix
B.2
Figure 2 shows an experiment in which single affine-linear layer was trained to bring the covariance of EMNIST digits (38) as close to 7 as possible (Details in Appendix A.2 The experimental result coincides with the prediction by Proposition 2
Due to the finite batch-size, a small difference between
theory and experiment remains.
5 Explicit convergence rate
In Section we showed how single coupling layer acts on the first two moments of given data distribution to whiten it. We now explicitly demonstrate how much progress this means in terms
5

---

Input covariance
Experiment: Output covariance
Theary: Output covariance
Difference
100 20o
300 400 500 700
100 200 300
600 700
100 200 400 s00 700
100
200 300 400 600 700
100
100
100
100
2.5
200
200
200
400
400
400
400
15
500
500
500
500
0.5
Figure 2: How single coupling layer can whiten the covariance at the example of the EMNIST digits covariance matrix (first panel). The covariance after single layer trained experimentally to minimize Son-Standardness S(m1, Σ1) (second panel), which matches closely the prediction of Proposition 2 (third panel). The difference between theory and experiment vanishes (last panel).
of the non-Standardness S(m,, E1) averaged over rotations Q (Theorems and and show the
consequences for multiple blocks (Theorem
5.1 Single coupling block guarantees
Proposition allows the computation of the minimum non-Standardness after a single coupling block given its rotation Q, by evaluating S(m1, Σ1). In fact, if we were to choose Q such that the data is rotated so that principal components lic on the axes (i.c. obtain Q using PCA), a single coupling block suffices to reduce the covariance to the identity: Mn QVQ" would be a diagonal matrix and E = I. This is not the case in practice, where this optimal orientation has zero probability: Q is chosen uniformly random before training from all orthogonal matrices. One could argue that one should whiten the data before passing it to the flow, reducing S to zero from the start. However, any change in the architecture could possibly alter the performance of the network with regard to reducing the -on-Gaausia G. Also, our work shows that coupling-base already well-equipped to bring the non-Standardness zero without such modifications. To properly describe the achievable von-Standardnnes S, we formulate all guarantees as expectations over the rotation Q, corresponding to the loss averaged
normalizing flows are
training runs.
We make two mild assumptions on our data that are part of usual data- prterprepro,,s,nng when the mean subtracted from the data and all data points are divided by the scalar E/D (not to be confused with diagonal preconditioning, which acts dimension-winn
Assumption 1. The data p(z) is centered: Em-p(z) [x] = 0. Assumption 2. The covariance normalized: tr Σ D.
The assumptions simplify the non-Standardness in Equation which now only depends on the determinant of 2:
S(E)
log det Σ = - log det Vg = -(Eo)
(13)
for arbitrary rotation Q. We aim to compute the average non-Standardness after single block (QEE(()) [S(E, (Q)). For any Q. S(E,) is again given by the determinant the covariance (Q)
as Assumptions D and 2 remain fulfilled: By Proposition
2 m = 0 and the diagonal preconditioning
M ensures that the trace of ES is D. We write det(2,, via Ma and Mp. the diagonal matrices that
make up the diagonal preconditioning in Equation 122 and use the Schur determinantal formula
for the determinant of block matrices: det det(Eo.pp) det (Jo Eo,ap -o E, pa = det(Eo) =
(E) ]. We thus get dct(E,) = det( Eu.p (ct(MM
Eu,ap 2 Eo Ma)
det M3) det (M22) det(E). Inserting this into Equation
13 we find:
S(Σ,) = (log det Σ + log det M² + +log det M2) < S(Eo) = S(E).
(14)
The inequality S(Σ,) < S(E.) holds because Σn Eg is admissible solution of the coupling layer optimization, but Σ, as given by Proposition is a minimizer of S(Σ₁).
We average this quantity over training runs, i.e. over rotations Q:
EQ)p(o) (S(11) = det Σ EQ~ [log det M2] + EQ) [log det Mt)).
(15)

---

The main difficulty lies in the computation of Eq-n(Q)llog det Ma). Here, we contribute the two strong statements Theorems
T and
2
below.
5.1.1 Precise guarantee
The first result relies projected orbital measures developed by 40) This theory describes the eigenvalues of submatrices of matrices in a random basis. We require such a result for integrating over p(Q) in Eq))(() [log det M2)- In contrast typical choices of p(Q), the theory to this date only covers data rotated by unitary matrices To comply with 40.. we make two more assumptions:
Assumption 3.
The distribution of rotations is the Haar measure over unitary matrices U (D).
Assumption 4. The eigenvalues of the covariance matrix E are distinct: X + > for i + j.
One could think that the step from orthogonal to unitary rotations takes us far away from the scenario want consider. We will later observe empirically that the difference between averaging over unitary and orthogonal matrices is negligible. Technically, the covariance matrix remains positive definite, so the -tandatanassssss S is always real (see Appendix B.3.4). We will write EQ-C(D)]-] to denote expectations over unitary matrices.
q is typically satisfied when working with real data that are in 'general position'. We now ready to compute the average training performance of a single coupling block:
Assumption
Theorem 1 (Proof in Appendix B.3 Given D-dimensional data with covariance with eigenvalues
\1 A AD Assume that Assumptions to hold. Then, after single coupling block, the expected non-Standarrness is bounded from above:
D
(berr)//8((:(00)))88==1)1 log (-1)
Σ
X
1-
: (z))) (16)
1=1
Here, A41= == (i dob and R. eg given by:
N
R(a; N
a b; and ex (x((b:)
E
bi ...... ...b.-
(17)
0c4 CiRS
Inequality (T)p sharply bounds the expected non-Standenn aon-Stadarddeess that can be achieved by single block. The only approximation made is an inequality which comes close to equality as the dimension D increases due to the concentration of the corresponding probability distribution.
Figure 3 shows an experiment confirming Theorem T (Details in Appendix A.3 We start with
covariance matrices using parametrized eigenvalue spectra. On each, we first apply single coupling block with random Q and train the coupling that maximally reduces S (Proposition 2). Then we iteratively append 32 additional blocks in the same manner, building flow of that depth. We average the resulting empirical ratio ((E))/S(E) over several orthogonal orientations of the rotation layer for each input covariance matrix. Then, we compare this to (i) experimentally averaging over umitary rotations and (ii) to the prediction by Theorem | and confirm that is a valid and close upper bound. Details for replication and more examples can be found in Appendix A.3
The proof explicitly integrates [[M2] using 40 (see Appendix B.3 Numerically evaluating
Equation (6) can be hard even for small D as the summands scale as (exx(D))) but the overall sum scales as ODD. High values cancel due It alternating in sign, and one requires arbitrary- precision floating point software to evaluate Equation (66).
5.1.2 Interpretable guarantee
The guarantee in Theorem yyields useful predictions, but does not lend itself to further analysis: How does the bound behave over several coupling blocks? What is the behavior for varying dimension D? Also, Assumption g restricts formal reasoning as we are interested in averaging over orthogonal and not unitary rotations. Our second single-block guarantec depends only on simple metrics of the covariance. Moreover, we drop Assumptions and averaging over orthogonal, not unitary, Q:
'The only result known would yield predictions for D
41 whereas interested in large D.
7

---

Reletive Sr--Staddrdneess Block
Relelive narr-Saanderdnesss after Block
0.8
0.3
0.6
Experiment
Experiment
Experiment (Unitary)
Experiment (Unitary)
0.2
Unitary Bound
0.2
'Unitary' Bound
Var-max' Bound
"Varmax' Bound
Loss-only' Bound
LLossonly Bound
10
non-Standardness before B ock
non-Standardness before Block
Figure 3: Comparison between predicted non-Standardnes: and experiment for 48- 18-dimensional parametrized eigenvalue spectra (insets), varied over a parameter which controls the spread of the spectrum and thus changes S. The experimental average over orthogonal rotations matrices (blue, shaded by Interquartile Range IQR) is closely matched by the experimental average over unitary matrices (dotted blue). The prediction by Theorem D is a close upper bound that closely matches the experimental behavior (orange). The predictions by Theorem 2 are less precise, but converge to the same value as the precise bound for covariances close to the identitiy: 'Var-max'' Equation (8a) (green) and 'Loss-only' is Equation (186 (red). More details and examples in Appendix
Theorem 2 (Proof in Appendix B.4. Given D D-dimensional data fulfilling Assumptions and with covariance Σ # I with eigenvalues .I....Ap. Then, after single coupling block, the expected loss can be bounded from above:
D
D2
Var
(oeeo))))(E{(()) ( SS(E) log
(18a)
2(D - 1) + +2) Amax
D
D2
v gD
$ <S(E) log
(D - (D-1)(D + 2) 1 + vI gg (1-9) <S(Z).
(18b)
Here, g is the geometric mean of the eigenvalues: g = -11 X =xp((=2(())/))) < which is a bijection of S(E).
These two new bounds on the average achievable non-Standardness after a single block are also depicted in Figure 3 They make useful predictions, but are less precise than Theorem [ The second bound will be especially useful what follows because it only depends on the non-Standardness before the block S(E).
The full proof given in Appendix B.4 It relies the integration of monomials of entries of random orthogonal matrices as described by 42 and the arithmetic mean-geomette mean incquality by 43
The first bound suggests important property of the non-Standardness convergence of a coupling- based normalizing flow terms of dimension: The performance only marginally depends on the dimension. To this, divide Equation (18a) by D obtain a statement about the San-Standarddess per dimension S/D. Then take several data sets with different dimension but same spectrum characteristies (i.e. same geometric mean, variance and maximum of covariance eigenvalues). The guarantee is then approximately constant in (it varies slightly with D2 D²/(D2 + - 2), which is always close
5.2 Deep network guarantee
The previous Section 5.1 was concerned with determining how much a single coupling block can
typically contribute towards reducing the to zero. Now, we extend this result to compute expected non-Standardness after deep coupling-based normalizing flow an explicit function of the number of blocks. We again treat the rotation layer of each block as random variable, as randomly determined before training.

---

Experiment
10'
with Boraag Bario
Average neh Strdddrrrrressss
Lois-ony bound
10-1
0.5
20-3
0.4
10-1
20 Number Coupling Blocks
Number Coupling
Figure 4: Deep network convergence of covariance on toy dataset. (Left) Each line shows the experimental convergence of S via the repeated application of Proposition averaged over 32 runs with different rotations Q. (Right) The empirical convergence rate (blue), I.e the ratio of S before and after block, is correctly bounded from above by our predictions in Theorem I (orange), and the bounds in Theorem 2 Equation (Ta)) (green) and Equation (T&b) (red). The solid lines show the ratio (bounds) averaged over the toy dataset and rotations, the shade is the IQR. The experiment
suggests that convergence rate like Theorem can also be derived for the remaining bounds.
We find that the convergence rate of the covariance to the identity is (at least) linear:
Theorem 3 (Proof in Appendix B.5 Given D-dimensional data fulfilling Assumptions) and covariance E. Then, after L coupling blocks, the expected loss is smaller than:
QUEOLD) (S((E)) ((E))
(19)
where the convergence rate depends on the non-Standardness before training:
1 / 1 7(8) = 1 18/D log 1 (D (D- 1 /1 -1- V1-9(S)D s(8)) <1
D2
(20)
The non- non-Standann Standardness decreases at least exponentially fast in the number of blocks. The convergence rate that holds for deep network is computed using the non-Stan con-Standardness of the input data S(E). This rate comes from Equation (88).. The proof uses that 7(S) improves from block to block as is the geometric mean of S decreases (see Appendix B.5 Again, (SS) exp(-2S/D)
eigenvalues of Σ, which increases from block to block.
Figure 4 shows the convergence of the non-Standardness to zero in experiment. We build toy dataset of various covariances where we aim to capture a plethora possible cases (see Appendix A.44). We apply single coupling block with random Q and the coupling that maximally reduces S via Proposition
2 We iteratively add such blocks 32 times, building a flow of that depth. The resulting convergence of S function of depth is averaged over 32 runs with different rotations. The measured curve confirms Theorem g We find that the rate y in Equation (20) is correct, but several experiments show even faster convergence in practice. Indeed, the experiments suggest that dividing all upper bounds for (E((())) in Theorems and by S(E) also bounds the non-Standardness ratio for subsequent blocks. Formally, we conjecture that TESS(E)) ([L)()))/SSL
S (B/SEE) where B is the
rhs. of Equations (T6) and ([8a) (Theorem B shows exactly this for Equation (86 ). We leave a proof
falsification of this conjecture open future work.
The experiment also suggests that all bounds agree after few blocks, leaving small gap to experiment. We can explicitly compute this limit value of (S)) by taking - 0:
(S)
S-0
D(D+2)-4 2D-111442)
[1/2,5/9].
(21)
The two experimental observations togetherr with this limit value suggest the heuristic that a single block typically reduces the con-Standardness S by factor of approximately
additional coupling 50% if previous blocks are left unchanged, and possibly faster if cooperations between blocks are taken into account.

---

Conclusion
To the best of our knowledge, this is the first work on coupling- based normalizing flows that provides quantitative convergence analysis in terms of the KL divergence. Specifically, a minimal convergence rate is established at which flows whiten the covariance the input data under this strong measure of discrepancy of probability distributions. Splitting the loss into the non-Gaussianity (negentropy) G and the non-Standardness S, we show that this whitening is a necessary condition for the flow to converge and give explicit guarantees. Our derivations suggest the rule of thumb that S can typically be reduced by about 50% per coupling block.
Our central idea was to separate out the contribution single isolated block can make to reduce the loss, arguing that end-to-end training can only outperform the concatenation of isolated blocks.
Having separated the tasks a normalizing flow has to solve, and having explained how the non- Standardness S be reduced to zero, we hope that explaining also the entire convergence of L = G + S with respect to the KL divergence is within reach. In particular, our theory did not yet explore how the non-linear part of each coupling block reduces the non- non-Gaussianity G.
Acknowledgments and Disclosure of Funding
This work is supported by Deutsche Forschungsgemeinnchatt (DFG, German Research Foundation) under Germany's Excellence Strategy EXC-2181/1 390900948 (the Heidelberg STRUCTURES Cluster of Excellence). It is also supported by the Vector Stiftung in the project TRINN (P2019-0092).
We thank our colleagues (in alphabetical order) Marcel Meyer, Jens Müller, Robert Schmier and Peter Sorrenson for their help and fruitful discussions.
References
[1] Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling- based Invertible Neural Networks Are Universal Diffeomorphism Approx- imators. In Advances in Neural Information Processing Systems, 2020.
[2] Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker. Normalizing Flows: An Introduc- tion and Review of Current Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1 11:3344-333774 2021.
[3] George Papamakarios, Eric Nalisnick. Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing Flows for Probabilistic Modeling and Inference. Journal of Machine Learning Research, 57(22): 2021.
[4] Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components Estimation. International Conference on Learning Representations, Workshop Truck, 2015.
[5] Laurent Dinh, Jascha Sohk-Dickstein, and Samy Bengio. Density estimation using Real NVP.
In International Conference on Learning Representutions, 2017.
[6] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, 2018
[7] Radek Mackowiak, Lynton Ardizzone, Ullrich Kothe, and Carsten Rother. Generative classifiers as a basis for trustworthy image classification. In IEEE CVF Computer Vision and Puttern Recognition Conference, 2021.
[8] Lynton Ardizzone, Radek Mackowiak, Carsten Rother, and Ullrich Köthe. Training Normalizing Flows with the Information Bottleneck for Competitive Generative Classification. In Advances Neural Information Processing Systems, 2020.
[9] Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Analyzing Inverse Problems with Invertible Neural Networks. International Conference on Learning Representations, 2018.
10

---

[10] Pablo Noever-Castelos Lynton Ardizzone, and Claudio Balzani. Model updating of wind turbine blade cross sections with invertible neural networks. Wind Energy, 259))957--599 2022.
[11] Jens Müller, Robert Schmier, Lynton Ardizzone, Carsten Rother, and Ullrich Köthe. Learning Robust Models Using The Principle of Independent Causal Mechanisms. In German Conference on Pattern Recognition, 2021.
[12] Frank Noé, Simon Olsson, Jonas Köhler, and Hao Wu. Boltzmann generators: Sampling cquilibrium states of many-body systems with deep learning. Science, 365(6457):caaw) 1147, 2019.
[13] Tim Adler, Lynton Ardizzone, Anant Vemuri, Leonardo Ayala, Janek Gröhl, Thomas Kirch- ner, Sehastian Wirkert, Jakob Kruse, Carsten Rother, Ullrich Köthe, and Lena Maier-Hein. Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks. International Journal of Computer Assisted Radiology and Surgery, 14(6):977-1007, 2019.
[14] Felix Draxler, Jonathan Schwarz, Christoph Schnörr, and Ullrich Köthe. Characterizing the Role of a Single Coupling Layer in Affine Normalizing Flows. In German Conference on Puttern Recognition, 2020.
[15] Frederic Koehler, Viraj Mehta, and Andrej Risteski. Representational aspects of depth and conditioning in normalizing flows. In International Conference on Machine Learning, 2021.
[16] Alison L. Gibbs and Francis Edward Su. On Choosing and Bounding Probability Metrics. International Statistical Review Revue Internationale de Statistique, 70(3):419 435, 2002.
[17] P. J. Huber and E. M. Ronchetti. Robust Statistics. John Wiley & Sons, Inc., second edition, 2009.
[18] Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design. In International Conference on Machine Learning, 2019.
[19] Zachary Ziegler and Alexander Rush. Latent Normalizing Flows for Discrete Sequences. In International Conference Machine Learning. 2019.
[20] Thomas Müller, Brian Mewilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. Neural Importance Sampling. ACM Transactions Graphics, 38(5):1-19, 2019.
[21] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Cubic-Spline Flows. In International Conference on Machine Learning, Workshop Track, 2019.
[22] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural Spline Flows. In Advances in Neural Information Processing Systems, 2019.
[23] Chin-Wei Huang, Laurent Dinh, and Aaron Courville. Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models. In International Conference on Learning Representations, Workshop Track, 2020.
[24] Antoine Wehenkel and Gilles Louppe. Unconstrained Monotonic Neural Networks. In Advances in Neural Information Processing Systems, 2019.
[25] Maan-Fraççois Cardoso. Dependence, Correlation and Gaussianity Independent Component Analysis. Journal of Machine Learning Research, 4:1 1771-772,0, 2003.
[26] Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural Autoregres- sive Flows. In International Conference on Machine Learning, 2018.
[27] Priyank Jaini, Kira A. Selby, and Yaoliang Yu. Sum-of-Squares Polynomial Flow. In Interna- tional Conference on Machine Learning, 2019.

---

[28] Holden Lee, Chirag Pabbaraju, Anish Sevekari, and Andrej Risteski. Universal Approximation for Log-coneave Distributions using Well-conditioned Normalizing Flows. In International Conference Machine Learning, Workshop Track, 2021.
[29] Priyank Jaini, Ivan Kobyzev, Yaoliang Yu, and Marcus Brubaker. Tails of Lipschitz Triangular Flows. In International Conference on Machine Learning, 2020.
[30] Isao Ishikawa, Takeshi Teshima, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Universal approximation property of invertible neural networks. arXiv preprint arXiv:2204007415 2022.
[31] Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation Capabilities Neural ODEs and Invertible Residual Networks. In International Conference Machine Learning, 2020.
[32] Takeshi Teshima, Koichi Tojo, Masahiro Ikeda, Isao Ishikawa, and Kenta Oono. Universal Approximation Property of Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems, Workshop Track, 2020.
[33] Scott Chen and Ramesh Gopinath. Gaussianization. In Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems, volume 13, MIT Press, 2000.
[34] Chenlin Meng. Yang Song, Jiaming Song, and Stefano Ermon. Gaussianization Flows. In AISTATS, 2020.
[35] Lynton Ardizzone, Till Bungert, Felix Draxler, Ullrich Köthe, Jakob Kruse, Robert Schmier, and Peter Sorrenson. Framework for Easily Invertible Architectures (FrEIA). http://ggttbbbcom//lLL//DifEEE 2018.
[36] Pierre Comon. Independent component analysis, new concept? Signal Processing, 36(3):287-
3 314, April 1994.
[37] Shun-ichi Amari and Hiroshi Nagaoka. Methods of Information Geometry. American Mathe- matical Society, 2007.
[38] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: An extension of MNIST to handwritten letters. arXiv preprint arXiv:1702.05373. 2017.
[39] Roger A Horn and Charles R Johnson, Matrix Analysis. Cambridge University Press, 2012.
[40] Grigori Olshanski. Projections of orbital measures, Gelfand-Tsetlin polytopes, and splines. Journal of Lie Theory, 23(4):1011-1022 2013.
[41] Jacques Faraut. Rayleigh theorem, projection of orbital measures and spline functions. Advances in Pure and Applied Mathematics, 6(4):261-283, 2015.
[42] Gorin. Integrals of monomials over the orthogonal group. Journal of Mathematical Physics, 43(6):3342-3351, 2002.
[43] D. I. Cartwright and M. Field. refinement of the arithmetic mean- geommtrr mean inequality. Proceedings of the American Mathematical Society, 71(1):36-38, 1978.
[44] P.R. Krishnaiah. Some recent developments complex multivariate distributions. Journal of Multivariate Analysis, 6(1):1-30, 1976.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section 6
12

---

(c) Did you discuss any potential negative societal impacts of your work? [Yes] Generative modeling, which this paper aims improve, can used in harmful ways generate Deepfakes for disinformation.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of theoretical results? [Yes] Full proofs are in Appendix B
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] See Appendix A
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix A
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "lImxYnZcwM8", "reviewer": "Reviewer_XD8h", "summary": "A **normalizing flow** (NF) is a generative model that transforms a complex distribution using a chain of invertible transformations into a simple distribution. Furthermore, a NF models the exact likelihood using the change of variables formula and can be trained via the maximum likelihood principle.\n\nMany different transformations have been considered in the past. One of those is the so-called affine coupling layer. **This paper is about** a theoretical analysis of the whitening convergence rate of affine coupling flows. To do so, the authors show that the KL divergence used for training can be split into two parts handling \"non-Gaussianity\" and \"non-Standardness\". The authors then provide bounds on the \"non-Standardness\" and argue that each affine coupling layer (followed by a rotation (for reasons)) typically reduces the \"non-Standardness\" by approx. 50%.", "questions": "**Q1:** To me, the use of $r$ and $u$ in Equation (4) appear to be non-standard within the literature and I assume they are learned as parameters during training. If those are not used (i.e. the standard affine coupling is being used) I assume the bounds do no longer hold. How large is the influence of having those? Can it be at least empirically shown that the same convergence is achieved? \n\n**Q2:** Would you agree that instead of learning 2D additional parameters per layer ($r,d$), one could learn the mean and a diagonal (or even the full covariance) of the Gaussian base distribution. Such a distribution is still easy to handle but would change the setting: the flow would just have to handle the (more challenging) part of \"non-Gaussianity\". Moreover, the analysis presented only considers bounds on the \"non-Standardness\" part of the loss, however, the \"non-Gaussianity\" part seems to be the real challenge. If \"non-Standardness\" decreases, \"non-Gaussianity\" could still remain a problem, no?\n\n**Q3:** What are the assumptions on the functions $s(\\cdot)$ and $t(\\cdot)$? Line 105 states they can be arbitrary but I would assume that their expressiveness largely influences the amount of layers/blocks needed.\n", "limitations": "The authors flagged that they discussed the negative societal impact of their work in line 410, however, I do not see such a discussion except the comment in lines 410-412.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\nThe paper is well written and mostly easy to follow. The authors begin by arguing that maximum likelihood estimation is equivalent to minimizing the KL divergence and decompose this KL divergence into two parts handling \"non-Gaussianity\" and \"non-Standardness\". Then, they derive bounds for the \"non-Standardness\" part for a single affine coupling layer as well as for a concatenation of such. Although I did not check every detail of the proofs, the derivation appears to be sound.\n\n**Weaknesses:**\n- The use of $r$ and $u$ in Equation (4) appear to be non-standard within the literature. It is unclear if the results still hold if this *architecture change* is not present. (See also Question 1 below.)\n- If the loss can be seen as \"non-Gaussianity\" plus \"non-Standardness\", the main challenge seems to be the \"non-Gaussianity\", not \"non-Standardness\". However, the paper is only concerned with bounding \"non-Standardness\". The more important part seems to be ignored. Hence, the usefulness of the bounds is questionable. (See also Question 2 below.)\n\n**Minor comments/questions:**\n- The pdf should be built without boxes around links\n- Why is the $x$-axis scaled non-linearly in Figure 1?\n- Please rephrase lines 89-91.\n- The actual (negative log-)likelihood objective could be stated in Section 3 along with a case of a flow that consists of multiple transformations/layers for completeness.\n- Please avoid starting a sentence with variables/symbols (e.g., lines 91 & 106).\n- lines 96, 127, etc.: (1) might be mistaken as a reference to an equation, hence use (i).\n- lines 99, 115: RealNVP does not mention any rotation. A rotation layer is used in Glow [6], however, the rotation is learned and not fixed. Nevertheless, I think Glow should be mentioned there.\n- line 103: The passive part is not transformed!\n- Equation (4): Please introduce $p_0$ and $a_0$, e.g., within the sentence above.\n- Lines 116-118 should be mentioned earlier.\n- Cite or provide a link for FrEIA.\n- line 181: The comma should be after Thm. 2.\n- line 291: \"We iterate adding such\"\n- line 302: additive\n- [4,5] are ICLR papers.\n- Why are arXiv identifiers provided, i.e., in [3,8,...]?\n\n**After the author feedback:**\nI have read the author feedback and the other reviews. Especially reviewer X9Dq shared my concern that only the non-Standardness part of the loss is considered whereas the non-Gaussianity part seems to be the real challenge. However, the authors provided clarifications, actually to both points I raised in **Weaknesses** above, which is why is raise my score to 7.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YQTbYn6xCO7", "reviewer": "Reviewer_r6j2", "summary": "This paper provides new theoretical insights on the affine coupling in normalizing flows.\nThe contributions of the paper are clearly presented within 4 points at the end of the introduction section.", "questions": "We find this work very interesting. We do not have any question.", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written. \n\nThe contributions are theoretical contributions that allow to better understand the underlying mechanism of the affine coupling in normalizing flows.\n\nIt is worth noting that many theoretical results in normalizing flows were derived for the optimal transport. This paper investigates affine coupling by providing novel insights. This includes providing a novel point of view of the affine coupling layers as whitening transformation, and convergence analysis with explicit convergence rate and deep network guarantee.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SkgyUAJdz5L", "reviewer": "Reviewer_X9Dq", "summary": "In the article, the authors analyze the convergence of affine coupling flows. So far, it has only been shown that this type of normalizing flows converges weakly to an arbitrary target distribution. However, from this result it cannot be concluded that it converges in maximum likelihood.\n\nThe authors obtain a stronger convergence guarantee by decomposing the KL-divergence into a part measuring the non-Gaussianity, i.e. the KL between the flow density and a Gaussian with mean and variance as the data, and the non-Standardness, i.e. the KL-divergence of the just mentioned Gaussian distribution and a Standard Normal distribution. They prove bounds of the non-Standardness for single and multiple affine coupling layers and thereby prove convergence while quantifying the convergence rate. The authors validate the bound and the convergence rate empirically on EMNIST as well as a toy dataset.", "questions": "* Where did you show that the non-Gaussianity is not increasing?\n* Could similar results be derived for other flow architectures?\n* On the right plot of Figure 2, it looks like that deviations are almost exclusively present in two quadrants, which might hint towards a systematic deviation. How do the authors explain this phenomenon? ", "limitations": "The main limitation is that the authors only proved and verified the convergence rate of the non-Standardness, but not the non-Gaussianity.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nAffine Coupling Flows are very popular, being used in countless scientific research projects as well as applications. Hence, getting a better understanding of whether and how they converge is a very important problem. The authors give an intuitive decomposition of the training objective, and I appreciate its visualization in Figure 1. They very thoroughly analyze the non-Standardness, provide and prove several bounds, thereby proving and quantifying convergence. The proofs in the appendix are very detailed and I tried to verify them. To the best of my knowledge, they seem correct but I am not an expert regarding the convergence of machine learning algorithms, so I might have missed something.\n\nI like the short paragraph before the conclusion section very much since the authors provide an intuitive heuristic derived from their results, i.e. that the non-Standardness is reduced by about 50% with every affine coupling layer added to a flow model. This will be very useful for practitioners.\n\n**Weaknesses**\n\nThe main weakness of the paper is that it only considers bounds for the non-Standardness. They only show empirically for EMNIST that the non-Gaussianity decreases similarly; however, this might be highly dependent on the nature of the target distribution, a fact not being reflected in the paper. The authors claim on lines 146-147 that it does not increase but it is not clear to me where this is shown. It might be buried somewhere in the appendix, but this result should be placed more prominently in the main paper.\n\nThe paper focuses on affine coupling flows. However, it would also be interesting to briefly discuss whether the bounds could be extended to other flow architectures.\n\nSince the authors computed the non-Standardness to verify their bound, I was wondering whether the non-Standardness could not be reduced by just whitening the data through a linear transformation before training a flow. In this case, only the non-Gaussianity needs to be minimized. I suppose that this is impractical for large high-dimensional datasets; however, the authors could try this on a smaller toy dataset.\n\n**Conclusion**\n\nI think this is a strong paper and I lean towards accepting it and, hence, I score this paper with a 6. If the authors can clarify some of the criticism I mentioned, I am willing to increase my score.\n\n**Edit**\n\nGiven that the authors addressed my concerns and even showed that their theorems extend to all coupling flows I will upgrade my score to 7.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HxXBOkYRBRs", "reviewer": "Reviewer_HZCb", "summary": "This paper studies normalizing flows, which is an approach for generative modeling relying on minimizing the KL divergence between a data distribution fed through a pushforward map implemented with neural networks and the standard Gaussian distribution. To generate samples, one can sample from the standard Gaussian and then invert this network. The paper studies a particular variant of normalizing flows called affine coupling flows, which use RealNVP layers. The main theoretical result analyzes a decomposed KL divergence into a non-Gaussian part and a nonstandardness part. Theorem 1 shows that the nonstandardness can decrease after the application of an affine coupling block, and the result is made interpretable in Theorem 2. Theorem 3 then shows that for deeper networks the nonstandardness can decrease at an exponential rate. Experiments confirm the upper bound and show the exponential decrease with respect to deep networks.", "questions": "- Are you able to write your bounds as high probability bounds rather than bounds on the expected decrease? \n- What proof techniques used might be more broadly interesting and applicable?", "limitations": "The limitations of the analysis do not appear to be discussed in the text, contrary to the fact that the authors mention that they are discussed in Section 6.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- There is not existing theory studying the optimization properties of RealNVP layers. This paper tackles an important open question evaluating theoretically the effectiveness of these layers.\n- The paper is generally well-written.\n- The decomposition of the KL divergence into parts seems to be a useful tool for the analysis of these layers.\n\nWeaknesses\n- The result stands alone and is not close to a complete result. What I mean by this is that the non-Gaussianity term and the non-standardness results are optimized together when the RealNVP layers are implemented. This means that, in theory, the actual minimization procedure may not decrease the non-standardness at all, since there may be a tradeoff between these terms. Therefore, studying one of these terms alone seems unrealistic, and I don't know how it should relate to practical methods.\n- While the exponential decrease is interesting and seems to hold in practice, I don't see how this theoretical result could be useful in guiding methodology. The authors suggest a heuristic for the reduction of the nonstandardness but this only seems to be motivated by the experiments and not the theory. Can be we guaranteed that the reduction is close to 50% for real data, or can we estimate somehow the expected reduction from a dataset?\n- The tight upper bound takes exponential time to compute and is hard to interpret, while the looser bounds appear to be quite loose according to the plots.\n\nUpdate: \nI still have reservations about the interaction of the non-Gaussianity term with the non-standardness term. However, the author's answers alleviate at least some of my concern, and so I will raise my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept