Implicit Normalizing Flows
Authors: Cheng Lu
Jianfei Chen
Chongxuan Li
Qiuhao Wang
Jun Zhu
Venue: ICLR 2021 Spotlight (2021)
Keywords: Normalizing flows
deep generative models
probabilistic inference
implicit functions

Abstract: Normalizing flows define a probability distribution by an explicit invertible transformation $\boldsymbol{\mathbf{z}}=f(\boldsymbol{\mathbf{x}})$. In this work, we present implicit normalizing flows (ImpFlows), which generalize normalizing flows by allowing the mapping to be implicitly defined by the roots of an equation $F(\boldsymbol{\mathbf{z}}, \boldsymbol{\mathbf{x}})= \boldsymbol{\mathbf{0}}$. ImpFlows build on residual flows (ResFlows) with a proper balance between expressiveness and tractability. Through theoretical analysis, we show that the function space of ImpFlow is strictly richer than that of ResFlows. Furthermore, for any ResFlow with a fixed number of blocks, there exists some function that ResFlow has a non-negligible approximation error. However, the function is exactly representable by a single-block ImpFlow. We propose a scalable algorithm to train and draw samples from ImpFlows. Empirically, we evaluate ImpFlow on several classification and density modeling tasks, and ImpFlow outperforms ResFlow with a comparable amount of parameters on all the benchmarks.

Published as a conference paper at ICLR 2021
IMPLICIT NORMALIZING FLOWS
Cheng Lu', Jianfei Chen', Chongxuan Li", Qiuhao , Jun Zhu 'Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center "Tsinghua- Bosch Joint ML Center, THBI Lab, Tsinghua University, Beijing, 100084 China *Center for Data Science, Peking University, Beijing, 100871 China {lucheng. lel chris. ci deszj0t singhussinuaa..... edu, en, wgh1 90pku. edu. cn
chen, chongxuan 991 } j .com,
ABSTRACT
Normalizing flows define probability distribution by explicit invertible trans- formation z = fxx). In this work, we present implicit normalizing flows (ImpFlows), which generalize normalizing flows by allowing the mapping be implicitly defined by the roots of an equation F(2,x) O. ImpFlows build residual flows (ResFlows) with proper balance between expressiveness and tractability. Through theoretical analysis, we show that the functiion space of ImpFlow is strictly richer than that of ResFlows. Furthermore, for any ResFlow with a fixed number of blocks, there exists some function that ResFlow has non- negligible approximation error. However, the function exactly representable by single-block ImpFlow. We propose scalable algorithm to train and draw sam- ples from ImpFlows. Empirically, we evaluate ImpFlow on several classification and density modeling tasks, and ImpFlow outperforms ResFlow with a compara- ble amount of parameters on all the benchmarks.
INTRODUCTION
Normalizing flows (NFs) (Rezende & Mohamed| 2015 Dinh et al. (2014) are promising methods for density modeling. NFs define a model distribution (x) by specifying an invertible transformation f() from x to another random variable z. By change-of-variablle formula, the model density is
Ixpx(x) = px + In (x))(
where Pa (z(z) follows simple distribution, such as Gaussian. NFs are particularly attractive due to their tractability, i.e., the model density Px Px(x) can be directly evaluated as Eqn. ( To achieve such tractability, NF models should satisfy two requirements: (i) the mapping between x and is invert- ible; (ii) the log-determinant of the Jacobian (x is tractable. Searching for rich model families that satisfy these tractability constraints is crucial for the advance of normalizing flow research. For the second requirement, earlier works such as inverse autoregressive flow (Kingma et al. 2016) and
RealNVP (Dinh et al. 2017) restrict the model family to those with triangular Jacobian matrices.
More recently, there emerge some free-form Jacobian approaches, such as Residual Flows (Res- Flows) (Behrmann et al. 22019] Chen et al. 2019). They relax the triangular Jacobian constraint by utilizing a stochastic estimator of the determmmnn,, enriching the model family. However, the Lipschitz constant of each transformation block is constrained for invertibility. In general, this is not preferable because mapping simple prior distribution to a potentially complex data distribution may require a transformation with a very large Lipschitz constant (See Fig. 3 for a 2D example). Moreover, all the aforemennioned methods assume that there exists explicis forward mapping z = [x). Bijections with explicit forward mapping only covers a fraction of the broad class of invertible functions suggested by the first requirement, which may limit the model capacity.
In this paper, we propose implicit flows (ImpFlows) to generalize NFs, allowing the transformation to be implicitly defined by an equation F(z, x) O. Given x z)rz), the other variable can be computed
by an implicit root pond-iinding procedure z = RootFind F x)). An explicit mapping z = f(x) used
in prior NFs can viewed a special case of ImpFlows in the form of F F(z, x) f(x)
"ooresponning Author.

---

Published as a conference paper at ICLR 2021
O. To balance between expressiveness and tractability, present specific from of ImpFlows, where each block is the composition of a ResFlow block and the inverse of another ResFlow block. We theoretically study the model capacity of ResFlows and ImpFlows in the function space. We show that the function family of single-block ImpFlows is strictly richer than that of two-block ResFlows by relaxing the Lipschitz constraints. Furthermore, for any ResFlow with fixed number of blocks, there exists some invertible function that ResFlow has non-negligible approximation error, but ImpFlow can exactly model.
On the practical side, we develop a scalable algorithm to estimate the probability density and its gradients, and draw samples from ImpFlows. The algorithm leverages the implicit differentiation formula. Despite being more powerful, the gradient computation of ImpFlow is mostly similar with that of ResFlows, except some additional overhead on root finding. We test the effectiveness of ImpFlow on several classification and generative modeling tasks. ImpFlow outperforms ResFlow on all the benchmarks, with comparable model sizes and computational cost.
2 RELATED WORK
Expressive Normalizing Flows There are many works focusing on improving the capacity of NF'. For example, Dinh et al. (2014) 2017); Kingma & Dhariwal (2018); Ho et al. 2019); Song et al.
(2019;; Hoogeboom al. (2019); De Cao et al. (2020); Durkan al. (2019) design dedicated model architectures with tractable Jacobian. More recently, Grathwohl et al. (2019); Behrmann al.] (2019); Chen et al. (2019) propose NFs with free-form Jacobian, which approximate the de-
terminant with stochastic estimators. In parallel with architecture design, Chen et al. (2020); Huang
al. (2020): (2020);(ornis) Cornish et al, (2020); Nielssn et al. (2020) improve the capacity of NEs by operating in higher-dimensiooall space. As mentioned in the introduction, all these existing works adopt explicit forward mappings, which is only a subset of the broad class of invertible functions. In contrast, the implicit function family we consider is richer. While primarily discuss the implicit generaliza- tion of ResFlows (Chen et al. 2019) in this paper, the general idea of utilizing implicit invertible functions could be potentially applied to other models as well. Finally, Zhang et al. (2020) formally prove that the model capacity of ResFlows is restricted by the dimension of the residual blocks. In comparison, we study another limitation of ResFlows in terms of the bounded Lipschitz constant, and compare the function family of ResFlows and ImpFlows with a comparable depth.
Continuous Time Flows (CTFs) (Chen al.) 2018b Grathwohl et al. 2019] Chen et al. 2018a) are flexible alternative to discrete time flows for generative modeling. They typically treat the invertible transformation as a dynamical system, which is approximately simulated by ordinary differential equation (ODE) solvers. In contrast, the implicit function family considered in this paper does not contain differential equations, and only requires fixed point solvers. Moreover, the theoretical guarantee is different. While CTFs typically study the universal approximation capacity under the continuous time case (i.e., "infinite depth" limit), we consider the model capacity of ImpFlows and ResFlows under a finite number of transformation steps. Finally, while CTFs are flexible, their learn- ing is challenging due to instability (Liu et al., 2020 Massaroli et al. 2020) and exceedingly many ODE solver steps (Finlay et al., 2020), making their large-scale application still an open problem.
Implicit Deep Learning Utilizing implicit functions enhances the flexibility of neural networks, enabling the design of network layers in a problem-specifee way. For instance, Bai et al. (2019) propose a deep equilibrium model as a compact replacement of recurrent networks; Amos & Kolter (2017) generalize each layer solve optimization problem: WWang et al. (2019) integrate logical reasoning into neural networks; RRshhiak & Webster (2019) utilize the implicit Euler method to improve the stability of both forward and backward processes for residual blocks; and Sitzmann et al. (2020) incorporate periodic functions for representation learning. Different from these works, which consider implicit functions as replacement to ned-forwwrrd networks, we develop invertible implicit functions for normalizing flows, discuss the conditions of the existence of such functions, and theoretically study the model capacity of our proposed ImpFlow in the function space.
3 IMPLICIT NORMALIZING FLOWS
We now present implicit normalizing flows, by starting with brief overview of existing work.
2

---

Published as a conference paper at ICLR 2021
3.1 NORMALIZING FLOWS
As shown in Eqn. (T. a normalizing flow f x + 2 is an invertible function that defines a proba- bility distribution with the change-of-variian formula. The modeling capacity of normalizing flows depends on the expressiveness of the invertible function f. Residual flows (ResFlows) (Chen et al.) 2019 Behrmann et 2019) are a particular powerful class of NFs due their free-form Jacobian. ResFlows use f JL o fi to construct the invertible mapping. where each layer fi is an invertible residual network with Lipschítz constraints bounded by fixed constant k:
[i() = x + (x); Lip(g)) S < 1,
where Lip(g) is the Lipschitz constant of a function g (see Sec. 4.1 for details). Despite their free-form Jacobian, the model capacity of ResFlows is still limited by the Lipschitz constant of the invertible function. The Lipschitz constant of each ResFlow block fi cannot exceed 2 (Behrmann) et al. 2019). the Lipschitz constant of L-block ResFlow cannot exceed 2 However, to transfer a simple prior distribution to potentially complex data distribution, the Lipschitz constant be required to be sufficiently large in general. Therefore, ResFlows be
of the transformation undesirably deep simply to meet the Lipschitz constraints (see Fig. F 3or a 2D) example). Below, we present implicit flows (ImpFlows) relax the Lipschitz constraints.
3.2 MODEL SPECIFICATION
In general, an implicit flow (ImpFlow) is defined as an invertible mapping between random variables x and of dimension d by finding the roots of F(z, x) = O. where F is function from [ to Rd fx) used prior flow instances (Chen al. 2019
In particular, the explicit mappings & Dhariwal 2018) can be expressed as an implicit function in the form F(z, x) [(x)
Kingma
0.
While lowssows are a powerful family to explore, generally they are not guaranteed to satisfy the invertibility and the tractability of the log-determinant as required by NFs. In this paper, we focus on the following specific form, which achieves a good balance between expressiveness and tractability, and leave other possibilities for future studies.
Definition 1. Let 9x. IRd + Rd and Ox Rd -> Rd be two functions such that Lip(gx) < 1 and Lip(ge) < 1, where Lip(g) is the Lipschitz constant of a function g. A specific form of ImpFlows is defined by
F(z, x) O, where F(z,x) (x(x) (z) -
(3)
The root pairs of Eqn. 3 form subset in Rd x Id, which actually defines the assignment rule of a unique invertible function f. To see this, for any Xo. according to Definition T we can construct contraction 1xx (z) = Fzz, xo) + with unique fixed point, which corresponds to unique root (w.rt. z) of Fiz, Xo) = 0, denoted by fxx). Similarly, in the reverse process, given a Zo. the root also exists and is unique, denoted by f f=)( These two properties sufficient to ensure the existence and the invertibility of f, as summarized in the following theorem.
(w.rt. of F F(Zg, x)
Theorem 1. Eqn. 3 defines a unique mapping f R2d + Id z = fxx), and f is invertible.
Theorem characterizes the validness of the ImpFlows introduced in Definition In fact, a single ImpFlow is stack of single ResFlow and the inverse of another single ResHlow, which will be formally stated in Sec We will investigate the expressiveness the function family of the ImpFlows in See and present scalable algorithm to lear a deep generative model built upon ImpFlows in Sec.
See proof in Appendix A.I I..I
4 EXPRESSIVENESS POWER
We first present some preliminaries on Lipschitz continuous functions in Sec. 4.1 and then formally study the expressiveness power of ImpFlows, especially in comparison to ResFlows. In particular, prove that the function space of ImpFlows is strictly richer than that of ResFlows in Sec. 4.2
(see an illustration in Fig. [ (a)). Furthermore, for any ResFlow with a fixed number of blocks, there exists some function that ResFlow has non-negligible approximation error. However, the function is exactly representable by a single-block ImpFlow. The results are illustrated in Fig. (b)
and formally presented in Sec. 4.3
3

---

Published as a conference paper at ICLR 2021
Lemma
R
F
Equation
Theorem composition
(2 commpsittoon)
Corollars
P
I
R
R2
Rt
R2
(a) Relationship between R2 and I.
(b) Relationship between R2 and I.
Figure 1: An illustration of our main theoretical results on the expressiveness power of ImpFlows and ResFlows. Panel (a) and Panel (b) correspond to results in Sec. 4.2 and Sec. 4.3 respectively.
20
13
1.-
15-
80
011
0.0
0.-
1.0
02
0.5
-1.
-05
00
03 1.5
0.5
1.5
(a) Target function
(b) ResFlow
(c) ImpPlow
(d) Composition
Figure 2: A 1-D motivating example. (a) Plot of the target function. (b) Results of fitting the
target function using ResFlows with different number of blocks. All functions have non-negligible approximation error due to the Lipschtiz constraint.
(c) An ImpFlow that exactly represent the target function. (d) A visualization of compositing a ResFlow block and the inverse another ResFlow block construct an ImpFlow block. The detailed settings be found in Appendix D
4.1 LIPSCHITZ CONTINUOUS FUNCTIONS
For any differentiable function / Rd RRd and any x E RRd, we denote the Jacobian matrix of f at as (x) E Rdxd
Definition 2. A function Rd + Rdd is called Lipschitz continuous if there exists a constant L, s.t. [x(x)) - fxx)) S L|xx - x2l1, Vx,,X, E Rdd.
The smallest L that satisfies the inequality is called the Lipschitz constant of f. denoted Lip(f).
Generally, the definition of Lip(f) depends on the choice of the norm || ||. while we use L2-norm by default in this paper for simplicity.
Definition 3. function Ed + Rd is called bi-Lipschitz continuous if it is Lipschitz continuous and has an inverse mapping f-1 which is also Lipschitz continuous.
It is useful to consider equivalent definition of the Lipschitz constant in following analysis. Proposition (Rademacher (Federer 1969), Theorem 3.1.6)) If Rd + Id is Lipschitz contin- uous, then is differentiable almost everywhere, and Lip(f) = sup (x) xE3
where M (up(ve[[v]a==1) ||Mv||a is the operator norm of the matrix M ddd
4.2 COMPARISON TO TLO-BLOCK RESFLOWS
We formally compare the expressive power of single-block ImpFlow and two-block ResFlow. We highlight the structure of the theoretical results in this subsection in Fig. (a) and present a 1D motivating example in Fig. All the proofs can be found in Appendix. A
On the one hand, according the definition of ResFlow, the function family of the single-block ResFlow
R= (f 9 Id, g C.
C (R. R ). <1}.
(4)
4

---

Published as a conference paper at ICLR 2021
where C² (RR,, , I consists of all functions from IR to R with continuous derivatives and Id de- notes the identity map. Besides, the function family of E-block ResFlows defined by composition:
R2 = f fro 0..00 for some f1...., .fe E R}.
(f
By definition of Eqn. 4 and Eqn. R1. R1 = R.
On the other hand, according to the definition of the ImpFlow in Eqn. (3) we can obtain (gx + Id)(x) (x) + - Ga(2)
(9x + [2), where denotes the composition of functions. Equivalently, we have (lg2 + Iel) (gx + Id)) (x), which implies the function family of the single-block ImpFlow is
I -= o fi for some S1, E RJ.
(6)
Intuitively, a single-block ImpFlow can be interpreted as the composition of ResFlow block and the inverse function of another ResFlow block, which may not have an explicit form (see Fig. 2 (c) and (d) for ID example). Therefore, it is natural to investigate the relationship between I and R2. Before that, we first introduce a family of "monotonically increasing functions" that does have an explicit Lipschitz constraint, and show that it is strictly larger than R.
Lemma 1.
R =FF=lfED:
inf v Jz(x)V > 0},
(7)
XER -
where D is the set of all bi-Lipschitz C'-diffeomorphissss from Ra to Rdd and S B means A is proper subset of B.
Note that it follows from Behrmann al. 2019 Lemma 2) that all functions in R are bi-Lipschitz, so R S D. In the 1D input case, we can get R (f E (R) :inf f'(x) > 0, DUSUPERR
< 2), and = { C²(R): inf=ER (x) > 0j. In the high dimensional cases, R. and J are hard to illustrate. Nevertheless, the Lipschitz constants of the functions in R is less than 2 (Behrmann al.) 2019), but those of the functions in F can be arbitrarily large. Based on Lemmal) [] we prove that the function family of ImpFlows I consists of the compositions of two functions in F, and therefore is a strictly larger than R2, as summarized in the following theorem.
Theorem 2. (Equivalent form of the function family of a single-block ImpFlow).
F= = {f f= fi for some f1, E F).
(8)
Note that the identity mapping Id E F, and is easy to get F C I. Thus, the Lipschitz constant of single ImpFlow (and its reverse) can be arbitrarily large. Because R F and there exists some functions in I \ R2 (see a constructed example in Sec. 4.3) we can get the following corollary.
Corollary 1. R S R2 F22=1 I.
The results on the ID example in Fig. (b) and (c) accord with Corollary [| Besides, Corollary be generalized to the cases with b6-block ResFlows and ê-block ImpFlows, which strongly motivates the usage of implicit layers in normalizing flows.
| can
4.3 COMPARISON WITH MULTI-BILOCK RESFLOWS
We further investigate the relationship between R. for l 2 and I, as illustrated in Fig. [ For a fixed E, the Lipschitz constant of functions in R₂ is still bounded, and there exist infinite functions that are not in R. but in I. We construct one such function family: for any L, E Rt define
P(L,r) (S F, C RRd Vx,y y E B,, lI((x) f(y)l22 7||x y112),
(9)
where B, is d-dimensional ball with radius of r. Obviously, P(L,r) is infinite set. Below, we will show that A < & < (L), R₂ has a neg-lggli approximation error for functions in P),,r) However, they are exactly representable by functions in I.
Theorem 3. Given
and we have
P(L,r) CI
5

---

Published as a conference paper at ICLR 2021
0 < logz(L), P(L,))RRe 2. Moreover, for any f E P(L, r) with d-dimensional ball B,, the minimal error for fitting f in B, by functions R4 satisfies
T inf sup [l g(x))(())))) 2/1 22) BER, xEB,
(10)
It follows Theorem 3 that model P(,,r), we need only dingle-block ImpFlow but least a log, (b)-block ResFlow. In Fig. 2 (b), we show ID case where a 3-block ResFlow cannot fit function that is exactly representable by a single-block ImpFlow. In addition, we also prove some other properties of ImpFlows. In particular, R3 g I. We formally present the results in Appendix B
5 GENERATIVE MODELING WITH IMPFLOWS
ImpPlows can be parameterized by neural networks and stacked to form deep generative model to model high-dimensional data distributions. We develop scalable algorithm to perform inference, sampling and learning in such models. For simplicity, we focus on single-block during derivation.
Formally, a parametric ImpFlow block z = [(x: 0) is defined by
F(z, 0) 0, where F(z, 0) 9x 0) 9z (z 0) x
(1I)
and Lix( < 1, Lip(g,) < 1. Let denote all the parameters in Jx and Iz (which does NOT mean Ih and Oz. share parameters). Note that x refers to the input of the layer, not the input data.
The inference process to compute given x in a single ImpFlow block is solved by finding the root of F(z,x; f)) 0 w.r.t. z, which cannot be explicitly computed because of the implicit formulation. Instead, we adopt quasi-Newton method (i.c. Broyden's 's method (Broyden. 1965)) to solve this problem iteratively, follows:
gl+ -aF x; 0), for 0,
(12)
where is low-rank approximation of the Jacobian inverse and a is the step size which we use line search method to dynamically compute. The stop criterion is [z x; 0) < ef. where s a hyperparameter that balances the computation time and precision. As Theorem guarantees the existence and uniqueness of the root, the convergence of the Broyden's 's method is also guaranteed, which is typically faster than a linear rate.
Another inference problem is to estimate the log- ligelikel Assume that z ~ p(z) where p(z) is a simple prior distribution (e.g. standard Gaussian). The log-likeidhod of x can be written by
In p(x) In p(z) + det Jo, (x)) - In det( + Jo. (z)),
(13)
where (x) denotes the Jacobian matrix of function f at x. See Appendix. A.4 for the detailed derivation. Exact calculation of the log-determinant term requires (id)) time cost and is hard to scale up to high- dimensionaii data. Instead, we propose the following unbiased estimator of in p(x) using the same technique Chen al. (2019) with Skilling Hutthhissntt trace estimator (Skilling 1989 Hutchinson 1989):
(-1)kt] (vT (x) v vJ (z)*)v)))
In ((x) In p(z) E
(14)
k
P(N 2 k)
where p(N) is a distribution supported over the positive integers.
The sampling process to compute given z can also be solved by the Broyden's method, and the hyperparameters are shared with the inference process.
In the learning process, we perform stochastic gradient descent minimize the negative log- likelihood of the data, denoted as L. For efficiency, we estimate the gradient w.r.t. the model parameters in the backpropagation manner. According to the chain rule and the additivity of the log determmermn,n,, in each layer we need to estimate the gradients w.r.t. x and of Eqn. (I3). In particular, the gradients computation involves two terms: one is 0 In det([ + Jx(x:: f)) and the
Øt.)
We refer readers Broyden 1965 for the calculation details for B.

---

Published as a conference paper at ICLR 2021
Table I: Classification error rate (%) test set of vanilla ResNet, ResFlow and ImpFlow of ResNet- 18 architecture, with varying Lipschitz coefficients c.
Vanilla
0.99
c 0.9
c= 0.8
c= 0.7
c 0.6
8.24
8.39
8.69
9.25
9.94
ResFlow
(+0.03)
(=0.01)
(=0003)
(+0.02)
(=0.02)
CIFARIO
6.61(+00022
7.29
7.41
7.94
8.44
9.22
ImpFlow
(#0.03)
(#0.03)
(=0.06)
(#0.04)
(=0.02)
31.02
31.88
32.21
33.58
34.48
ResFlow
(+0.05)
(+0.02)
(+0000)
(#0.02)
(+0.03)
CIFAR100
27.83(+0.03)
29.06
30.47
31.40
32.64
34.17
ImpFlow
(+00.03
(+0.03)
(+:0.03)
(+0.011
(+0.02)
Oz other is oz B(-) where g is a function satisfying Lip(g) and (+) denotes x or 0. On the one hand, for the log- deg-determnnatt term, we can use the same technique as Chen et al. (2019), and obtain an unbiased gradient estimator as follows.
@ In det(1 + (x(x;)) 0)) v
(-1)* P(N k)
0Jz 0) 0(-)
Σ
v 0x
0(-)
(15)
where p(N) is distribution supported over the positive integers. On the other hand, computed according to the implicit function theorem as follows (See details Appendix A.5):
0C 80
can be
dn 80)
OL Oz ac Oz (.)) Oz J (z)
OF(z,x:0)
where Gizz; 0) = Iu (z: 0)
(16)
a(-)
In comparision to directly calculate the gradient through the quasi-Newton iterations of the forward gradient
pass, the gradent above simple and memory-efficient treating the root solvers as a black- box. Following Bai et al. (2019), compute 26 (z) by solving linear system iteratively, as
detailed in Appendix C.T
The training algorithm is formally presented in Appendix C.4
6 EXPERIMENTS
We demonstrate the model capacity of ImpFlows on the classification and density modeling task In all experiments, constrants. where the Lipschitz constant upper bound of each layer (called Lipschitz coefficient) is denoted as c. For the Broyden's method, we use ey = 10 and e, 10 for training and testing to numerically ensure the invertibility and the stability during training. Please see other detailed settings including the method of estimating the log-determinant, the network architecture, learning rate, batch size, and so on in Appendix D
use spectral normalization (Miyato al., 2018) to enforce the Lipschitz
6.1 VERIFYING CAPACITY ON CLASSIFICATION
We first empirically compare ResFlows and ImpFlows on classification tasks. Compared with gen- erative modeling, classification is because it isolates the function fitting from generative modeling subtleties, such as log-determinant estimation. We train both models in the same settings on CIFARIO and CIFAR 100 (Krizhevsky & Hinton 2009). Specifically, we use an architecture similar to ResNet- 18 (He et al. 2016). Over- all, the amount parameters of ResNet-19 with vanilla ResBlocks, ResFlows and ImpFlows are the same of 6.5M. The detailed network structure can be found in Appendix(D The classification results are shown in Table To see the impact of the Lipschitz constraints, we vary the Lipschitz coefficient to show the difference between ResFlows and ImpFlows under the condition of fixed Lipschitz upper bound. Given different values c, the classification results of ImpFlows are consistently better than those of ResFlows. These results empirically validate Corollary I which claims that the
direct measure of the richness of the functional family,
"See http:///gtttiiiooooioooio/iiiiia flows for details.

---

Published as a conference paper at ICLR 2021
Table 2: Average test log-likelihood (in nats) tabular datasets. Higher is better.
POWER
GAS
HEPMASS
MINIBOONE
BSDS300
RealNVP (Dinh et al. 2017)
0.17
8.33
-18.71
-13.55
153.28
FFJORD Grathwohl et al.
2019
0.46
8.59
-14.92
-10.43
157.40
MAF Papamakanos et al.
2017)
0.24
10.08
-17.70
-11.75
155.69
NAF Huang et al.
2018
0.62
11.96
-15.09
-8.86
157.73
ImpFlon low (Z. 20)
0.61
12.11
-13.95
-13.32
155.68
ResFlow (L = 10)
0.26
6.20
-18.91
-21.81
104.63
ImpFlow (L 5)
0.30
6.94
-18.22
-21.50
113.72
Table 3: Average bits per dimension of ResFlow and ImpFlow CIFARIO, with varying Lipschitz coefficients c. Lower is better.
c == 0.9
c c= 0.8
C c= 0.7
C == 0.6
ResFlow (I. = 12)
3.820(+0.0003)) ImpFlow (L 6) 3.452(450.0033) 3.511(+000002) 3.607(L0.0003) 3.814(+0.0005)
(.400(++)00044)
3.533(+0.00022
3.627(+000004))
functional family of ImpFlows is richer than ResFlows. Besides, for large Lipschitz constant upper bound ImpFlow blocks are comparable with the vanilla ResBlocks in terms of classification.
6.2 DENSITY MODELING ON 2D ToY DATA
For the density modeling tasks, we first evaluate ImpFlows on the Checkerboard data whose density is multi-modal, as
shown in Fig. 3 (a). For fairness, we follow the same experiment settings as Chen et al. (2019) (which are specified in Appendix D). except that we adopt a Sine (Sitzmann et al., 2020) activation function for all models. We note that the data distribution has a bounded support while we want to fit transformation f mapping it to the standard Gaussian dis- tribution, whose support is unbounded. A perfect f requires sufficiently large [x)]22 for some mapped far from the mean of the Gaussian. Therefore. the Lipschtiz constant of such f is too large to be fitted by a ResFlow with blocks (See Fig. B (b)). A 4-block ImpFlow can achieve result of 5.05 bits, which outperforms the 5.08 bits of a 8-block ResFlow with the same number of parameters. Such results accord with our theoretical results in Theorem 2 and strongly motivate ImpFllos.
Checkerboard data (5.00 bits)
ResFlow,
ImpFlow,
L = 8 (5.08 bits)
4 (5.05 bits)
Figure 3: Checkerboard data density and the results of 8-block ResFlow and 4-block ImpFlow.
6.3 DENSITY MODELING ON REAL DATA
We also train ImpFlows on some real density modeling datasets, including the tabular datasets (used by Papamakarios al. (2017). CIFARIO and 5-bit 64 64 CelebA (Kingma & Dhariwal 2018). For all the real datasets, we use the scalable algorithm proposed in Sec. 5
We test our nerformance on five tabular datasets: POWER (d 6). GAS (d = 8), HEPMASS (d = 21), MINIBOONE (d = 43) and BSDS300 (d = 63) from the UCI repository (Dua & Graff, 2017). where is the data dimension. For fair comparison, on each dataset we use a 10-block ResFlow and a 5-block ImpFlow with the same amount of parameters, and 20-block ImpFlow for a better result. The detailed network architecture and hyperparameters can be found in Appendix D Table shows the average test log- likeeioodoodd for ResFlows and ImpFlows. ImpFlows achieves better density estimation performance than ResFlow consistently on all datasets. Again, the results demonstrate the effectiveness of ImpFlows.
8

---

Published as a conference paper at ICLR 2021
Then we test ImpFlows the CIFARIO dataset. We train multi-scale convolutional version for both ImpFlows and ResFlows, following the same settings as Chen et al. (2019) except that we use a smaller network of 5.5M parameters for both ImpFlows and ResFlows (see details in Appendix D)
As shown in Table 3 Impflow achieves better results than ResFlow consistently given different values of the Lipschitz coefficient c. Moreover, the computation time of ImpFlow is comparable to that of ResFlow. See Appendix C.2 for detailed results. Besides, there is a trade-off between the expressiveness and the numerical optimization of ImpFlows in larger models, Based on the above experiments, we believe that advances including an lower-variance estimate of the log-determinant can benefit ImpFlows in larger models, which is left for future work.
We also train ImpFlows on the 5-bit 64 x 64 CelebA. For a fair comparison, we use the same settings as Chen et al. (2019). The samples from our model are shown in Appendix
CONCLUSIONS
We propose implicit normalizing flows (ImpFlows), which generalize normalizing flows via utilizingg an implicit invertible mapping defined by the roots of the equation F(z, x) = O. ImpFlows build on Residual Flows (ResFlows) with good balance between tractability and expressiveness. We show that the functional family of ImpFlows is richer than that ResFlows, particularly for modeling functions with large Lipschitz constants. Based on the implicit differentiation formula, we present a scalable algorithm to train and evaluate ImpFlows. Empirically, ImpFlows outperform ResFlows on several classification and density modeling benchmarks. Finally, while this paper mostly focuses on the implicit generalization of ResFlows, the general idea of utilizing implicit functions for NFs could be extended to a wider scope. We leave it as future work.
ACKNOWLEDGEMENT
We thank Yuhao Zhou, Shuyu Cheng, Jiaming Li, Kun Xu, Fan Bao, Shihong Song and Qi' An Fu for proofrcading. This work was supported by the National Key Research and Development Program of China (Nos. 2020AAA0104304), NSFC Projects (Nos. 61620106010, 62061 136001, U19B2034, U181146, 62076145), Beijing NSF Project (No. JQ19016), Beijing Academy of Artificial Intel- ligence (BAAI), Huange Joint Research Program, Huawei Hisilicon Kirin Intelligence Engineering Development, the MindSpore team, a grant from Tsinghua Institute for Guo Qiang, Tiangong Institute for Intelligent Computing, and the NVIDIA NVAIL Program with GPU/DGX Acceleration. C. Li was supported by the fellowship of China postdoctoral Science Foundation (2020M680572), and the fellowship of China national postdoctoral program for innovative talents (BX20190172) and Shuimu Tsinghua Scholar. J. Chen was supported by Shuimu Tsinghua Scholar.
REFERENCES
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pp. 1356145,220 2017.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Advances in Neural Information Processing Systems, 2019.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Hearik Jacobsen. Invertible residual networks. In International Conference on Machine Learning, pp. 573-582, 2019.
Charles G Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of Computation, 1999)):577553, 1965.
Changyou Chen, Chunyuan Li, Liqun Chen, Wenlin Wang, Yunchen Pu, and Lawrence Carin Duke. Continuous-time flows for efficient inference and density estimation. In International Conference on Machine Learning, pp. 023. 2018a
Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. Vflow: More expressive generative flows with variational data augmentation. In International Conference on Machine Learning, 2020.

---

Published as a conference paper at ICLR 2021
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jömn- Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, pp- 9916-9926, 2019.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differ- ential equations. In Advances in Neural Information Processing Systems, pp. 6571-6583, 2018b.
Rob Cornish, Anthony L Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. International Conference on Machine Learning, 2020.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Block neural autoregressive flow. In Uncertainty in Artificial Intelligence, pp. 1263-1273, PMLR, 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti-
mation, In International Conference on Learning Representations Workshop. 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In
International Conference Learning Representations, 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:::://arrcive
ics.uci
uci. www.mdd/mm...c edd/ml
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, pp. 7511-7522, 2019.
Herbert Federer. Grundlchren der mathematischen wissenschaften. In Geometric measure theory, volume 153. Springer New York, 1969.
Chris Finlay, Jörn- Jacobsen, Levon Nurbekyan, and Adam M Oberman. How train your neural ode: the world of jacobian and kinetic regularization. In International Conference on Machine Learning. 2020.
Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In International Con- ference on Learning Representations, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedingy the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In International Conference Machine Learning, pp. 2722-2730, 2019.
Emiel Hoogeboom, Rianne Van Den Berg, and Max Welling. Emerging convolutions for generative normalizing flows. In International Conference on Machine Learning, pp. 2771-2780, 2019.
Chin-Was Bhin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078 1778-2087, 2018.
Chin Wei Huang, Laurent Dinh, and Aaron Courville. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models. arXiv:2002.07101. 2020.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Aaitistics-Simllaaii and Computation, 1833):0599-006, 1989.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowiez, Xi Chen, llya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pp. 4743-4751 2016.
10

---

Published as a conference paper at ICLR 2021
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsich. How does noise help robustness? explanation and exploration under the neural sde framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 282-290, 2020.
Stefano Massaroli, Michael Poli, Michelangelo Bin, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Stable neural flows. arXiv preprint arXiv:2003.08000 000000000000,,,. 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018.
Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling, Survae flows: Surjections to bridge the gap between vaes and flows. arXiv preprint XXii:2007.0273111 2020.
George Papamakarios. Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Viktor Reshniak and Clayton Webster. Robust learning with implicit residual networks. arXiv preprint arXivv:905. 10479 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional Conference on Machine Learning, pp. 1530-1538, 2015.
Vincent Sitzmann, Julien NP Martel, Alexander W Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. arXiv preprint arXiv:2006.09661, 2020.
John Skilling. The eigenvalues of mega-dimennt diga-diimnsional matrices. In Maximum Entropy and Bayesian Methods, pp. 455-466 Springer, 1989.
Yang Song, Chenlin Meng, and Stefano Ermon. Mintnet: Building invertible neural networks with masked convolutions. In Advances Neural Information Processing Systems, pp. 11002-11012, 2019.
Po- Po-Wei Wang. Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and log- ical reasoning using a differentiable satisfiability solver. In International Conference on Machine Learning, pp. 6545-6554, 2019.
Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural odes and invertible residual networks. In International Conference on Machine Learning, 2020.
A ADDITIONAL LEMMAS AND PROOFS
A.1 PROOF FOR THEOREM
Proof. (Theorem
Firstly, VXo E Rdd, the mapping
hxo (z) F(z, x²) +z
a contrative mapping, which can be shown by Lipschitz condition of 9z:
[[( xo) 21) (F(22, xo) + 22)1l = (z1) 92 (22))l < || 22ll.
Therefore, hxo (z) has unique fixed point, denoted by f(xo):
hxo (xx)) = (xx) + F(f (xo),
Similarly, we also have: Vzg E Rdd, there exists an unique g( satisfying Flzo. g(zo)) = 0. Moreover, Let 2o f(xg), have ((f g(f(xx)) = Xo. VXn E Rd Similarly, f(g(x))) = Xo. VX. E Rd. Therefore, f is unique and invertible.
0. By the uniqueness, have

---

Published as a conference paper at ICLR 2021
A.2 PROOF FOR THEOREM 2
We denote D as the set of all bi-Lipschitz C'-diffeomorphissms from R to R4.
Firstly, we prove Lemma 1 in the main text.
Proof. (Lemma 1. vf E R, we have
sup xERd
[| 111 1,
which is equivalent
sup XER VER* 1lvlla=1 VER',
[1(}[(x) - livl2 < (Definition of operator norm.)
sup XER xERM,VER< 1ER]:_|v|/2=1
VJ (x) - (x) - 1)v <
v (x) Jx' <0
sup X VERR
Note that (x) is nonsingular, so Vx, Rd, || = we have v [x > 0. Thus,
v (x)) 2v Jz(x)v 2
-2
sup XER V.R".vER".Ilv ||v|[2=1
sup
VER',VER',lvvll
So have
inf VERd,VERd,lvV l|v|l2-1
TT J) > O.
Note that the converse is not true, because Jx()vV > O does not restrict the upper bound of Lipschitz constant of f. For example, when fxx mx where m a positive real number, we have
inf xER xER vERd |lvll2== v Jr(x)V xER .ER l|v||==1
inf
vJ (mljv
0
However, m can be any large positive number. So we have R & F.
Lemma 2. vf D.
inf xER" VER" liv||==
v=J >0,
(17)
then
inf xER veRd liv|==1
v J
(18)
Proof.
(Proof of Lemma 2. By Inverse Function Theorem,
(x- 1
Because f is from 12d gd we have
inf xERd |M
v 1
inf xER vER* llvll==1
,TJ (f-
v vJJ (x)v vERl
inf
Let = ( and vo |म]. we have // 1, and u =f
The above equation uses this fact: for a real dx matrix A, Vx E R -A = xxTAx)" = ATx because x" Ax R.
Note that f is Lipschitz continuous, [x)x E Tip(S). So
= ||v||2 [x) /ا|l2 s Lip(f)luull,
12

---

Published as a conference paper at ICLR 2021
which means
|ul| 2 Lip(f)
Thus,
inf vER",ER" [|v|
JJ (x)v (xx
inf
||u| (x)
xER vER', VER*.,vvl22=1
inf
||ul|3
inf
(zx)vv
xER ,
XER M1
inf
v J)()v
Tip(S)² xE3d vER.
Lemma 3. AA VS R, we have
inf
J 0.
(19)
xEX" VER" ||v||==1
Proof. (Proof Lemma 3 vf DD R. then from Lemma 1
we have
inf
(=))-( 0.
xER VER", M ||v|[==
Note that f-1 D, from Lemma
we have
inf xER VER" l|v]==1
Jy(x(x O.
Lemma 4. Af E D if
inf VER Iv1==1
Jz((x) 0,
then 3oo > 0, s.f. v0<a<an
sup (xJ - -1112 < 1.
xERd
Proof. (Proof of Lemma Denote
4h. Note that f is Lipschitz continuous, so Lip(f) supxERs (x)
3 inf XER4 WeRRd/Iv/ll.. vJJx(x)V.
And
3 > 0.
a0 =
Lip(
13

---

Published as a conference paper at ICLR 2021
4 < du, have
sup [xad{(x) - 1 xERd
sup
- (x) - 1)v
xER', v|el
=1+
sup
JF J,
vER",ERR* M|vll
- + o²
(x))
sup
xER* VER VER 11vl221
+ 2a
XER VER M|v (-v"J,(xx)
sup
+1+a2
sup
v (x)
|[v|[21
2a
inf
(z)vv
= +1+ a² sup XER"
[x); - -2aB
+ a(Lip((f)) 29)
< (p( 233)
a08
< 1.
The above equation uses this fact: for a real dxd matrix A, V. E x , = (x) = A" because x" Ax R.
Proof. (Theorem 2 Denote
P E f2 E DDJ S1, where
inf XER VERd /|v||/===1
v Jr, (x)v > 0, inf v"JJ((x'Vv > 0}.
xER'
Firstly, show that I P. Vf I, assume = f2 fi, where fi R and fz" E R. By
Lemma aad Lemma 3 we have
inf XER VER" 1lv 1==1
vF vz) 0,
inf VER
v='J > 0.
Thus, f E. So C.. Next, we show that P CI. assume / = f2 S1, where
inf xER vER". 12=1
v"' Jj, (x)v > 0,
inf vER
vT [)2 O.
From Lemma 2 we have
inf XER VERS.VER'.IIVV ||v||2=1
v ,- >
From Lemma 3 3aj 02 > 0, s.t. 0A < min )2)
sup
|aJ (x) 7122
xERd
sup
|aJ)-,(x) T112 1.
xERd
Let a
1 min(01,02) Let 92 91, where
91 (x) a (x),
(x(x)= f2 x
14

---

Published as a conference paper at ICLR 2021
We have g(x) f2( f2() f(x), and
99 (x) = a (x),
1 (x) = af2 (x).
82
J 8/
(x) ad 1 (x).
So have
sup
Jg (x) 1 sup |aJJ fi (x) /|2 1,
xE3d
xEX
sup
Il|2
sup ||J | (x) 1| 1.
xERd
xCR
Thus, 91 R and 92 1 In conclusion, I P.
R and f 92 o gl- So f EI. Therefore, P CI.
A.3 PROOF FOR THEOREM
Firstly, we prove lemma of bi-Lipschitz continuous functions.
Lemma 5. ff (RR ll II) (IR",, || II) is bi- bi-Lipschitz continuous, then
1 (x,) - f(x2)]l
LLi((f), Vx,, X2 E R2d, X1 # X2.
Lip(f-1) ||=] - X22l
Proof. (Proof of Lemma 5). VX1,Xo E &R²xx1
x1 + X2, we have
[x((x)) [(x2)]] S Lip(/)llxi -x2ll
||x x2 = 1ffffl (f(xi)) (fx2) & Lip(f=))lf(x)) - f(x2)ll
Thus, we get the results.
Assume a residual flow f = fL ..... o where each layer ft is an invertible residual network: fi(x) = x + (x),, Lip(g)) S < <
Thus, each layer fi is bi-Lipschitz and it follows by Behrmann et al. (2019) and Lemma 5 that
(x,) - fxxx)) x -2 1 1
<1+*<2 2 Vx,, X2 E Rd, X2-
(20)
By multiplying all the inequalities, we can get bound of the bi-Lipschitz property for ResFlows, as shown in 6
Lemma 6. For ResFlows built by f2 o... S1. where fi(x) + (x(). Lip(gr) < then
(1 ()= [1(x))
- -(xx)) < (1++ Vx1, E Rd. x1 + X2.
[x] -xell
Next, we prove Theorem 3
Proof. (Theorem 3 According to the definition of (PL,,), r), we have P(L,r) CCI.
40 vO<l4 logz(L), we have L - 2f > 0. V R2 by Lemma 6 we have
ll*(x) g(x)ll2 < 2 y/12, Vx, E B,.
Thus, V Xo E Br, we have [x(())
g(x)]12
[xx f(xo) + - + f(xo) - g(x0)1l2
[x f(xo)l2 llg(xx) g(x) f(xu) g(xx))l2
[x 1(x))l12 |lg(xp) g(x)]l2 [1((x) - g(xo)112
(L 20) x (1f(x)) -((x))122
15

---

Published as a conference paper at ICLR 2021
So
sup [xf(x) g(x)]l2 2 sup (L 2|x - xol12 [1f(x)) - g(xo)
x
xEB,
29)r l) - g(xx)ll2
Notice that the inequality above is true for any E B, so we have
sup [x g(x)l1e ž sup
(L 2¹) yr llf(xo) - 9(x0)1l2
x
EB
= (L 2ty inf II gxx0)ll2
x,EB,
2 (L 2 sup lIf(x)) g(xo)ll
xoEB,
Therefore,
sup 1f(x) 9000)lx222222 2 ( (L-2)), 22), Vg e R xEB,
So get
inf sup lIf(x) g(x)ll2 2 r 2 2 gERt xEB,
Because Vʃ P(L,r) inlgeR, supxeB, 11[(x) g(x) 0. have Re (P((,)) Ø.
A.4
PROOF FOR EQUATION
Proof. (Equation 13 By Change of Variable formula:
logp(x = log p(z) + log 10z/0x/ 0x
Since z fxx defined by the equation
F(z.x) 9x (x) gz(z) xx 2 0,
by Implicit function theorem, we have
Öz/Ox = (x = - - -J = (I + Jon (z)) (z))- + Jon (x)).
Thus,
log 0z/0x|= = In det(1 + J/b (x)) - In det(1 / + J (z))
Note that any eigenvalue À of Jo, (x) satisfies | < @J,,. (x)) 1r (x) < 1, so A E (-1,1).
Thus, dT + Jyn (x)) > 0. Similarly, det(T 7 Jy, (z)) 0. Therefore,
log jdz/Ux| In det([I + Jg. (x)) In det(T + (z))
A.5
PROOF FOR EQUATION 16
Proof. (Equation 16
By implicitly differentiating two sides of F(z. x; 0)
by x, we have
(xx(x;0)
Ogz (z; 0) Oz
dr
+1
0,
8x
Oz
0x
Ox
So have
Oz
(
0g 0)
(
(xx
1
1
Ox
Oz
Ox
OF (z, x;
J (z)
Ox
16

---

Published as a conference paper at ICLR 2021
By implicitly differentiating two sides of Fzz,x; 0) = 0 by A, we have
(gx(x: 0) (z 0)
0g (z; 0)
dz
Oz
= 0,
00
00
Oz 00
00
So we have
Oz
Oga (2; 0)
-1 0y (x; 0)
0ga(z;0) (2: 0)
1
00
Oz
00
00
OF x:
J (z)
00
Therefore, the gradient from z to (-) is
0L Oz
0C J (z)
OF (F x;
Oz (.))
Oz
a(.)
B OTHER PROPERTIES OF IMPLICIT FLOWS
In this section, we propose some other properties of ImpFlows. Lemma 7. For single implicit flow f E I, assume that =5z² f1, where
fx( = x + gx(x,, Lip(g)) $ 1, f2(x) x + 92(x,, Lip(g2) S K < 1,
(21)
(22)
then
1 K
lf(x)) - f(x))
1+ K
VxI, X2 E Rd # X2.
(23)
+ | X2|
Proof. (Proof Lemmal
7 According to Eqn.
20 we have
[x11) fxxx))l
< K. Vx,, X2 E RRd, x1 # X2:
(24)
X1 X2ll
1 (x1) -5² (x2)ll S X2 R² + X2.
(25)
1+K Xx1 - -x2ll
By multiplying these two inequalities, can get the results.
Theorem 4. (Limitation of the single ImpFlow).
I {f f D, VXER (J) 2], where A(A) denotes the set of all eigenvalues of matrix A.
(26)
Proof. (Proof of Theorem
Proof by contradiction. Assume 3f E I and E Rd, s.t. YE E X(Jr(x)), A < 0.
There exists a vector u + (x))u = Au. By Theorem 2 3f1.f2 E F.f = f2 o f1, hence
Jy(x) (f) (x). We denote = (f) B Jn (x). Since f1, F, have
vT Av 0, w" Bw > O, Vv, w + 0, v, Rd
Note that B is the Jacobian of bi-Lipschitz function single point, so B is non-singular. As u f O, we have Bu + O. Thus,
(Bu)" A(Bu) (Bu)" ((AB)u) Au Au Bu
The last equation this fact: for real d x d matrix A, Yx Rd. Ax x"" x"'A"" x because x" Ax E R. Note that the left side is positive, and the right side is negative. It's contradiction.
Ax)
17

---

Published as a conference paper at ICLR 2021
Therefore, I cannot include all the bi-Lipschitz C' -diffeomorphhiss. As corollary, have R3 &
I.
Corollary 2. Ra @ I.
Proof. (Proof for Corollary 2 Consider three linear functions in R:
00.46 0.85
-0.20
f1(x) = x+
0.00
x
-0.20
-0.70
f2(x) =x+ 0.30
-0.60
-0.50
- 0.60
fa(x) =x
0.20
0.55
We can get that f f1 f2 o f3 is in Rs, and f is also linear function with Jacobian 0.2776
-0.4293
However, this a matrix with two negative eigenvalues: 10.1881, -0.2100. Hence is not in I. Therefore, R3 & I.
0.5290 -0.6757
C COMPUTATION
C.I APPROXIMATE INVERSE JACOBIAN
The exact computation for the Jacobian inverse term costs much for high dimension tasks. We use the similar technique in Bai (219) compute ac J (z): solving the linear system of variable 8%
y:
Oc 8z
(z))"
(27)
where the left hand side is vector-Jacobian product and it can be efficiently computed by autograd packages foy any y without computing the Jacobian matrix. In this work, we also use Broyden's method to solve the root, the same as methods in the forward pass, where the tolerance bound for the stop criterion is Gp-
Remark. Although the forward, inverse and backward pass ImpFlows all need to solve the root of some equation, we choose small enough e, and e, to ensure the approximation error is small enough. Thus, there a trade-off between computation costs and approximation error. In practice, use ef 10-6 and é,, 10-10 and empirically does observe any error accumulation. Note that such approximation is rather different from the variational inference technique in CCen et al. (020): Nielsen al. 2200)), because only focus on the exact log density itself.
C.2 COMPUTATION TIME
We evaluate the average computation time for the model trained on CIFARIO in Table 3 on a sin- gle Tesla P100 (SXM2-16GB). See Table for the details. For fair comparision, the forward (inference) time in the training phase of ImpFlow is comparable to that of ResFlow because the log-determinant term the main cost. The backward time of ImpFlow costs more than that of Res- Flow because it requires to rewrite the backward method in PyTorch to solve the linear equation.
The training time includes forward, backward and other operations (such as the Lipschitz iterations for spectral normalization). We the same method as the release code of ResFlows (fixed-poni -oint iterations with tolerance 10 5) for the sample phase. The sample time of ImpFlow less than that of ResFlow because the inverse of L-block ImpFlow needs to solve L fixed points while the inverse of 2L-bookk ResFlow needs to solve 2L fixed points. Fast sampling is particularly desirable since is the main advantage of flow- based models over autoregressive models.
Also, we evaluate the average Broyden's 's method iterations and the average function evaluation times during the Broyden's 's method. See Table 5 for the details.
18

---

Published as a conference paper at ICLR 2021
Table 4: Single-batch computation time (seconds) for ResFlow and ImpFlow in Table on single
Tesla P100 (SXM2-16GB).
Model
Forward (Inference)
Backward
Training
Sample
Fixed-point
Log-det
Others 0.090
Inv-Jacob 0.562
Others 0.441
ImpFlow
0.445
2.370
4.152
0.138
0.5
2.905
1.003
ResFlow
2.656
0.031
2.910
0.229
Fixed-point 0.499
Log-det 2.356
Others 0.120
Inv-Jacob 0.451
Others 0.800
ImpFlow
4.415
0.159
0.6
2.973
1,25
ResFlow
2.649
0,033
2.908
0.253
Fixed-poie point 0.533
Log-det 2.351
Others 0.157
Inv-Jacob 0.525
Others 0.887
ImpFlow
4.644
0.181
0.7
3.041
1.412
ResFlow
2.650
0.030
2.908
0.312
Fixed-point 0.602
Log-det 2.364
Others 0.139
Inv-Jacob 0.641
Others 0.943
ImpFlow
4.881
0.206
0.8
3.105
1.584
ResFlow
2.655
0.030
2.910
0.374
Fixed-point 0.707
Log-det 2.357
Others 0.137
Inv-Jacob 0.774
Others 1.033
ImpFlow
5.197
0.258
0.9
3.201
1.807
ResFlow
2.653
0.030
2.916
0.458
Table Single-batch iterations of Broyden's method during forward and backward pass for a single block of ImpFlow in Table 3
Broyden's Method Iterations
Function Evaluations
Forward Backward
7.2
8.2
0.5
12.5
13.5
Forward
8.3
9.3
0.6
Backward
14.9
15,9
Forward
9.4
10.4
0.7
Backward
17.9
18.9
Forward
10.8
11.8
0.8
Backward
22.4
23.4
Forward
12.9
13,9
0.9
Backward
27.4
28.4
C.3 NUMERICAL SENSITIVITY
We train 20-block ImpFlow on POWER dataset with e, 10 (see Appendix. D for detailed settings), and then test this model with different e to see the numerical sensitivity of the fixed-point iterations. 6 that our model is not sensitive to e, in fair range.
C.4 TRAINING ALGORITHM
We state the training algorithms for both forward and backward processes in Algorithm D and Algo-
rithm 2
19

---

Published as a conference paper at ICLR 2021
Algorithm 1: Forward Algorithm For Single-Block ImpFlow
Require: 9x;00 9m:9 Eqn. B3. stop criterion
cf-
Input: x.
Output: z = f(x) and (x),, where / the implicit function defined by 9xe0t and Om:0
Define h(2)) = Flz, (2,,x;00 x)
z f 0
while |1(z)]2 cy do
- The estimated inverse Jacobian of h(z) (e.g. by Broyden's method)
<- LincScarch B)
2 2 - (z)
if training then
Esitamate In det T + J (x; 8)) by Eqn.
15
Esitamate In det [ g. [z; 0)) by Eqn. 15
else
Esitamate In det [ Jgx (x; 8) by Eqn.
Esitamate In det I + gx (z; 8)) by Equ. 14
Compute In)(x) by Eqn.
13
Algorithm 2: Backward Algorithm For a Single-Block ImpFlow
Require: 9x:0, 92:9 Eqn. (3). stop criterion eb. a2
Input: x, z, 8x
Output: The gradient for x and from z, i.e. 0C dz 8z 0x ac
and Oz
82 06
Define G(z; 0) (2; 0) + and h(y) Jz
y <
while [ > eb do
B The estimated inverse Jacobian of h(y) (e.g. by Broyden's method)
< LineSearch(y,, h B)
y
(y))
Compute aF(2.x:0) and (F(xx.0) by autograd packages. 8x
89
8 (2.x.0))
du The
y
tx
BC He
aF (z.x:0)
y
88
20

---

Published as a conference paper at ICLR 2021
Table 6: Average test log-likelihood (in nats) for different ey of ImpFlow POWER dataset.
cf 10-8
10-?
10-6
10-6
10-4
10-3
10-2
10-1
log likelihood
0.606
0.603
0.607
0.611
0.607
0.607
0.602
0.596
D NETWORK STRUCTURES
D.1 1-D EXAMPLE
We specify the function (data) to be fitted is
0.1x, 10x,
x < 0
f(x) =
120
For I, we can construct fully-connected neural network with ReLU activation and parameters as following:
9m (x) ReL.U--099))
9=(=) = VO.9ReLU(V0099)
The two networks be implemented by spectral normalization. Assume the implicit function defined by Eqn. 3 using the above (x) and ga(2) is fz. Next we show that f = fz.
Let (ix) = x + ReLU(-0.9x)) and /2(x) = (0. we have 1x) = x ReI.U(9x)) Therefore, /z = 52 f f-
For every residual block of R,R2 and Ra. we train 4-layer MLP with ReLU activation and 128 hidden units, and the Lipschitz coefficient for the spectral normalization is 0.99, and the iteration number for the spectral computation is 200. The objective function
min B [(fe(z) - f()))²]
where fe is the function of or 2 or 3 residual blocks. We use a batch size of 5000. We use the Adam optimizer, with learning rate 10- and weight decay 10-5, We train the model until convergence, on single NVIDIA GeForce GTX 1080Ti.
The losses for R,R2 and R3 are 5.25, 2.47, 0.32, respectively.
D.2 CLASSIFICATION
For the classification tasks, remove all the BatchNorm layers which are inside of certain Res- Block, and only maintain the BatchNorm layer the downsampling layer. Moreover, as a single ImpFlow consists of two residual blocks with the same dimension of input and output, the downsampling shortcut by a identity shortcut in each scale of ResNet- 18, and add downsam- pling layer (a convolutional layer) with BatchNorm after the two residual blocks of each scale. Thus, each scale consists two ResBlocks with the same dimension of input and output, which (6.5M parameters) different from the vanilla ResNet-18 architecture (11.2M parameters). Note that the "vanilla ResNet-188 18" in our main text is refered to the 4.5M-parameter architecture, which is the same as the versions for ResFlow and ImpFlow.
replace
We use the comman settings: batch size of 128, Adam optimizer with learning rate 10-3 and no weight decay, and total epoch 150. For the spectral normalization iterations, we use a error bound of 10 3 the same as Chen et al. (2019). We train every experiment on single NVIDIA GeForce GTX 2080Ti.
D.3 DENSITY MODELING ON Toy DATA
Following the same settings Chen al. 2019), 4-layer multilayer perceptrons (MLP) with cully-connected layers of 128 hidden units. We use the Adam optimizer with learning rate of 10-3 and weight decay of 10-5 Moreover, find that उे sin(2 is better activation for this
21

---

Published as a conference paper at ICLR 2021
task while maintain the property of 1-Lipschitz constant, experiments, which can lead to faster convergence and better log- ligliiklihood for both ResFlows and ImpFlows, shown in Fig.
this activation function for all
We do not use any ActNorm or BatchNorm layers. For the log-determinant term, we use brute-force computation in Chen et al. (2019). For the forward and backward, use the Broyden's method to compute the roots, with c = 10 The Lipschitz coefficient for spectral normalization 0.999, and the iteration number for spectral normalization is 20. The batch size is 5000, and train 50000 iterations. The test batch size is 10000.
Also, vary the network depth depth L, we use an L-block ImpFlow and 2L-block ResFlow with the same settings as stated above, and train 3 times with different random seeds. As shown Figure [ the gap between ImpFlow and ResFlow shrinks as the depth grows deep, because the Lipschitz constant of ResFlow grows exponentially. Note that the dashed line a 200-block ResFlow in Chen (2019). and we tune our model better that our models perform better with lower depth.
the difference between ImpFlow and ResFlow. For every
(a) LipSwish (5.22 bits)
(b) Sine (5.08 bits)
Figure 4: 8-block ResFlow with different activation function trained on Checkerboard dataset.
ImpFlow
5.12
ResFlow (2x depth)
200-depth ResFlow (Chen, 2019)
5.10
NLL (bits)
5.08
5.04
5
6
7
Depth
Figure 5: Test NLL (in bits) by varying the network depth. Lower is better.
D.4 DENSITY MODELING ON TABULAR DATASETS
We use the same data preprocessing as Papamakarios et al. 2017). including the train/valid/test datasets splits. For all models, rate 10-3 -3 for the Adam optimizer. The main settings are the same as Chen et al. (2019) on the toy21) dataset. The residual blocks 4-leyer ML.ss with 128 hidden units. The ResFlows use 10
batch size of 1000 (both training and testing) and learning
22

---

Published as a conference paper at ICLR 2021
blocks and ImpFlows use blocks ensure the same amount of parameters. And we use 20-block ImpFlow for a better result. Also, we use the Sine activation as 2x sin(2xx). We do not use any ActNorm or BatchNorm layers. For the Lipschitz coefficient, we 0.9 and the iteration error bound for spectral normalization 10-3
For the settings of scalable algorithms, we use brute-force computation of the log-determinant term for POWER and GAS datasets and use the same estimation settings as [Shen et al. (2019) for HEPMASS, MINIBOONE and BSDS300 datasets. In particular, for the estimation settings, always exactly compute 2 terms in training process and 20 terms in testing process for the log- determinant series. We use geometric distribution of p 0.5 for the distribution p(N) for the ogg-determinatt term. We use single sample of (n, v) for the log- log-determinant estimators for both training and testing.
We train each expeirment and 6 days for ImpFlows. For 20-block ImpFlow, we train our model for about weeks. However, find that the 20-block ImpFlow will overfit the training dastaset for MINIBOONE because this dataset quite small, so we use the early-stopping technique.
single NVIDIA GeForce GTX 2080Ti for about 4 days for ResFlows
D.5 DENSITY MODELING ON IMAGE DATASETS
For the CIFAR1O dataset, we follow the same settings and architectures as Chen al. (2019). In particular, every convolutional residual block is
LipSwish + 3 x Conv + LipSwish + 1 x Conv + LipSwish - 3 x Conv.
The total architecture is
Image Logit - le x ConvBlock - [Squeeze + k x ConvBlock) x 2,
where ConvBlock is i- ResBBlock for ResFlows and ImpBlock for ImpBlock, and k = 4 for ResFlows and 2 for ImpFlows. And the first Conv Block does not have LipSwish as pre-activation, followed as Chen et al. (2019). We use ActNorm2d after every ConvBlock. We do not use the FC layers (Chen et al. 2019). We use hidden channels as 512. We use batch size of 64 and the Adam optimizer of learning rate 10-3 The iteration error bound for spectral normalization is 10-3 We use a = 0.05 for CIFAR10.
For the settings of our scalable algorithms, use the same Chen et al. (2019) for the log- determinant terms. In particular, we always exactly compute 10 terms in training process and 20 terms in testing process for the fog-determinant series. We possion distribution for the distri- bution p(N) for the log-determinant term. We use single sample of (n, v) for the log-determinant estimators for both training and testing.
We train each ResFlow on a single NVIDIA GeForce GTX 2080Ti and each ImpFlow on two cards NVIDIA GeForce GTX 2080Ti for about days for ResFlows and 8 days for ImpFlows. Although the amount of parameters are the same, ImpFlows need more GPU memory due to the implementa- tion of PyTorch for the backward pass of implicit function.
For the CelebA dataset, we use exactly the same settings as the final version of ResFlows in Chen al. (2019), except that we use the Sine activation of the form 2x sin(2
E IMPFLOW SAMPLES
23

---

Published as a conference paper at ICLR 2021
Figure 6: Qualitative samples on 5bit x44 CelebA by ImpFlow, with a temperature of 0. (ingmaa & Dhariwal 2018)
24


Reviews:
[{"review_id": "pqwZD5Dojuj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\n\nThe authors propose ImpFlow, an implicit normalizing flow defined around solving the equation x + g_x(x) = z + g_z(z) where g_x and g_z have Lipschitz < 1.\n\nSolving z given x, or x given z, gives the forward and inverse passes of ImpFlow. Both directions require solving a root finding problem, hence this model is implicit in both directions, contrasting with prior works (e.g. ResFlow) where at least one direction is always explicit.\n\nImpFlow is equivalently a composition of ResFlow and the inverse of a ResFlow. \n\nIn the context of expressiveness, the forward pass a ResFlow is (1 + L)-Lipschitz while, importantly, the inverse is (1 / (1 - L))-Lipschitz, for some L < 1. Since ImpFlow makes use of the inverse of ResFlows in its construction, it can model arbitrary Lipschitz transformations. \n\nIn the context of likelihood evaluation and training, the estimators from ResFlows can be used and the main difference is the use of implicit function theorem to differentiate through the root finding procedure. Authors claim the total compute cost is comparable, though solving the root finding problem does seem to introduce some overhead, going from 3.189s to 4.462s (~40% increase) for each iteration of training on CIFAR-10.\n\nStrong points:\n\nThe use of the inverse of ResFlows in both directions of a normalizing flow is interesting, and result of having higher Lipschitz is convincing. \n\nIn addition to the well-written series of lemmas and theorems, the additional capacity of ImpFlow over ResFlow is nicely illustrated on a 1D example.\n\nExperiments indicate ImpFlow is slightly more performant in log-likelihood than ResFlows for the same number of parameters.\n\nWeak points:\n\nIs being implicit really useful? Outside of the Lipschitz requirement in this particular formulation of ImpFlows using the inverse of ResFlows, it's not clear whether having implicit formulations in both directions of a normalizing flow is advantageous. In addition to the use of a general title and method name advocating for implicit formulations, the authors do claim that \"bijections with explicit forward mapping only covers a fraction of the broad class of invertible functions suggested by the first requirement, which may limit the model capacity\" in the introduction. This ideally should be accompanied by a reference or rephrased to be a conjecture.\n\nThe experiments do not compare to other existing normalizing flows outside of ResFlows. The numbers reported in Table 2 (for tabular experiments) are worse than the baselines reported in Table 1 of Papamakarios et al. \"Masked Autoregressive Flow for Density Estimation\" while the results on CIFAR10 (Table 3) are closer to Real NVP (2016) than models like Glow or FFJORD (2018). This is likely simply due to the use of smaller models than those used previously, but if so, perhaps the authors could comment a bit more on the problems for scalability of ImpFlows. A ~40% additional time cost is mentioned once when compared to ResFlows, and an extra usage of GPU memory cost is only mentioned in the Appendix. The other parts of the paper are rather well done, but I'd liked to have seen explicit diagnostics of ImpFlows. For instance: \n  - How many function evaluations were used for solving the implicit formulation? \n  - Are results sensitive to the value of epsilon (for convergence of fixed point) hyperparameters? (A problem for implicit formulations is controlling the numerical error. Is this a problem for ImpFlows?)\n  - Since the Lipschitz of ResFlows grows exponentially, does the empirical difference between ImpFlow and ResFlow shrink as model sizes increase?\n\nSince the differences between ResFlow and ImpFlow are rather small (e.g. 0.01~0.02 bits/dim on CIFAR10), it'd be good to have some standard deviation across random seeds for all experiments.\n\nAdditional comments:\n\nThe citation under Lemma 1, \"Chen et al. (2019, Lemma 2)\" should probably be \"Behrmann et al. (2019, Lemma 2)\". (I assume the authors meant to refer to the bi-Lipschitz bounds derived in Behrmann et al.?)\n\nPost-discussion:\n\nI thank the authors for responding swiftly to all of my main concerns and for providing additional experimental results. I'm happy that authors have promised standard deviations and comparisons to ResFlow in the newly updated experiments for the camera-ready version, and I have adjusted my rating accordingly. I think this paper makes a very solid contribution to the normalizing flows literature.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cgpJ1CZq18A", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper introduces an invertible transformation implicitly via the roots of an equation. As a result, they claim that this implicit transformation is in theory a superset of Residual Flows. Further, they show improved performance empirically in NLL on CIFAR10 in varying settings.\n\nReview:\nThe paper introduces a novel an interesting method to parametrize invertible transformations via an implicit. The method in its current form can actually be interpreted as two Residual Blocks, but where the inverse parametrisation of the second block is used in the forward direction (as the authors show in Eq. 6). Nevertheless, I find the proposed perspective interesting and a promising future direction. The authors give detailed proofs of the flexibility of their method. The authors also propose a more memory efficient solution for training.\n\nWeaknesses: Experimentally there are some unanswered questions.\n- As Implicit flows require iterative solutions during training, they will most likely be slower. Even if ImpFlows are slower, it is really necessary to clearly highlight this difference in computation time. Further, ImpFlows are probably even faster during sampling as half of the Residual Blocks are now aligned with the sampling direction.\n- The new gradient computation from Eq (16) is only very briefly discussed in the paper. Since the authors claim gains in memory efficiency, these results need to be backed up empirically.\n- Thm 1 can also be derived from the perspective of two stacked residual blocks (second inverted). It would help readers to already introduce that perspective here. \n\nIf the authors address the points raised in Weaknesses, I will consider raising my score. \n\nMinor comments:\nTypos. The paper has some typos and in many cases a spell checker could correct these. For instance: Inverible (Sec 3.2) Esitamated (algorithm 1.).\nRephrasing (sec. 4.2). \"Note that we define R_1 = R\" -> \"By definition of Eq. 4 and Eq. 5, R_1 = R\"\nThe overload in Algorithm 1 is somewhat difficult to parse, consider changing \"g(z) = F (z, x; theta)\"\n\nAfter rebuttal:\nI am satisfied with the reply of the authors and I have raised my score to 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "S8iQLY4TX8Q", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThe authors concerns the question of how expressive invertible functions can be constructed. Their ansatz is the defining an invertible layer implicitly, using the root of an equation. While this approach is more general, they employ residual flows (ResFlows) to formulate a particular realisation of such an equation, calling the model ImpFlow. They show that the resulting function space is strictly richer than that of ResFlows. They further demonstrate how ImpFlows can be trained and evaluated. Empirically, ImpFlows outperform ResFlows on all considered tasks.\n\n# Strong and weak points\n## Pros:\n- Implicit functions are a new way to formulate invertible functions\n- The proposed formulation using ResFlows is clearly presented\n- ImpFlows are strictly more expressive than ResFlows, allowing arbitrary Lipschitz constants\n- Experiments highlight the improvements\n- In general: very concise and well-guiding writing\n\n## Cons:\n- Implicit functions are more expensive to evaluate than explicit functions (here, +50% in execution time)\n\n# Recommendation\nThe development of expressive invertible functions is key in applications that involve invertible functions, like density estimation using normalising flows. This paper proposes a novel framework to formulate such invertible expressive functions implicitly. The results are not game-changing, but consistently outperform its closest relative. Together, this is a solid work that should clearly be accepted.\n\n# Questions\n- I would like to see some analysis on how a Lipschitz function influences the Lipschitzness of the inverse. In particular, in Section 6 you could mention the relation between the \"Lipschitz coefficient\" c and the achievable Lipschitz constants of an ImpFlow layer. I naively would guess the bound $L < (1 + c) / (1 - c)$.\n- Can you give an intuition how large the improvements on the density estimation datasets are? I checked [paperswithcode](https://paperswithcode.com/sota/density-estimation-on-uci-power) on the POWER dataset and the models are clearly outperformed by other methods. To be explicit: I don't require that every new architecture has to beat the state of the art in all possible tasks, especially when it comes with a fresh idea and thorough theory. But can you give your best guess about why other approaches achieve significantly better results?\n- When comparing ImpFlow and ResFlow, I suggest that you should additionally have a variant of ResFlow with the same *execution time* in addition to *number of parameters* as a corresponding ImpFlow. I think the tradeoff involved is not only between quality and the number of parameters to store, but the also execution time of a model.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4aO1sjSt7N", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThis work is about a new architecture for normalizing flows inspired by implicit neural networks. In particular, the authors show that a specific implicit neural network build from residual blocks defines a bijective map. From this insight, they show how to efficiently train such architecture by estimating the log Jacobian and solving the inverse problem related to the implicit architecture. On top of this achievement, the authors thoroughly study the attractiveness of their architecture compared to i-res-flow. They demonstrate analytically and empirically that implicit normalizing flows are strictly more expressive than i-res-flow.\n\n# Major comments\n## Pros:\nOverall the paper is well written and pleasant to read. The idea is novel and is well introduced. Moreover, the paper provides a theoretical justification for the expressivity gain made by the newly introduced architecture with respect to I-res-flow. The experimental results are good although not defining a new standard in density estimation tasks. \n\n## Cons:\n1) As I said the paper is well written overall. However, I took quite some time to fully understand section 4 (in particular 4.2), I think this section should be clarified. After taking the time, everything appears convincing but I think the flow could be improved. I checked the proofs in 4.2 and everything seems correct, great job! You should mention that everything is proved in the appendix, I was not sure it was the case the first time I read it.\n2) Regarding the experiments I have two concerns. First, you provide only results about one specific value of L without stating how you chose it. It feels like you selected it to show the performance gain brought by your method, it may be fine but you should at least mention it clearly. Else you should perform a full architecture search both for resflow and for impflow and show the best results of each architecture. My second concern is about table 2, I think you should mention other sota NF architectures (such as the ones based on monotonic transformation) and maybe discuss a possible explanation for the large gap.\n\n## What could be done to address my comments\n1)  You could maybe directly state that $\\mathcal{R}_2 \\subset_\\neq \\mathcal{F}_2 = \\mathcal{I}$ instead of corollary 1.\n2) Clarify how you selected the number of steps. I saw you mention that you copy the experimental setting of Chen et al. 2019. However, they use 200 steps for the 2D grid and do not provide results for the UCI dataset (or I missed it maybe).\n\n# Minor comments\np3: \"inverible\".\n4.1: $J_f$ inconsistent notation wrt (1).\ndef 2: missing . at the end of the equation.\n\"2-norm\": do you mean $L_2$ norm?\n\"in 1-D input\" -> \"In the 1D input\"\nD.4: \"Tablular\"", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a clear accept. Solid and timely work extending normalizing flows to implicitly defined mappings. Convincing presentation. Supported by all four reviewers. Best paper in my batch. Has the potential to spark further developments in the field. I recommend to feature this paper as a spotlight.
Decision: Accept (Spotlight)