Invertible Monotone Operators for Normalizing Flows
Authors: Byeongkeun Ahn
Chiyoon Kim
Youngjoon Hong
Hyunwoo J. Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: Normalizing flows
invertible neural networks
monotone operators
deep generative models

Abstract: Normalizing flows model probability distributions by learning invertible transformations that transfer a simple distribution into complex distributions. Since the architecture of ResNet-based normalizing flows is more flexible than that of coupling-based models, ResNet-based normalizing flows have been widely studied in recent years. Despite their architectural flexibility, it is well-known that the current ResNet-based models suffer from constrained Lipschitz constants. In this paper, we propose the monotone formulation to overcome the issue of the Lipschitz constants using monotone operators and provide an in-depth theoretical analysis. Furthermore, we construct an activation function called Concatenated Pila (CPila) to improve gradient flow. The resulting model, Monotone Flows, exhibits an excellent performance on multiple density estimation benchmarks (MNIST, CIFAR-10, ImageNet32, ImageNet64). Code is available at https://github.com/mlvlab/MonotoneFlows.

Invertible Monotone Operators for Normalizing Flows
Byeongkeun Ahn', Chiyoon Kim', Youngjoon Hong"", Hyunwoo J. Kim
Korea University', Sungkyunkwan University² (byeongkeunahn, kimchiyoon, hyunwoo kim) @korea. kr hongy @skku. edu
Abstract
Normalizing flows model probability distributions by learning invertible transformations that transfer simple distribution into complex distributions. Since the architecture of ResNet-based normalizing flows more flexible than that of coupling-basedd models, ResNet-based normalizing flows have been widely studied in recent years. Despite their architectural flexibility, it is well- known that the current ResNet- based models suffer from constrained Lipschitz constants. In this paper. we propose the monotone formulation overcome the issue of the Lipschitz constants using monotone operators and provide an in-depth theoretical analysis. Furthermore, we construct an activation function called Concatenated Pila (CPila) to improve gradient flow. The resulting model, Monotone Flows, exhibits an excellent performance on multiple density estimation benchmarks (MNIST, CIFAR- 10, ImageNet32, ImageNet64). Code is available at https://githhb....
https: /ttg//gtttubbbte/ wttp://wwwwwoooooooeoo....oo.
Introduction
Normalizing flows 1 2 are method for constructing complex distributions by transforming probability density through a series of invertible transformations. Normalizing flows are trained using plain ligliikel function, and they capable of exact density evaluation and efficient sampling. Applications include image generation 2 3
4 7 8) image super-resolution I0,11 image audio synthesis [14 15], anomaly detection [16 17I. and computational
noise modelling 12 13
physics [8
20
21 highlighting the importance of developing expressive normalizing flows.
However, major hurdle to designing normalizing flow architectures is that not only should each transformation invertible but also its Jacobian determinant needs to tractable 22 23 Despite
multiple efforts to satisfy the aforementioned constraints 2 4 24 25 the expressive power of the normalizing flow transformations stays limited due to the adoption specialized architectures (5).
In this regard, ResNet-based normalizing flows 5 7 are compelling alternative since they do not impose any structural restrictions. However, they keep the Lipschitz constant of the residual branch less than 1 in order to ensure invertibility. As result, the Lipschitz constant of each residual block is less than 2. This causes a serious issue on the expressive power since large Lipschitz constant is often required in converting between distributions. A recent work [26] tackled this problem, but as we show this paper, their method is equivalent to simple rescaling of ResNet-basee normalizing flows, which has similar expressive power.
In this work, we propose the approach of monotone operators alleviate the Lipschitz constraint. More precisely, in order to parameterize monotone operators, we use the well-known fact from monotone operator theory: the Cayley operator of maximally monotone operators (MMOs) is function. Lipschitz.uunction The residual branch of ResNet-based normalizing flows can be used to parameterize the Cayley operators.
"Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Rotate -45"
Rotate +450
Figure 1: The duality between 1- 1-Lipschitz operators (left) and monotone operators (right) in ID.
In summary, contributions are as follows:
We propose Monotone Flows, which greatly loosens the Lipschitz constraint while retaining invertibility and architectural flexibility.
We derive the monotone formulation, monotone operator-based normalizing flow by parametrizing the Cayley operator, and provide efficient training and inference schemes.
We propose a new activation function called Pila and its variant Concatenated Pila (CPila) alleviate the saturated gradient problem.
We theoretically analyze the expressive power of the monotone formulation and related models 66 26). and prove our formulation outperforms the other models in practice.
Monotone Flows consistently outperform comparable baseline normalizing flows on multiple image density estimation benchmarks as well as on 2D toy datasets. In addition, ablation studies demonstrate the effectiveness of the proposed methods.
2 Preliminaries
The monotone formulation is a normalizing flow that parametrizes monotone operators in terms of Cayley operators. The parametrization, which must be I-Lipschitz, and the methods for log determinant computation,
borrowed from ResNet-based normalizing flows. Also, the resulting model is an implicit neural network involving inverse functions. Here, we briefly review these topics.
Normalizing flows [ explicitly model probability distributions by transforming tractable source distribution, e.g., Gaussian, into a target distribution with differentiable invertible transform F R. > R Suppose F maps a sample space X to latent space Z. Given source distribution pz(2), the change-of- variiables theorem yields the log- lieeliidl any æ E X:
log (x) logPz(=) + log det Jrl.
(I)
where z = F(x) and JFF = 0z/Ox is the Jacobian. Normalizing flows are beneficial since the evaluation of (I) yields the exact density (x). Furthermore, target data efficiently sampled by first sampling 2 ~ pz pz(=) and then mapping 2 to X by = F (2). When F consists of multiple transformations applied in succession, each transformation is called a flow.
px(z) can be
ResNet-based normalizing flows have been initially proposed in i-ResNets SS which are built with G(x)). By requiring G(x) to be a contraction map, they ensured the invertibility R(x) with the Banach fixed point theorem. The contractiveness of G(xr) is implemented by spectral normalization (27) and I-Lipschitz activation functions. However, they use a biased truncation the log Taylor series for computing the log determinant. Residual Flows [6] address this issue with unbiased estimator based on randomized truncation. They also introduce new activation function LipSwish to avoid the gradient saturation problem. i-DenseNets (7) improve Residual Flows with DenseNet variant accompanying learnable concatenation coefficients. They also introduce the Concatenated LipSwish activation function to further improve the gradient flow.
variant of the residual blocks R(x))
Implicit neural networks map an input to an output through an implicit equation F(x, y) = 0, in contrast with explicit models of the form y = F(x)). The main advantage of implicit mappings is their ability to express a richer family of functions under the same parameter budget 28). Implicit Flows 26 propose to use invertible implicit neural networks normalizing flows and study an example where residual blocks and the inverse of residual blocks are stacked in an alternating fashion.

---

Monotone operator F: R" + R is an operator that satisfies (u v, y) 2 0 for all z e R" and all F(x), U F(y). This definition generalizes monotonic functions in R. to R The resolvent of a monotone operator F is defined as R = (Td + F)-1, where Id is the identity operator. The Cayley operator of F defined as Cp = 2 Td. well-known that the Cayley operators of monotone operators are exactly 1-Lipschitz operators. The Cayley operator can be interpreted as 45-degrees rotation (up to constant scaling) of the corresponding monotone operator in R" x IR; see Figure [ for ID illustration. Minty surjectivity theorem ([29) and [30 Theorem 21.1J) shows that the Cayley operators of maximally monotone operators are exactly I-Lipschitz functions, which are defined at every point. We formally state this correspondence in Theorem 2 See [1 for survey and 33 for a comprehensive treatment on monotone operators.
3 Monotone Flows
In this section, we propose a novel approach, the monotone formulation, and describe the training and inference procedure. Moreover, we introduce a new activation function Concatenated Pila (CPila).
3.1 Motivation from monotone operator theory
ResNet-based normalizing flows consist of blocks of the form R(x) = x + G(z), where G(x) is a parametrized neural network. If G(x)) is contraction (the Lipschitz constant Lip(G) 1), the Banach fixed point theorem shows that R(x) is invertible. In this case, Lip( Rt) is strictly less than 2, resulting in severe limitation on the expressive power of the neural network R since large Lipschitz constant is required in general to transform the source distribution into the target distribution.
However, as shown in the next example, G(x) need not be a contraction for R(z) to be invertible.
Example 1. Consider (a))) 5a. Then, Rí (1)) = x + Gi (I:) 6a is invertible even though G1 (2) not a contraction.
Hence, when considering a relaxed condition and parametrization, is possible to enhance the expressive power to include a larger class of functions as in Example [ For this, we introduce monotone operators which have been widely studied in the fields of functional analysis and partial differential equations. We first recall various definitions of the monotonicity of operators in R":
Definition 1. Consider an operator F: 12" + R
(i) F is monotone for all æ, y E R", (F(x) Fly), x-0))--900 4) 2 O.
(ii) F is strictly monotone for all x, E RR (x + v). (F(x) Fly), - 0.
(ii) F is r-strongly monotone (1) > 0) if for all x, E R", (F(x)- F 2 n - v
(iv) F is maximally monotone if F is monotone and not proper subset of any monotone operator.
It is noteworthy that monotonicity does not guarantee injectivity nor surjectivity, and striet monotonicity guarantees injectivity but not surjectivity. However, we have the following theorem:
Theorem 1. An r-strongly monotone continuous function F: R - IR" is invertible for any v > O.
Proof. See Corollary 20.28 and Proposition 22.11 in (30].
Theorem I casily shows that R1 (z) in Example is invertible, Moreover, if G(x) is L-Lipschitz (L 1). H(r) is (1-14 --strongyy monotone (see Appendix (..1) and thus invertible by Theorem
As in Example I the ResNet-based formulation of normalizing flows does not include all possible invertible and monotone functions. To address this issue, directly parameterize monotone operators in terms of their Cayley operators. We first formally state the rationale behind our construction.
Theorem 2. Let F: IR" + R" be an operator and CF = 2(ld + F) ld be its Cayley operator.
Then the followings hold.
(i) F is monotone + Cr is I-Lipschitz.
(ii) F is maximally monotone <> CF is I-Lipschitz and dom Cp = R
Proof. See Proposition 23.8 and Proposition 4.4 in 30
We adopt the notation F(x) = {yl (ar v) E F} since F' is an operator. However, it suffices to consider single-valued functions defined at every point in our development of theory, in which case the definition reduces to (F(x) F(w), x y) 2 0.
3

---

Theorem 2 shows that each 1 1-Lipschitz function characterizes unique MMO since the Lipschitz continuity of Cp implies Cp is single-valued. Hence, the Lipschitz continuous parametrizations of the residual branch in ResNet-based models fit naturally for specifying Op.
We revisit Example 1 and apply Theorem 2 The Cayley operator of Ri (zx 6x is given by CR,, (x) = 2(1d R1) (T) - = (5/7)2, which is indeed I-Lipschitz in accord with Theorem 2
In general, however, MMOs can be multivalued or undefined at some points. However, when the Cayley operator Cp is an L-Lipschitz function with < 1, then F is - L) + L)-strongly monotone and (1 L) (1 L)-Lipschitz continuous; Appendix (A.8 for complete derivation. Theorem then implies F is invertible. Hence, we use L-Lipschitz functions for Cayley operators.
3.2 Deseription of the monotone formulation
Definition 2. Let G R + R be a Lipschitz contiiits function with Lipschitz constant L 1.
The monotone formulation of G is defined as the following function F: E + R":
ld + G
F(x)=
(x) -
where Id denotes the identity function.
Here, well-defined since (Td + G2/2 has unique inverse by the Banach fixed point theorem [.] The definition of F is direct application of Theorem 2 On the other hand, from the Cayley operator identity CF -Cr- = O, we deduce that F=1 has the Cayley operator -G. Hence, y = F(x).
Id G
y,
which is well-defined by analogous argument. Thus arrive the following theorem:
Theorem 3. For each G with Lip(G) < 1, there is unique invertible function F as in (2).
Note that by rearranging or we obtain the implicit equation connecting x and y::
x
G(x + y)-
(4)
To parametrize G. we use the architecture proposed in i- N-DenseNets [7], but replace the activation function with a new function called Concatenated Pila (CPila), which we introduce later the paper.
Remark. While we parametrize each layer as a monotone operator, the resulting functions need not be monotone. For example, consider the x 7/3 counterclockwise rotation operator Rx/3 /3 in R Although R is monotone, the composition of / with itself is not monotone. This illustrates that monotone functions are not closed under composition, hinting neural networks generated by the composition of invertible monotone operators can represent a much broader class of invertible functions than just monotone operators. Indeed, functions of the monotone formulation sup-universal diffeomorphism approximators since the monotone formulation includes the "nearly-Id" functions (R, in section 4) whose composition suffices for sup-universal diffcomorphism approximation [32)
3.3 Training and inference
Training and inference largely follow previous approaches developed in 66 26 except for minor modifications adapt them our monotone formulation. For the function of Definition 2 implicit differentiation gives
log det Jp = tr |oog[I - JG) - log(! + Ja)],
(5)
where = = (w(ww) at w = (3 (z); see Appendix A.3 for derivation. Now, we construct unbiased estimator of the log determinant using the Hutchinson trace estimator and the unbiased Russian roulette estimator, following the approach [6]. Combining
[[] and (5).
(-1) (-1)6+1 of JJ Σ
n
1o x (x) = 1zg Em-px(n)
(6)
P(N 2 k)
We omit the absolute value sign since the strong monotonicity of F' implies det Jr > 0.

---

(b)
1.4
1.2
0,8
0.6
-5 3
2
1
2
3
4
5
5
3
-1
1
2
3
4
5
Figure 2: Graphical illustrations of Pila and CPila. (a) The graph of Pila (red) and its first (green) and second derivatives (blue) with 5. (b) The speed of the curve of CPila (red) with k 5 and CLipSwish (green) with A = 1.
where px (n) is a distribution with support over the positive integers. Note that the derivative of the
log-likelihood (6) can be evaluated by the memory-efficien Neumann gradient estimator as in (6].
Training consists of forward and backward propagations using 2 and (6). For forward propagation,
F can be evaluated with fixed-point iterations since is a contraction mapping while suffices to
( (x) backpropagate through (Id G)-1 since F(x) 2
(Id -2-) For
this, we follow the approach of [26]. Let G be parametrized by 8. Denote the training objective ( as
l and write (Id + G) Gu) (where 2x). Implicit differentiation yields
de =
de (I Ja)
de
de Ow
OG
(I Ja)
Ou dw
08
=
80
where G and JG are evaluated at (Id + G)); see Appendix A.5 for derivation. Since the
linear map Jg IR" + R a contraction mapping. the vector-matris inverse product 7 can be evaluated by fixed-point iteration. For this, we adopt the secant method, but when convergence is not improved for 10 consecutive iterations, the Krasnoselskii-Man
iteration is used instead.
Inference consists of forward propagation through 3 which is almost the same as the forward propagation in training except that the sign of G is the opposite.
3.4 Concatenated positive identity 1-Lipschitz activation function
The LipSwish activation function proposed in 6 is based on the premise that having a non-vanishing second-order gradient would improve training. However, LipSwish still suffers from the vanishing gradient problem since its first-order derivative stays around 0.5 the neighborhood of x = 0 where most of the preactivations would reside. Concatenated LipSwish (CLipSwish) [ improves upon LipSwish by concatenating LipSwish(z) and LipSwish(-2a r)), but a similar issue still occurs as shown in Figure (b).
In this regard, we propose Pila (positive identity I-Lipschitz activation) function, which makes the gradient flow equal to the identity function converges to zero as v + -oo while matching the identity function at æ 0 up to the third derivative. The Pila function is described as follows:
0 during backpropagation. When 0, Pila
k2 {7
I
if 2 0,
Pila(r)
(8)
ko + x kk if <
Here, & > 0 is treated either as a fixed hyperparameter or a learnable parameter. In all of our experiments, we set and do not learn k. Note that Lip(Pila) 1 for any 0.
As in 77 we introduce a concatenated version of Pila, which we call Concatenated Pila (CPila):
CPila(x) = a. [Pila[z - a2), Pila(-alaa))) - where a₁ = 1/1.06 and a2 = 0.2. By rescaling and translating with and a2 we fabricate the CPila function be I-Lipschitz. Figure 2 (b) describes the speed of the curve") which shows CPila
has a speed closer to around x = 0 compared to the speed of CLipSwish.
'speed of the time t: [v(t) / (1) (t)²
5

---

1+L
ML
1-1
It
1+L T+L L
1-L
1+L
T
Figure 3: Comparison of R2 (blue), Iz (green), and Mr (red). The figure depicts the possible ranges 21 I2 for each function class, assuming the domain
of F(21) F(x22) with respect and the codomain of F are both RR.. For visualization, we choose 0.8 and fix p as a unit vector pointing in the ty direction.
4 Expressive power
In this section, we analyze the expressive power of the monotone formulation and related models. We prove that the monotone formulation is superior the residual formulation [6] and the implicit formulation of Lu et al. [26] when the Lipschitz constant L of the G-network is less than unity. In practice, since L should not be close to unity, the monotone formulation is better choice than the residual formulation or the implicit formulation of Lu et al.
We start with formal definition of the function classes of each model. Definition 3. For 0 < L 1, we define the following function classes:
L-Lipschitz functions
GD G E (R",", RR" (R"")LLP(G)) }}
Residual formulation
R = G|
Inverse residual formulation
I₁
ML = ((( Id 9L
Monotone formulation
Here, (RR R") denotes the set of twice continuously differentiable funccitoon
5
Note that the implicit formulation of Lu al. [26] defines one block as the composition of two functions Fz11 F1 for any FI, E RR. We perform our analysis only on the second layers, which corresponds to the comparison between RR and In. For convenience, we use the following notation.
Definition 4. Let A be one of 91. RL, In, or ML, and > 0. Then,
aA= {aF) A}, R+ {3FIVB E Rt FE,FFAA
We now examine the relations between the function spaces. Theorem 4. For < I the following holds.
1 + L
(i) I, 1- T,2 Rr, (i)) Mr
I2 R
(ii) Rr Mi, (iv) & Mr.
Proof. See Appendix for a complete proof.
Figure 3 shows the ranges of q F(x1) F((2)) with respect to p 21 w2 for R1, Ir, and Mr. clearly demonstrating the relations (i)-(iv) of Theorem 4
Theorem 4 shows the inverse residual formulation In only express functions that constant multiple of the functions expressed by the residual formulation R2. Hence, the inverse residual formulation has advantage over the residual formulation in this respect. On the other hand, the monotone formulation Mr can express functions that are not expressible by any constant multiple of the functions in R2, because (1/(1 (1 L?) L whenever L 1.
When L can approach unity arbitrarily close, the monotone formulation has the same expressive power as the residual formulation or the inverse residual formulation. However, this is not possible
'e require continuous second-order derivatives the gradient of the log Jacobian determinant is of second-order. However, the results still hold when weaker function class is considered (e.g.. ... C² RR))..

---

in practice because the variance of the log determinant estimator (6) diverges to infinity when L is greater than specific threshold less than unity that depends on the distribution Pv (v(n) [3.. Since the log determinant estimator (6) is a linear combination of the log determinant estimator of the same G-network's & residual formulation, (6) has a finite variance whenever the corresponding residual formulation's s estimator possesses a finite variance. Suppose the constraints, in practice, bound the Lipschitz constant of the G-networks to be at most Lmax < 1. Then, the residual formulation gives rise I
to IR RR Lmaa whereas the monotone formulation provides IR Mi
R
22max
? RR RL
1+-
An analogous argument holds for RT Hence, the analysis shows that the monotone formulation outperforms the other formulations in terms of the expressive power.
Related work
Normalizing flows. Normalizing flows require each layer be invertible and the Jacobian determinant of each layer to be tractable, which have spurred multiple approaches for architecture design. Restricted Jacobian models introduce specific structures to the Jacobian to make the Jacobian determinants tractable. They include determinant identity-based models [1] [25 which use the
Weinstein Aronszajn identity with reduced intermediate layer dimensions, and coupling- based 3 34 35] which split the input into two parts and transform one part conditioned on another. However, the restrictions on the Jacobian limit their expressivity. In contrast, free-form Jacobian models are not subject to the same constraints and include ResNet-based models (discussed in section 2 and Neural ODEs. Neural ODEs (36] parametrize the rate of change in ODEs with neural networks. They ensure invertibility through the existence and uniqueness theorem of ODEs. However, they have relatively high time complexity since they require solving ODEs numerically on every iteration.
models 2
So far, the normalizing flows we have discussed are non-autoregressive models. In contrast, autoregressive models (3733) 39 decompose probability distributions as conditional distributions depending only on the previous values. While autoregressive models typically achieve higher log- likelihoods, they are orders of magnitudes slower to train or sample [4]. Therefore, we focus on non-autoregressive models in our paper.
Monotone operators in neural networks. [40] pioneers monotone operator neural networks to assure the convergence of fixed-point iterations in implicit depth networks. They build linear monotone operators by parametrizing the symmetric and antisymmetric parts separately. Positive defininteness enforced only on the symmetrie part. One shortcoming that the construction limited linear monotone operators, which our work addresses using nonlinear Cayley operators. first neural network model of nonlinear MMOs by parametrizing the Cayley operators. One key difference is that the 1-Lipschitz constraint on Cayley operators is enforced through a penalty loss, which may fail to control the Lipschitz constant globally [5) Our method systematically ensures Lipschitzness by the use of spectral normalization [27 and 1 1-Lipschitz activations, following [5] 6 7 Also, their work does not consider normalizing flows. To the best of our knowledge, our work proposes the first normalizing flow based on monotone operators.
[4] proposes the
Experiments
We start by validating the modeling capacity of our model with a ID toy function. We then evaluate Monotone Flows on density estimation of 2D toy and image datasets and conduct ablation studies. In all experiments, we parameterize the G-network in Definition 2 using i-DenseNets [7] with our activation function CPila (except for the ID toy experiments that use CReLU and for part of the ablation study where we ablate CPila). The G-network consists of fully-connected layers for ID and 2D toy tasks and convolutional layers for image tasks. We use the Poisson distribution for px (n) whenever the estimator 6 is used. We use Adam 42 for all experiments.
6.1 Toy 1D experiments
Figure illustrates the expressive power of the monotone formulation with 1D toy example. We fit a ID function that consists of four consecutive step-like shapes designed to exhibit high Lipschitz constant. We consider four different 2-layer networks, shown in Figure 4 Experimental details
7

---

3
3
3
3
2
2
2
1
1
1
0
0
0
-2 -2-1- 0 1
-2
2
-2-1 0 2
2
1
(a) RBs w/o LS
2 RBs LS
(c) RB IRB w/ L.
(d) MBs w/ LS
(MSE: 1.6 X 10
(MSE: 5.8 X 10-3)
(MSE: 4.5 10-3)
(MSE: 7.6 X 10-6)
Figure 4: Comparison of 2 RBs without learnable scaling, RBs, RB followed by IRB, and 2 MBs. All experiments except (a) performed with learnable scaling (LSS). Blue and red lines represent the target function and the approximation by neural networks, respectively.
Sumple data
i-DenseNet
Monotone Flow
Table I: Toy density modelling results in nats. We display the average of the test loss for the last 20 tests at checkpoints (iterations 48 1800, 48200, 50000) for a single run.
Rings
Data
i-DenseNet
Monotone Flow
2 Spirals Checkerbonrd
2 Spirals 2.729
2.658
Gaussians
2.840
2.840
Checkerboard
3.609
3,540
Circles
3,280
3.276
Moons
2.401
2.400
Pinwheel
2.343
2.333
Rings
2.884
2.665
Swissroll
2.680
2.676
Figure 5: 2D toy density modeling results (full results in Appendix D).
are discussed in Appendix C.T We denote residual block, inverse residual block, and monotone block by RB, IRB, and MB. Note that the learnable scaling (LS) is used only for Figure 4 (b)-(d). The G-networks in the experiments (a)-(d) have same structure, and hence the same number of parameters are used. Consequently, differences in the result solely come from the differences in the formulations. Figure 4 (a) and (b) demonstrate that the expressive power is improved when using R*R, R2 instead of R2 Figure (b) and (c) reveal that the expressive power of IR RR and R. II similar, in agreement with Theorem 4 Figure (d) shows that our monotone formulation R+ MM exceedingly outperforms the other networks.
6.2 Density estimation 2D toy data
We use the 2D toy datasets provided with the official source code i-DenseNets 7 Following 7 use ten flow blocks (sec Appendix C.2 details). Exact Jacobian evaluation is used instead of the stochastic estimation (6) as it is inexpensive for 2D data. We train the models for 50K epochs with the learnable concatenation of i-DenseNets enabled after 25K epochs. We use a batch size of 500.
Qualitative and quantitative results in Figure and Table show that Monotone Flows outperform i-DenseNets in almost all settings, especially on challenging datasets. One limitation of normalizing flows manifests in the qualitative results. For instance, in the dataset 'rings', the eircles cannot be closed because normalizing flows are required to preserve topology. Despite the inherent restriction, our model finds an excellent approximation.
6.3 Density estimation on images
We evaluate our method on MNIST 443 CIFAR-10 44], and the downscaled version of ImageNet in x2 and 64 64 45 We use for estimating the Jacobian determinant. We use uniform
'There are two versions of the downscaled ImageNet. The one used for evaluating normalizing flow models is the version packed in far. It has been removed from the official website but is still available through other routes (e.g., Academic Torrents).

---

Table 2: Density estimation results on images in bits- diss-per-iimesionn (bpd) with the number of parameters of each model. All numbers except for the last row are with uniform dequantization. VDQ: variational dequantization.
MNIST
CIFAR-10
ImageNet32
ImageNet64
Model
bpd 4
params
bpd
params
bpd 4
params
bpd 4
params
Real NVP
1.06
3.49
6.4M
4.28
46.0M
3.98
96.0M
Glow 4
1.05
3.35
44.2M
4.09
66.1M
3.81
111.1M
FFJORD
36
0.99
3.40
i-Res.Net
5
1.06
3,45
44.2M
Residual Flow
0
0.97
16.6M
3.28
25.2M
4.01
47.1M
3.76
53.3M
i-DenseNet
3.25
24.9M
3.98
47.0M
Monotone Flow
0.928
20.9M
3.215
24.9M
3.961
47.0M
3.734
48.9M
Monotone Flow + VDQ
3.062
46.9M
3.901
69.0M
(a) CIFAR-10 train data.
(b) Monotone Flows trained on CIFAR-IO.
(c) ImageNet32 train data.
(d) Monotone Flows trained on ImageNet32.
Figure 6: Train data and generated samples of CIFAR-10 and ImageNet32.
dequantization for both training and testing. For MNIST and CIFAR-10, 10, the learnable concatenation enabled after 25 epochs; for ImageNet32 and ImageNet64, is enabled from start. We train for 100, 1,000, 20, 20 epochs with batch sizes 64, 64, 256, 256 with learning rates 0.001, 0.001, 0.004, 0.004 for MNIST, CIFAR-I0, 10, ImageNet32, ImageNet64, respectively. We report single-seed results following 4 5 6 77. Details are in Appendix C.3 Samples from the trained models are displayed in
Figure
More samples can be found Appendix
Table shows our model outperforms baselines on all datasets considered, demonstrating the effectiveness our proposal. In concrete numbers, our model reduces 0.042 bpd MNIST, 0.035 bpd in CIFAR-I0. 100, 0.019 bpd in ImageNet32, and 0.023 bpd in ImageNet64 compared to baseline normalizing flow models.
Remark on comparison. Flow++ [34 introduced variational dequantization which improves the ligeeiiddoood reporting 3.08 and 3.86 bits/dim on CIFAR-10 and ImageNet32, respectively. With variational dequantization. Monotone Flows achieve 3.062 and 3.901 bits/dim on the same benchmarks. Also, there is a line of research that achieves better likelihood values by introducing latent variables in extra dimensions, at the cost of losing exact likelihood computation [46] 47 48 On the other hand, ScoreFlow score-matee matehin objective that lower bounds the log likelihood. We leave it for future research to adapt these methods Monotone Flows.
49 achieves better likelihood values by training with weighted
6.4 Ablation study
Our model has two new components: the monotone formulation and CPila activation function. We quantify their contributions by ablating either of the two components while leaving the other intact. When ablating the monotone formulation (row #2), we used the residual connections of 7 when ablating CPila (row #3), we used CLipSwish instead. Table compares the performance of i-DenseNets (row #1) and the ablated models on the CIFAR-10 10 dataset under the same experimental setup as section 6.3 Details
Table 3: Ablation study for density estimation on CIFAR-10. MFL: monotone formulation.
#
MFL.
CPila
bpd
1
x
x
3.252
2
x
3.243
3
V
3.229 x
in Appendix C.5 Results confirm that both components are
4
3.215

---

beneficial for achieving low bits-per-dimension (bpd). The monotone formulation provides a higher performance gain (0.028 bpd) than CPila (0.014 bpd).
7 Conclusion
We presented Monotone Flows, novel parametrization of normalizing flows based monotone operators (the monotone formulation) combined with a new activation function called Concatenated Pila (CPila). Our theoretical analysis elucidated why the monotone formulation is more expressive than the residual or inverse residual formulations. Experimental results demonstrated the effectiveness of the proposed method various density estimation benchmarks. We believe contribution is solid step towards improving the expressivity of normalizing flows.
Acknowledgments and Disclosure of F Funding
Our work supported by ICT Creative Consilience program (ITTP-202-2020-0001111)) supervised by the IITP, and Kakao Brain Corporation. The work of Y. Hong was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-222RRIA2C1093) and the Korea government (MSIT) (No. 2022R A4A3033571).
References
[1] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.
[2] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. In ICLR Workshop, 2015.
[3] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In ICLR, 2017.
[4] Durk P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NeurlPS, 2018.
[5] Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and förn-Henrik Jacobsen. Invertible residual networks. ICML, 2019.
[6] Ricky T. Q. Chen, Jens Behrmann, David K. Duvenaud, and Jöm--eerik Jacobsen. Residual flows for invertible generative modeling. In NeurlPS, 2019.
[7] Yura Perugachi-Diaz Jakub Tomezak, and Sandjai Bhulai. Invertible densenets with concatenated lipswish. In NewrlPS, 2021.
[8] Chin-Wei Huang, Ricky T. Q. Chen, Christos Tsirigotis, and Aaron Courville. Convex potential flows: Universal probability distributions with optimal transport and convex optimization. In ICLR, 2021.
[9] Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. Srflow: Learning the super-resolution space with normalizing flow. ECCV, 2020.
[10] Younghyun Jo, Sejong Yang, and Seon Joo Kim. Srflow-da: Super-resolution using normalizing flow with deep convolutional block. In CVPR Workshop, 2021.
[11] Andreas Lugmayr, Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte. Normalizing flow as a flexible fidelity objective for photo-realistic super-resolution. In WACV, 2022.
[12] AbdeIrahman Abdelhamed, Marcus Brubaker, and Michael Brown. Noise flow: Noise modeling with conditional normalizing flows. In ICCV, 2019.
[13] Yang Liu, Zhenyue Qin, Saeed Anwar, Pan Ji, Dongwoo Kim, Sabrina Cadwell, and Tom Gedeon. Invertible denoising network: A light solution for real noise removal. In CVPR, 2021.
10

---

[14] Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In NeurIPS, 2020.
[15] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP, 2019.
[16] Marco Rudolph, Tom Wehrbein, Bodo Rosenhahn, and Bastian Wandt. Fully convolutional cross-scale-flows for image-based defect detection. In WACV, 2022.
[17] Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In WACV, 2022.
[18] M. S. Albergo, G. Kanwar, and P. E. Shanahan. Flow-based generative models for markov chain monte carlo in lattice field theory. Phys. Rev. D, 100:034515, August 2019.
[19] Christina Gao, Stefan Höche, Joshua Isaacson, Claudius Krause, and Holger Schulz. Event generation with normalizing flows. Phys. Rev. D, 101:076002, April 2020.
[20] Benjamin Nachman and David Shih. Anomaly detection with density estimation. Phys. Rev. D, 101:075042, April 2020.
[21] Jonas Köhler, Andreas Krämer, and Frank Noe. Smooth normalizing flows. In NeurlPS, 2021.
[22] Ivan Kobyzev, Simon J. D. Prince, and Marcus A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Trans. Pattern Anal. Mach. Intell., 43(11):3964-39) 3979, 2021.
[23] George Papamakarios, Eric T. Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Mach. Learn. Res., 22:57:1-57:64, 2021.
[24] Emiel Hoogeboom, Victor Garcia Satorras, Jakub Tomezak, and Max Welling. The convolution exponential and generalized sylvester flows. In NeurlPS, 2020.
[25] You Lu and Bert Huang. Woodbury transformations for deep generative flows. In NeurlPS, 2020.
[26] Cheng Lu, Jianfei Chen, Chongxuan Li, Qiuhao Wang, and Jun Zhu. Implicit normalizing flows. In ICLR, 2021.
[27] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
[28] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In NeurIPS, 2019.
[29] George J. Minty. Monotone (nonlinear) operators in hilbert space. Duke Math J., 29333:34 September 1962.
[30] Heinz H. Bauschke and Patrick L. Combettes. Convex analysis and monotone operator theory in Hilbert spaces. Springer, Cham, 2011.
[31] Ernest K. Ryu and Stephen Boyd. A primer on monotone operator methods. Appl. Comput. Math., 15(1):3-45 43,3-433222 2016.
[32] Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling-based invertible neural networks are universal diffeomorphism approximators. In NeurIPS, 2020.
[33] Robert Comish, Anthony I.. Caterini, George Deligiannidis, and Amaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. In ICMI., 2020.
[34] Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-based generative models with variational dequantization nd architecture design. In ICMI., 2019.
11

---

[35] Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows, In NeurIPS, 2019.
[36] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In ICLR, 2019.
[37] Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In ICML, 2018.
[38] Durk P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In NeurlPS, 2016.
[39] George Papamakarios, Theo Pavlakou, and Iain Murray, Masked autoregressive flow for density estimation. In NewrIPS, 2017.
[40] Ezra Winston and J. Zico Kolter. Monotone operator equilibrium networks. In NeurIPS, 2020.
[41] Jean-Christophe Pesquet. Audrey Repetti, Matthieu Terris, and Yves Wiaux. Learning maximally monotone operators for image recovery. SIAM J. Imaging Sci., 140)):220-12377 2021.
[42] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[43] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Process. Mag., (996))141-144 2012.
[44] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
[45] Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML, 2016.
[46] Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. VFlow: More expressive generative flows with variational data augmentation. In ICML, 2020.
[47] Chin-Wei Huang, Laurent Dinh, and Aaron Courville. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models, arXiv:2002.07101.
[48] Matej Greié, Ivan Grubišić, and Siniša Segvić. Densely connected normalizing flows. In NeurlPS, 2021.
[49] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In NeurlPS, 2021.
[50] Giovanni S. Alberti and Luigi Ambrosio. geometrical approach to monotone functions in R". Math. Z, 230:259-316, 1999.
[51] M. D. Kirszbraun. Über die zusammenziehendden und lipschitzschen transformationen. Fund. Math., 22:77-108, 1934.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] In Appendix.
12

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes] In the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [No] We use single-seed results following 4 6
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] In Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] In section 6.3
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplementary material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "yJDA-iu5cyi", "reviewer": "Reviewer_crL7", "summary": "The paper proposed a bigger class of invertible residual-type blocks based on monotone operators. Theoretically, the paper shows with the same Lipschitz constraints on the network, their blocks can express more functions than the simple residual formulation. They also provide algorithms to estimate log-det and compute gradients. They then introduced Pila activation to tackle vanishing gradients. Experimentally, they show the proposed flow achieves better density estimation results than other flow-based methods on some standard image datasets. ", "questions": "- Thm 4 says $\\mathcal{M}_L=\\frac{1+L^2}{1-L^2} \\mathcal{R}_K, K=\\frac{2L}{1+L^2}$. Since $K<1$ when $L<1$, is choosing a function from $\\mathcal{G}_L$ and applying your formulation equivalent to choosing a function from $\\mathcal{G}_K$ and applying the residual formulation, if we disregard the constant $\\frac{1+L^2}{1-L^2}$? If these are the same then what is the advantage of your method? If there is a difference then what is the key difference here?\n- Thm2: what is the relationship between $F$ and $G$? Is there a typo here?\n- Sec3.4: what is the advantage of using concatenation? \n- What is the complexity of your algorithms (log-det estimator, forward and back-prop)? How is it compared to previous methods especially ResFlow? \n- What is the sampling speed of your method in experiments? What about quality evaluations such as FID? These numbers will make the experiment section more complete. \n\n\n----------------------------------------------\n\nAfter rebuttal:\n\nThanks for the author response. All my questions are answered. I think this is a very solid paper. I'll raise my score to 7. ", "limitations": "Yes, they discussed limitations in the appendix. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very clearly written. The application of monotone operators on invertible flows is novel to my knowledge. This paper is interesting to researchers who work on theory and applications of flow-based models, in the area of generative modeling. \n\nStrengths:\n- The biggest strength is that they derived $\\mathcal{M}_L$ with the monotone formulation, and show this class is rigorously larger than the ones in i-ResNet or implicit NFs. \n- The derivations of log-det estimator and back-prop are mostly standard. These provide ways to train their model with similar techniques as previous work (e.g. ResFlow). \n- They introduced the new activation function called Pila. I am not an expert in activation functions but it seems to work well according to Table 3. \n- They show better NLL than other flow-based models on several standard image datasets. \n\nWeakness:\n- As mentioned in Appendix F, their model has similar weaknesses as previous models such as computation and generation quality. \n- In experiments, improvement on density estimation is very marginal, and cannot compete with other generative methods outside normalizing flows. For example, some score-matching based models achieved NLL<2.5 on CIFAR10.\n- The experiments lack some other metrics such as sampling speed and FID of generated samples.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "F0aYdHk5z3P", "reviewer": "Reviewer_BzEz", "summary": "The paper proposes a new type of ResNet-based Normalizing Flows. In contrast to prior studies, which required the Lipschitz constant $L$ of each layer to be less than 1, the authors use monotone operators, which they show to be strictly more expressive. A new activation function called Concatenated Pila (CPila) is also proposed. The suggested model is evaluated on multiple toy datasets as well as standard image datasets, outperforming the baseline i-DenseNet model.", "questions": "* If possible with time, it would be nice if the authors could run their model with variational dequantization on CIFAR10 for the rebuttal.\n* Regarding monotone and maximally monotone operators: On line 79, what is meant by $u \\in F(x), v \\in F(y)$? From my understanding, $F(x)$ is an $n$-dimensional vector and not a set, so how can it have elements? The same applies on line 108: Given that $F$ is a function, what does it mean for $F$ not to be a proper *subset* of any monotone operator?\n* Line 118: What is $G$ in the definition of $C_F$? Should there be an $F$ in the numerator?\n", "limitations": "Yes, the limitations are adequately addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n**Originality**\n\nThe formulation of monotone operators is something I haven’t encountered previously in the field. The concept seems intriguing and well-conceived.\n\n**Quality**\n\nI did not check the proofs in the appendix, but the mathematical theory in the main part is sound and comprehensible. The experiments and ablations follow the established standard of the field.\n\n**Clarity**\n\nThe paper is very clearly written and easy to read. The authors do a good job at separating the main mathematical theory from the details and the proofs which are presented in the appendix.\n\n**Significance**\n\nGiven the originality of the monotone formulation, the paper may inspire future work in the field of normalizing flows. The reported improvements are small, but consistent over multiple datasets.\n\n### Weaknesses\n\n* The paper says that *The resulting model, Monotone Flows, exhibits an excellent density estimation performance and outperforms existing state-of-the-art normalizing flow models on multiple density estimation benchmarks (MNIST, CIFAR-10, ImageNet32, ImageNet64).* This is true when uniform dequantization is used during training. However, the Flow++ paper ([34]) reports an even lower bits-per-dimension score when using variational dequantization. Therefore, the proposed model should also be trained with variational dequantization and compared to Flow++ before making the state-of-the-art claim.\n* Some parts of the definition of monotone and maximally monotone operators are a bit confusing. (Further details in the Questions section below.)\n* Each of the experiment was only run a single time, which makes it harder to assess the significance of the reported numbers. However, this is a standard practice in the field of normalizing flows because each training run can take a lot of time. So this is only a minor problem, given the state of the field.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E_35onA-aTY", "reviewer": "Reviewer_ZJjf", "summary": "The paper \n* constructs normalizing flows based on monotone operators. Specifically, by Definition2, the paper defines $F(x)=(\\frac{Id+G}{2})^{-1}(x) - x$ for a $L$-Lipschitz ($L<1$) function $G$ (thus $G=C_F$ is the Cayley operator of $F$). Next from the  Cayley operator identity $C_F+C_{F^{-1}} = 0$, $F^{-1}$ can be defined by \n$$F^{-1}(x) = C_{-G}(x) = (\\frac{Id-G}{2})^{-1}(x) - x.$$\nFor $G$, the authors use the architecture in i-DenseNets and propose a new activation function, CPila. For training and inference, log determinant can be computed via\n$$\\log \\det J_F = \\text{tr}[\\log (I-J_G) - \\log (I+J_G)]$$\nand estimated with some existing unbiased estimators. Training and inference schemes are also developed well using existing techniques in this work. \n\n* analyzes the expressive power of Monotone Flows theoretically, and compares with residual flows and implicit flows. Experimentally, the paper evaluates Monotone Flows on image density estimation tasks and shows the superiority to some normalizing flow models. \n\n\n\n", "questions": "* As mentioned in the weakness.B, some works are omitted in the experiments. ", "limitations": "Limitations seen above. Societal impact: none", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\nA. To the best of my knowledge, constructing normalizing flows with monotone operator is novel. \n\nB. Theoretic analysis of expressive power is valid. Experimental results also shows its superiority to some residual flows. Training and inference schemes based on existing techniques are also valid. Moreover, based on the observation that LipSwish suffers from the vanishing gradient problem, the paper proposes a new activation function CPila, which is also a novel contribution. Ablation studies also support the effects of CPila. \n\nC. I think Monotone flows is a valid and novel contribution to the community of normalizing flow models.\n\n**Weaknesses**:\n\nA. Some notations in the paper are quite confusing. In Theorem2, the authors say $C_F = (\\frac{Id+F}{2})^{-1} - Id$ is its Cayley operator, which is quite confusing and different from Line81. I think the definition in Line81 is the correct one and then the Definition2 is the 'inverse' Cayley operator so $G=C_F$. \n\nB. The paper claims that they achieves the SOTA in density estimation task, but to my knowledge, there are some works not included in the experiment section such as Flow++ [1], VFlow [2], Scoreflow [3]\n\n**Minor**:\n\n**Reference**\n\n[1] Ho, J., Chen, X., Srinivas, A., Duan, Y., and Abbeel, P. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722– 2730, 2019.\n\n[2] J. Chen, C. Lu, B. Chenli, J. Zhu, and T. Tian. Vflow: More expressive generative flows with variational data augmentation. In International Conference on Machine Learning, pages 1660–1669. PMLR, 2020.\n\n[3] Song, Y., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models. arXiv e-prints, pp. arXiv–2101, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AOAlSsurbx-", "reviewer": "Reviewer_cueX", "summary": "This paper proposes Monotone Flows which uses monotone operator theory to parameterize Residual Flows. The authors also introduce a novel activation function in Pila and CPila that improves upon the LipSwish activation function. As a theoretical contribution, the authors demonstrate the Monotone Flows using the Cayley operator are able to express a larger family of functions when compared to vanilla Residual and Inverse Residual Flows. Empirically the authors show the benefits of Monotone Flows over standard baselines in qualitative samples and density estimation.", "questions": "1. In the original Residual Flows paper there was a precise characterization of the memory cost (Fig 3). How does Krasnoselskii-Mann iteration affect this?\n2. An effect of the activation through training like Fig 8 in the ResFlow paper could be illuminating. Can you please consider including it?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The approach to parametrize Residual Flows using Monotone Operators and Cayley Operators is both an interesting and novel direction. \n- The presented theory is both clear and integrates nicely with the overall goal of building a novel class of Normalizing Flow.\n- The ablation experiments help justify the choice of the activation function.\n- The experiments do enough to demonstrate that Monotone Flows are extremely expressive and powerful.\n- Theorem 4, is especially crisp at outlining the main advantages of the Cayley Formulation from a theoretical point of view.\n\nWeakness:\n- One of the claims in the abstract is that Monotone Flows outperform SOTA flows. Looking at the presented numbers and comparing it with relevant literature this is not true. Here is a non-exhaustive list of papers that outperform Monotone Flows (albeit using other advances that could also potentially be complementarily used here): Flow++[1], Augmented Normalizing Flow [2], Densely Connected Normalizing Flows [3].\n- The authors should also include FFJORD [4] as a relevant baseline.\n- The use of Russian Roullete, Neumann series,  and Hutichinsons trace estimator for training and inference borrows much of what is already known in the flow literature. The authors should try to do a better job of disambiguating the main technical differences in Monotone Flows from the rest of the Residual Flow models. \n- The reported tables should have standard deviations as the numbers are very close.\n\n\nMinor:\n- line 131, T hasn't been introduced yet.\n- Definition 4, A is the set of what function? Be more precise.\n- Why do Residual Flows have higher precision---i.e. one more digit after the decimal---in Table 2 for Cifar10?\n\n[1] Ho, Jonathan, et al. \"Flow++: Improving flow-based generative models with variational dequantization and architecture design.\" International Conference on Machine Learning. PMLR, 2019.\n\n[2] Huang, Chin-Wei, Laurent Dinh, and Aaron Courville. \"Augmented normalizing flows: Bridging the gap between generative flows and latent variable models.\" arXiv preprint arXiv:2002.07101 (2020).\n\n[3] Grcić, Matej, Ivan Grubišić, and Siniša Šegvić. \"Densely connected normalizing flows.\" Advances in Neural Information Processing Systems 34 (2021): 23968-23982.\n\n[4] Grathwohl, Will, et al. \"Ffjord: Free-form continuous dynamics for scalable reversible generative models.\" arXiv preprint arXiv:1810.01367 (2018).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept