Invertible DenseNets with Concatenated LipSwish
Authors: Yura Perugachi-Diaz
Jakub Mikolaj Tomczak
Sandjai Bhulai
Venue: NeurIPS 2021 Poster (2021)
Keywords: invertible
densenets
normalizing flow

Abstract: We introduce Invertible Dense Networks (i-DenseNets), a more parameter efficient extension of Residual Flows. The method relies on an analysis of the Lipschitz continuity of the concatenation in DenseNets, where we enforce invertibility of the network by satisfying the Lipschitz constant. Furthermore, we propose a learnable weighted concatenation, which not only improves the model performance but also indicates the importance of the concatenated weighted representation. Additionally, we introduce the Concatenated LipSwish as activation function, for which we show how to enforce the Lipschitz condition and which boosts performance. The new architecture, i-DenseNet, out-performs Residual Flow and other flow-based models on density estimation evaluated in bits per dimension, where we utilize an equal parameter budget. Moreover, we show that the proposed model out-performs Residual Flows when trained as a hybrid model where the model is both a generative and a discriminative model.

Invertible DenseNets with Concatenated LipSwish
Yura Perugachi-Diaz
Jakub M. Tomezak Vrije Universiteit Amsterdam j tomczak@vu.n ml
Sandjai Bhulai Vrije Universiteit Amsterdam bhulail dnu.nl
Vrije Universiteit Amsterdam
y m. sitruggachidaz@@u
Abstract
We introduce Invertible Dense Networks (i-DenseNets), more parameter efficient extension of Residual Flows. The method relies on an analysis of the Lipschitz continuity of the concatenation in DenseNets, where we enforce invertibility of the network by satisfying the Lipschitz constant. Furthermore, we propose learnable weighted concatenation, which not only improves the model performance but also indicates the importance of the concatenated weighted representation. Additionally, introduce the Concatenated LipSwish activation function, for which show how to enforce the Lipschitz condition and which boosts performance. The new architecture, i-DenseNet, out out-performs Residual Flow and other flow-based models on density estimation evaluated in bits per dimension, where we utilize equal parameter budget. Moreover, performs Residual Flows when trained as a hybrid model where the model is both generative and discriminative model.
show that the proposed model out-
Introduction
Neural networks are widely used to parameterize non-linear models in supervised learning tasks such as classification. In addition, they are also utilized to build flexible density estimators of the true distribution of the observed data [25, 33]. The resulting deep density estimators, also called deep generative models, can be further used to generate realistic-looking images that are hard to separate from real ones, detection of adversarial attacks [9, 17J. and for hybrid modeling [27] which have the property to both predict label (classify) and generate.
Many deep generative models are trained by maximizing the (log- )likelihood function and their architectures come in different designs. For instance, causal convolutional neural networks are used to parameterize autoregressive models [28, 29] or various neural networks can be utilized in Variational Luto-Encoders [19, 32). The other group of likelihood- based deep density estimators, wawaattrrs....wwwwaooooooooooooe flow-based models (or flows), consist of invertible neural networks since they are used to compute the likelihood through the change of variable formula [31, 37, 36). The main difference that determines an exact computation or approximation of the likelihood function for flow-based model lies in the design of the transformation layer and tractability of the mecobian-determinant Many flow-based models formulate the transformation that is invertible and its Jacobian is tractable [3, 6-8, 21, 30, 31, 38).
Recently, Behrmann et [2] proposed a different approach, namely, deep-residual blocks as transformation layer. The decp-residual networks (ResNets) of [12] are known for their successes in supervised learning approaches. In a ResNet block, each input of the block is added the output, which forms the input for the next block. Since ResNets are not necessarily invertible, Behrmann al. [2] enforce the Lipschitz constant of the transformation to be smaller than 1 (i.e., it becomes a contraction) that allows applying an iterative procedure to invert the network. Furthermore, Chen et al. [4] proposed Residual Flows, an improvement j-ResNets, that uses an unbiased estimator for the logarithm of the Jacobian-determinant
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

Wirs
hg
P(ar)
(a) Residual block
(b) Dense block
Figure 1: A schematic representation for: (a) a residual block, (b) dense block. The pink part in (b) expresses convolution reduce the dimension of the last dense layer. W; denotes the (convolutional) layer at step i that satisfy ||ill <
In supervised learning, architecture that uses fewer parameters and is even more powerful than the deep-residual network is the Densely Connected Convolution Network (DenseNet), which was first presented in [15]. Contrary to a ResNet block, DenseNet layer consists of concatenation of the input with the output. The network showed to improve significantly in recognition tasks on benchmark datasets such as CIFARIO 10, SVHN, and ImageNet, by using fewer computations and having fewer parameters than ResNets while performing at a similar level.
In this work, we extend Residual Flows [2, 4], and use densely connected blocks (DenseBlocks) as a residual layer. First, we introduce invertible Dense Networks (i-DenseNets), and we show that we can derive bound on the Lipschitz constant to create an invertible flow-based model. Furthermore, we propose the Concatenated LipSwish (CLipSwish) as an activation function, and derive a stronger Lipschitz bound. The CLipSwish function preserves more signal than LipSwish activation functions. Finally, we demonstrate how i-DenseNets can be efficiently trained as generative model, outperforming Residual Flows and other flow-based models under equal parameter budget.
2 Background
Flow-based models Let us consider a vector of observable variables æ E Rd and vector of latent variables z e RRd. We define a bijective function f R -> Rdd that maps latent variable datapoint f(2). Since f is invertible, we define its inverse as F = f-1 We use the change of variables formula compute the likelihood datapoint after taking the logarithm, that is:
px (x) In (2() + In dot [x)),,
where Pz (z) is base distribution (e.g., the standard Gaussian) and JF(x) is the Jacobian of F' The bijective transformation is typically constructed as a sequence of K invertible transformations. f1(z), and single transformation fie referred to as flow [31]. The change of variables formula allows evaluating the data in tractable manner. Moreover, the flows are trained using the log likelihood objective where the of the invertible transformations.
= fK
compensates the change of volume
Residual flows Behrmann al. [2] construct invertible ResNet layer which is only constrained in Lipschitz continuity. A ResNet is defined as: F(x) = x g(x), where is modeled by (convolutional) neural network and F represents ResNet layer (see Figure ((a)) which is in general not invertible. However, is constructed in such a way that it satisfies the Lipschitz constant being strictly lower than 1, Lip(g) < 1, by using spectral normalization of [10, 26]:
Lip(g) < 1, if |Wa]|2 1,
where ||- 1|2 the &g matrix norm. Then Lip(g) = K < 1 and Lip(F) < 1++ case the Banach fixed- exed-point theorem holds and ResNet layer has unique inverse. As a result, the inverse be approximated by fixed-point iterations.
1.1K K. Only in this specific
To estimate the dog-determinant is, especially for high-dimensional spaces, computationally in- tractable due to expensive computations. Since ResNet blocks have constrained Lipschitz constant,

---

the logllikelhhood estimation of Equation (1) can be transformed to a version where the logarithm of the Jacobian-detemmnn determinant cheaper to compute, tractable, and approximated with guaranteed convergence [2]:
(-1)
In (np())=In(((())) + tr Σ k
where (2(x) is the Jacobian of 9 at that satisfies || < The Skilling- HolllingHutthhnsoon trace estimator [35, 16] is used to compute the trace at a lower cost than to fully compute the trace of the Jacobian. Residual Flows [4] use an improved method to estimate the power series at an even lower cost with an unbiased estimator based on "Russian roulette" of [18]. Intuitively, the method estimates the infinite of the power series by evaluating a finite amount of terms. In return, this leads to less computation of terms compared to invertible residual networks. To avoid derivative saturation, which occurs when the second derivative is zero in large regions, the LipSwish activation is proposed.
Invertible Dense Networks
In this section, we propose Invertible Dense Networks by using a DenseBlock as a residual layer. We show how the network can be parameterized as flow-based model and refer to the resulting model as i-DenseNets. The code can be retrieved from: sstps://githig wwwcccom//e//eeee yytpy/pccuduaaaa.... invertible deteeeddddssss...
3.1 Dense blocks
The main component of the proposed flow-based model is DenseBlock that is defined as function F Rdd + Rd with F(x) = x + g(x), where g consists of dense layers {hr Note that an
important modification make the model invertible is to output z g(z) whereas standard DenseBlock would only output g(x). The function g is expressed as follows:
g (xx) = Wx+1 o hn .... h1 (x))
(4)
where W.+: represents 1 X convolution to match the output size of Rd A layer h; consists of two parts concatenated to each other. The upper part is a copy of the input signal. The lower part consists of the transformed input, where the transformation is multiplication of (convolutional) weights W; with the input signal, followed by non-lineed -on-linearity ó having Lip(ø) s 1, such as ReLU, ELU, LipSwish, tanh. As an example, dense layer h2 can be composed as follows:
h (x) = ( (2)1)(z))) = h1 (x) (6W2h1
[
(5)
In Figure 1, we schematically outline a residual block (Figure ((a)) and a dense block (Figure 1(b)). We refer to concatenation depth as the number of dense layers in DenseBlock and growth as the channel growth size of the transformation in the lower part.
3.2 Constraining the Lipschitz constant
If we enforce function to satisfy Lip(g) < 1, then DenseBlock F' is invertible since the Banach fixed point theorem holds. As a result, the inverse can be approximated in [2]. To satisfy Lip(g) < 1, we need to enforce Lip(h;) < 1 for all n layers, since Lip(g) < Lip(ha+1) ... Lip(h1). Therefore, we first need determine the Lipschitz constant for dense layer h, For the full derivation, see Appendix A. We know that a function f is K-Lipschitz if for all points v and w the following holds
the same manner as
dy (f() & Kdx (u,w),
(6)
where we assume that the distance metries dx dy d are chosen to be the normm Further, let two functions fi and fa be concatenated h:
[1(v)] [2(6)
[w]
h =
hw =
[2(w)
(7)
3

---

where function fi is the upper part and f2 is the lower part. We can now find an analytical form to express limit K for the dense layer in the form Equation (6):
d(hn, hww = (o),, + d()(()) d(h hw) (KF + Kī)d(e, w)²,
(8)
where we know that the Lipschitz constant of h consist of two parts, namely, Lip(fi) = K1 and Lip(f2) K2. Therefore, the Lipschitz constant layer h can be expressed as:
Lip(h) = (K} + K3).
(9)
With spectral normalization of Equation (2), we know that we can enforce (convolutional) weights W; to be at most 1-L.ipchhizz. Hence, for all n dense layers we apply the spectral normalization on the lower part which locally enforces Tip(f2) = K₂ < 1. Further, since we enforce each layer h; to be at most 1 1-LLiscchitz and we start with h1, where fi (x) 2. we know that Lip(f1) = 1. Therefore,
the Lipschitz constant of an entire layer can be most Lip(h) <v1² 2 + 1 =V2, thus dividing by this limit enforces each layer to be at most I-Lipschitz.
3.3 Learnable weighted concatenation
We have shown that we enforce an entire dense layer to have Lip(h;) < by applying a spectral norm on the (convolutional) weights W; and then divide the layer h; by /2. Although learning a weighting between the upper and lower part would barely affect a standard dense layer, it matters in this case because the layers are regularized to I-Lipschitz. To optimize and learn the importance of the concatenated representations, we introduce learnable parameters 771 and 7/2 for, respectively, the upper and lower part of each layer h4
f/2
1
f)
Since the upper and lower part of the layer can be at most 1-Lipschitz, mul- tiplication by these factors results in functions that are at most 7/2 -iissccctt and 7/2- -ppschis As indicated by Equation (9), the layer then most V + 1/ Lipschitz. Dividing by this factor results in a bound that is at most 1-Lipschitz
Figure 2: Range of the possible normalized parameters fn and î2.
In practice, we initialize 171 and f/2 at value 1 and during training use a softplus function to avoid them being negative. The range of the normalized parameters is between i. 1/2 E [0. 1] and can be expressed on the unit circle shown in Figure 2. In the special case where 771 172, the normalized parameters are î fi2 V2. This case corresponds the situation in Section 3.2 where the concatenation is not learned. An additional advantage is that the normalized n and flz express the importance of the upper and lower signal. For example, when i > f2. the input signal is of more importance than the transformed signal.
3.4 CLipSwish
When deep neural network bounded be 1-Lipschitz, in practice each consecutive layer reduces the Jacobian-non norm. As result, the Jacobian -oorm of the entire network is becoming much smaller than and the expressive power is getting lost. This known as gradient norm attenuation [1, 24]. This problem arises in activation functions in regions where the derivative is small, such as the left tail of the ReLU and the LipSwish. Non-linearities modeled i-DenseNets are required to be at most I-Lipschitz and, thus, face gradient-norm attenuation issues. For this reason we introduce new activation function, which mitigates these issues.
Recall that Residual Flows use the LipSwish activation function [4]:
LipSwish(z)) (x) = (3x))/1111
(10)
where (Br) 1/(1 exp(-x33) is the sigmoid, , 33 is a learnable constant, initialized at 0.5 and passed through a softplus to be strictly positive. This activation function is not only Lip(TipSwish) = 1 but also resolves the derivative saturation problem [4]. However, the LipSwish function has large ranges on the negative axis where its derivative is close to zero.

---

Therefore, we propose the Concatenated LipSwish (CLipSwish) which concatenates two LipSwish functions with inputs x and =r. This is a concatenated activation function as in [34] but using a LipSwish instead of a ReL.U. Intuitively, even if an input lies in the tail of the upper part, it will have larger derivative in the bottom part and thus suffer less from gradient norm attenuation. Since using CLipSwish increases the channel growth and to stay inline with the channel growth that non-conee non-concatenated activation functions use, we use a lower channel growth when using CLipSwish. To utilize the CLipSwish, we need to derive Lipschitz continuity of the activation function & defined below and enforce it to be I-Lipschitz. We could use the result obtained Equation (9) to obtain /2-bound, however, by using knowledge about the activation function Φ, we can derive a tighter 1.004 < v2 bound. In general, a tighter bound is preferred since more expressive power will be preserved in the network. To start with, we define function +b R I for point as:
(z) (2(x)) LipSwish(-22
LipSwish(z)
(x)) =
CTipSwish(x) = (xx)/Lip((p)
(11)
where the LipSwish is given by Equation (10) and the derivative of ((x) exists. To find Lip(O) we use that for differentiable f2-I.ipschitz bounded function + the following identity holds:
Lip(e) () sup || (2)
(12)
where Jo is the Jacobian of & and ||- ||2 represents the induced matrix norm which is equal to the spectral norm of the matrix. Rewriting the spectral norm results in solving: (x) Jo))) XI,) = O, which gives us the final result (see Appendix A.3.1 for the full derivation):
sup (1))l22 = sup max (Js (J)(z)) = sup /
(00)(()) dr
( Ör
@0bb (Dib((r) (2r)
=sup
+
(13)
where max (-) is the largest singular value. Now Lip( is the upper bound of the CLipSwish and is equal to the supremum of: Lip(@)= = sup, || (2)]l2 s 1.004, for all values of B. This can be numerically computed by any solver, by determining the extreme values of Equation (13). Therefore, dividing ((x) by its upper bound 1.004 results in Lip CLipSwish) = 1. The generalization to higher dimensions can be found in Appendix A.3.2. The analysis of preservation of signals for (CLip) Swish activation by simulations can be found in Section 5.1.
4 Experiments
To make a clear comparison between the perfor- mance of Residual Flows and i-DenseNets, we train both models on 2-dimensional toy data and high-dimensional image data: CIFAR10 [22] and ImageNet32 [5]. Since we have a constrained com- putational budget, the exploration of the network architectures. An in- depth analysis of different settings and experiments can be found in Section 5. For density estimation, run the full model with the best settings for 1,000 epochs on CI and 20 epochs on Ima- geNet32 where we use single-seed results following [2, 4, 20]. due to little fluctuations in performance. In all cases, we use the density estimation results of the Residual Flow and other flow-based models using uniform dequantization to create a fair com- parison and benchmark these with i-DenseNets. We train i-DenseNets with learnable weighted concate- nation (LC) and CLipSwish as the activation function, and utilize a similar number of parameters for i-DenseNets as Residual Flows; this can found in Table 2. i-DenseNets uses slightly fewer parameters than the Residual Flow. A detailed description of the architectures can be found in Appendix B. To speed up training, we use 4 GPUs.
Sampled data Residual Flow
i-DenseNet
smaller architectures for
Figure 3: Density estimation for smaller ar- chitectures of Residual Flows and i-DenseNets, trained on 2-dimensional toy data.
5

---

4.1 Toy data
We start with testing i-DenseNets and Resid- ual Flows toy data, where we smaller architectures. Instead of 100 flow blocks, we use 10 flow blocks. We train both models for 50,000 iterations and, at the end of the training, visualize the learned distributions.
Table 1: Negative log-likelihood results on test data in nats (toy data). i-DenseNets w/ and w/o LC are compared with the Residual Flow.
Model
circles
checkerboard 2 moons
Residual Flows
3.44
3.81
2.60
--DessNNets
3.32
3.68
2.39
The results of the learned density distributions are presented in Figure 3. We observe that Residual Flows are capable to capture high- probability areas. However, they have trouble with learning low probability regions for two circles and moons. i-DenseNets are capable of capturing all regions of the datasets. The good performance of i- i-DenseNets is also reflected in better performance in terms of the negative-log-likee likelihood (see Table 1).
i-DenseNets+LC
3.30
3.66
2.39
4.2 Density Estimation
We test the full i-DenseNet models with LC and CLipSwish activation. To utilize a similar number of parameters as the Residual Flow with 3 scale levels and flow blocks set 16 per scale trained on CIFARIO, we set for the same number of blocks, DenseNets growth to 172 with a depth of 3. Resid- ual Flow trained on ImageNet32 uses 3 scale levels with flow blocks per scale, therefore, we set for the same number of blocks DenseNets growth to 172 and depth of utilize similar number of parameters. DenseNets depth set to 3 proved to be the best settings for smaller architectures; see the analysis in Section
Table 2: The number of parameters of Resid- ual Flows and i-DenseNets for the full models trained in Chen et al. [4]. In brackets, the number of parameters of the smaller models.
Model/Data
CIFARIO
ImageNet32
Residual Flows 25.2M (8.7M)
47.1M
i-DenseNets
24.9M (8.7M)
47.0M
The density estimation on CIFARIO and Ima- geNet32 are benchmarked against the results of Residual Flows and other comparable flow-based models, where the results are retrieved from Chen et al. [4]. We measure performances in bits per dimension (bpd). The results can be found in Ta- ble 3. We find that i-DenseNets out-perform Resid- ual Flows and other comparable flow-based models on all considered datasets in terms of bpd. On CIFARIO. 10, i-DenseNet achieves 3.25bpd, against
Table 3: Density estimation results in bits per dimension for models using uniform dequanti- zation. In brackets results for the smaller Resid- ual Flow and i-DenseNet run for 200 epochs.
Model
CIFARIO
ImageNet32
Real NVP ())
3.49
4.28
Glow (20)
3.35
4.09
FFJORD
3.40
Flow++ (13)
3.29
3. 3.28 of the Residual Flow. On ImageNet32 i-
ConvSNF (4]
3.29
DenseNet achieves 3. 3.p8 against 4.01bpd of the Residual Flow. Samples of the i-DenseNet models can be found in Figure 4. Samples of the model presented in Figure 4(b)
i-ResNet
3.45
Residual Flow 14]
3.28 (3.42)
4.01
i-DenseNet
3.25 (3.37)
3.98
trained on CIFARIO and samples of the model trained on ImageNet32 in Figure 4(d). For more unconditional samples, Appendix C.I. Note that this work does not compare against flow-based models using variational dequantization. Instead we focus on extend- ing and making fair comparison with Residual Flows which, similar to other flow-based models, use uniform dequantization. For reference note that Flow++ [13] with variational dequantization obtains 3.08bpd on CIFARIO 10 and 3.86bpd ImageNet32 that is better than the model with uniform dequantization which achieves 3, 3.29bpd on CIFARIO.
4.3 Hybrid Modeling
Besides density estimation, we also experiment with hybrid modeling [27]. We train the joint distribution p(x, y) = p(x) p(y]x), where p(x) is modeled with a generative model and p(y[x) is modeled with a classifier, which uses the features of the transformed image onto the latent space.

---

(a) Real CIFARIO images.
(b) Samples i-DenseNets trained on CIFARIO.
(c) Real ImageNet32 images.
(d) Samples of Nets trained ImageNet32.
Figure 4: Real and samples of CIFARIO and ImageNet32 data
Due to the different dimensionalities of y and x, the emphasis of the likelihood objective more likely to be focused on p(x) and a scaling factor for weighted maximum likelihood objective is suggested, Ez,y-D p(yl)))) + À log p(x)), where is the scaling factor expressing the trade-off' between the generative and discriminative parts. Unlike [27] where linear layer is integrated on top of the latent representation, we use the architecture of [4] where the set of features are obtained after every scale level. Then, they concatenated and are followed by linear softmax classifier. We compare our experiments with the results of [4] where Residual Flow, coupling blocks [7] and x 1 convolutions [20] are evaluated.
Table 4 presents the hybrid modeling results on CIFARIO where we used A {0, 1. 1}. We run the three mod- els for 400 epochs and note that the was not fully con- verged in both accuracy and bits per dimension after training. The classi- fier model obtains a converged accu- racy after around 250 epochs. This is in line with the accuracy for the model with x = /. yet based on bits per dimension the model was not fully converged after 400 epochs. This indicates that even though the accuracy is not further improved, the model keeps optimizing the bits per dimension which gives room for future research. Results in Table 4 show the average result over the last 5 epochs. We find that Dense Blocks out-perform Residual Blocks for all possible À settings. Interestingly, Dense Blocks have the biggest impact using no penalty (A = 1) compared to the other models. We obtain an accuracy of 75.67% with 3.31bpd, compared 70.32% accuracy and 3.39bpd of Residual Blocks, indicating that Dense Blocks significantly improves classification performance with more than 5%. In general, the Dense Block hybrid model our-performing Real NVP, Glow, FFJORD, and i-ResNet in bits per dimension (see Appendix C.2 for samples of the hybrid models).
Table 4: Results of hybrid modeling on CIFARIO. Arrows indicate if low high values of importance. Results average over the last epochs.
model with
À
A
Model NEvaluation
Acc f
bpd
Ace 1
bpd $
Coupling
89.77%
87.58%
4.30
67.62%
3.54
+1 1
90.82%
87.96%
4.09
67.38%
3.47
Residual Blocks (full)
91.78%
90.47%
3.62
70.32%
3.39
Dense Blocks (full)
92.40%
90.79%
3,49
75.67%
3,31
5 Analysis and future work
To get a better understanding of i-DenseNets, perform additional experiments, explore different settings, analyze the results of the model and discuss future work. We use smaller architectures for these experiments due to limited computational budget. For Residual Flows and i-DenseNets use scale levels set to 4 Flow blocks instead of 16 per scale level and train models on CIFAR10 for 200 epochs. We will start with short explanation of the limitations of 1-Lipschitz deep neural networks.
5.1 Analysis of activations and preservation of signals
Since gradient-norm attenuation can arise I-Lipschitz bounded deep neural nets, we analyze how much signal of activation functions is preserved by examining the maximum and average distance ratios sigmoid, LipSwish, and CLipSwish. Note that maximum distance ratio approaches the Lipschitz constant and it is desired that the average distance ratio remains high.
7

---

We sample 100,000 datapoints v, to ~ (00, 1) with dimension set D {1, 128, 10243. We compute the mean and maximum of the sampled ratios with: (z)()) ((w))//2(1) w) and analyze the expressive power of each function. Table 5 shows the re- sults. We find that CLipSwish for all dimensions preserves most of the sig- nal on average compared to the other non-linearities. This may explain why i-DenseNets with CLipSwish activation achieves better results than using, e.g., LipSwish. This experiment indicates that randomly sampled points, CLipswish functions suffer from considerably less gradient norm attenuation. Note that sampling from a distribution with larger parameter values is more pronounced in preference of CLipSwish, see Appendix D.1.
Table 5: The mean and maximum ratio for different dimen- sions with sample size set to 100,000.
Activationl Measure
D
128
1024
Mean
Max
Mean
Max
Mean
Max
Sigmoid
0.22
0.25
0.21
0.22
0.21
0.21
LipSwish
0,46
1.0
0.51
0.64
0.51
0.55
ClipSwish
0.72
1.0
0.71
0.77
0.71
0.73
Identity
1.0
1.0
1.0
1.0
1.0
1.0
5.2 Activation Functions
We start with exploring different activation functions for both networks and test these with the smaller architectures. We compare our CLipSSishh to the LipSwish and the Leakyl LeakyLSSwish LSiish as an additional baseline, which allows freedom of movement in the left tail as opposed to a standard LipSwish:
LeakyLSwish(z) = az + (1 - a))ipSwish((z),
(14)
with a E (0. 1) by passing it through a sigmoid function a. Here a a learnable parameter which initialized at a (-3) to mimic the LipSwish initialization. Note that the dimension for Residual Flows with CLipSwish activation function is set to 652 instead of 512 to maintain a similar number of parameters (8.7M) as with LipSwish activation.
Table 6 shows the results of each model using different activation func- tions. With 3.37bpd we conclude that i-DenseNet with our CLipSwish the activation function obtains the best performance compared to the other activation functions, LipSwish and LeakyLSwish. Furthermore, all i- DenseNets out-perform Residual Flows with the same activation function. We want to point out that CLipSwish as the activation function not only boosts performance of i-DenseNets but also significantly improves the performance of Residual Flows with 3.38bpd. The running time for the forward pass, train time and sampling time, expressed in percentage faster or slower than Residual Flow with the same activation functions, can be found in Appendix D.2.
Table 6: Results in bits per dimensions for small architec- tures, testing different activation functions.
Model
LipSwish Leakyl.Swiss
CLipSwish
Residual Flow
3.42
3.42
3.38
i-DenseNet
3.39
3.39
3.37
5.3 DenseNets concatenation depth
Next, we examine the effect of different concate- nation depth settings for i-DenseNets. We run experiments with concatenation depth set to 2, 3, 4, and 5 with CLipSwish. Furthermore, to uti- lize 8.7M parameters of the Residual Flow, choose a fixed depth and appropriate DenseNet growth size have a similar number param- eters. This results in a DenseNet depth 2 with growth size 318 (8.8M), depth 3 with growth 178 (8.7M), depth with growth 122 (8.7M), and depth 5 with growth 92 (8.8M). The effect of each architecture can be found in Figure 5. We observe that the model with a depth of obtains the best scores and after 200 epochs achieves the lowest bits per dimension with 3.37bpd. A concatenation depth of 5 results in
3.50
Depth:
Depth:
Bits dimersion
3.46
3.40
3.00
3,38
25
125
175
epach
Effect of different concatenation depths with CLipSwish activation function for i- DenseNets in bits per dimension.
Figure 5:
8

---

3. 3.42bpd after 200 epochs, which is the least preferred. This could indicate that the corresponding DenseNet growth of 92 is too little to capture the density structure sufficiently and due to the deeper depth the network might lose important signals. Further, the figure clearly shows how learnable weighted concatenation after 25 epochs boosts training for all i-DenseNets. See Appendix D.3 for an in-depth analysis of results of the learnable weighted concatenation. Furthermore, we performed an additional experiment (see Appendix D.4) where we extended the width and depth of the ResNet connections in g(x) of Residual Flows in such a way that it matches the size of the i-DenseNet. As result on CIFARIO 10 this puts the extended Residual Flow at a considerable advantage as it utilizes 19.1M parameters instead of 8. 8. However, when looking performance the model performs worse (7. (22ppd) than i-Densenets (3.39bpd) and even worse than its original version (3.42bpd) in terms of bpd. A possible explanation of this phenomenon is that by forcing more convolutional layers to be 1-Lipschitz, the gradient norm attenuation problems increase and in practice they become considerably less expressive. This indicates that modeling DenseNet in g(x:) is indeed an important difference that gives better performance.
5.4 Future Work
We introduced new framework, i-D I-DenseNet, that is inspired by Residual Flows and i-ResNets. We demonstrated how i-DenseNets out-performs Residual Flows and alternative flow-based models for density estimation and hybrid modeling. constraint by using uniform dequantization. For future work, we want to address several interesting aspects we came across and where i-DenseNets may be further deployed and explored.
First of all, we find that smaller architectures have more impact on performance than full models compared to Residual Flows. Especially for exploration of the network, we recommend experimenting with smaller architectures or when a limited computational budget is available. This brings us to the second point. Due to a limited budget, we trained and tested i-DenseNets on 32 32 CIFARIO 10 and ImageNet32 data. will be interesting to test higher resolution and other types of datasets.
Further exploration of DenseNets depth and growth for other or higher resolution datasets may be worthwhile. In our studies, deeper DenseNets did not result in better performance. However, it would also be beneficial to further examine the optimization of DenseNets architectures. Similarly, we showed how to constrain DenseBlocks for the fg-norm. For future work, it may be interesting to generalize the method to different norm types, as well as the norm for CLipSwish activation function. Note that CLipSwish as activation function not only boosts performance of i-DenseNets but also for Residual Flows. We recommend this activation function for future work.
We want stress that we focused on extending Residual Flows, which uses uniform dequantization. However, we believe that the performance of our network may be improved using variational dequantization or augmentation. Finally, we found that especially hybrid model with A 1, achieve better performance than its predecessors. This may be worthwhile to further investigate in the future.
Societal Impact We discussed methods to improve normalizing flow, a method that learns high- dimensional distributions. We generated calistic-lloking images and also used hybrid models that both predict the label of an image and generate new ones. Besides generating images, these models can be deployed to, e.g., detect ad adversarial attacks. Additionally, this method is applicable to all different kind of fields such as chemistry or physies. An increasing concern is that generative models in general, have an impact on society. They can not only be used to aid society but can also be used to generate misleading information by those who use these models. Examples of these cases could be generating real-looking documents, Deepfakes or even detection of fraud with the wrong intentions. Even though current flow-based models are not there yet to generate flawless reproductions, this concern should be kept in mind. It even raises the question if these models should be used in practice when detection of misleading information becomes difficult or even impossible to track.

---

Conclusion
In this paper, we proposed i-DenseNets, a parameter--fficient alternative Residual Flows. Our method enforces invertibility by satisfying the Lipschitz continuity in dense layers. In addition, we introduced a version where the concatenation of features learned during training that indicates which representations are of importance for the model. Furthermore, we showed how to deploy the CLipSwish activation function. For both i-DenseNets and Residual Flows this significantly improves performance. Smaller architectures under an equal parameter budget were used for the exploration of different settings.
The full model for density estimation was trained on 32 x 32 CIFARIO and ImageNet32 data. We demonstrated the performance of i-DenseNets and compared the models Residual Flows and other comparable Flow-based models on density estimation in bits per dimension. Yet, it also demonstrated how the model could be deployed for hybrid modeling that includes classification in terms of accuracy and density estimation in bits per dimension. Furthermore, we showed that modeling ResNet connections matching the size of an i-DenseNet obtained worse performance than the i-DenseNet and the original Residual Flow. In conclusion, i-DenseNets out out-perform Residual Flows and other competitive flow-based models for density estimation on all considered datasets in bits per dimension and hybrid modeling that includes classification. The obtained results clearly indicate the high potential of i-DenseNets as powerful flow-base basedased models.
Acknowledgments
We would like thank Patrick Forré for his helpful feedback the derivations. Furthermore, this work was carried out on the Dutch national e-infrastructure with the support of SURF Cooperative.
Funding Transparency Statement
There are no additional sources of funding to disclose.
10

---

References
[1] Cem Anil, James Lucas, and Roger Grosse. Sorting out Lipschitz function approximation. In International Conference Machine Learning. 2019.
[2] Jens Behrmann. Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference on Machine Learning. 2019.
[3] Rianne van den Berg, Leonard Hasenclever, Jakub M Tomezak, and Max Welling. Sylvester normalizing flows for variational inference. arXiv preprint arXiv: 1803.05699 2018.
[4] Ricky T. Q. Chen, Jens Behrmann, David Duvenaud, and for invertible generative modeling. In Advances in Neural Information Processing Systems, 2019.
Jacobsen. Residual flows
[5] Patryk Chrabaszez, Iya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint 170.088707,,889, 2017.
[6] Nicola De Cao, Wilker Aziz, and Ivan Titov. Block neural autoregressive flow. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence, 2019.
[7] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv:1410.8516, 2015.
[8] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. arXiv1665.008803, 2017.
[9] Ethan Fetaya, Jörn- Hörn-K Jacobsen, Will Grathwohl, and Richard Zemel. Understanding the limitations of conditional generative models. arXiv:196.00177,, 2019.
[10] Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree. Regularisation of neural networks by enforcing Lipschitz continuity. arXiv preprint arXiv:1804.04366. 2018.
[11] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. International Conference on Learning Representations, 2019.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[13] Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design. arXiv preprint 1rxivv 00275, 2019.
[14] Emiel Hoogeboom, Victor Garcia Satorras, Jakub M Tomezak, and Max Welling. The convolu- tion exponential and generalized Sylvester flows. arXiv preprint ar arXiv:2006.0/1 2000000000,,,,, 2020.
[15] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[16] Michael Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Skatistis-Simullation and Computation, 19(2):433 450, 1990.
[17] Jöm-Henrik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive invariance causes adversarial vulnerability, arXiv:1871.00-40 2018
[18] Herman Kahn, Use of different Monte Carlo sampling techniques. Proceedings of Symposium on Monte Carlo Methods, 1955.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013.

---

[20] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, pages 10215-10224, 2018.
[21] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Infor- mation Processing Systems, pages 4743-4751 2016.
[22] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[23] David C Lay, Linear Algebra and its Applications. Pearson, 2006.
[24] Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger Grosse, and Preventing gradient attenuation in Lipschitz constrained convolutional networks. In Advances Neural Information Processing Systems, pages 15390-15402, 2019.
Jacobsen.
[25] David JC MacKay and Mark Gibbs. Density networks. Statistics and neural networks: advances the interface. Oxford University Press, Oxford, pages 129-144, 1999.
[26] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint 1802.05977 2018.
[27] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Hybrid Models with Deep and Invertible Features. International Conference on Machine Learning, 2019.
[28] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint 160iv:1609.03490 2016.
[29] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747-1756, 2016.
[30] George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pages 2338-2347, 2017.
[31] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows arXiv preprint
2015.
[32] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014.
[33] Oren Rippel and Ryan Prescott Adams. High-dimensional probability estimation with deep density models. arXiv preprint arXiv:1302.5125, 2013.
[34] Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. arXiv: 0t...6000000101 2016.
[35] John Skilling. The eigenvalues of mega-dimensional matrices. In Maximum Entropy and Bayesian Methods, pages 455 466. Springer, 1989.
[36] Esteban G Tabak and Cristina V Turner A family of nonparametric density estimation algo- rithms. Communications on Pure and Applied Mathematics, 66(2):145-164, 2013.
[37] Esteban G Tabak, Eric Vanden Sunden-Eijnden, et al. Density estimation by dual ascent of the log- likelihood. Communications in Mathematical Sciences, 8(1):217-233, 2010.
[38] Jakub M Tomezak and Max Welling. Improving variational auto-encoders using householder flow. arXiv preprint arXiv: 161 1,09630, 2016.
12


Reviews:
[{"review_id": "sOui16Fpt3", "reviewer": "Reviewer_psZ5", "summary": "This paper proposes to use a DenseNet as the residual function of Residual Flows. The paper carefully describes how to ensure the Lipschitz constant is bounded when performing the concatenation of representations inherent in DenseNets. The paper also introduces a new activation function called Concatenated LipSwish that may be of independent interest. Finally, empirical results show that the proposed i-DenseNets perform better than Residual Flows and other baselines.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Summary of review**\nOverall, this is a clearly written paper that proposes a novel model architecture, provides important insights into the choices and architecture, and gives good empirical results. \n\n**Strengths:**\n- Very clear discussion of prior works and core contributions including the learnable weight concatenation.\n\n- Proposed a careful incorporation of DenseNets as residual function fo Residual Flows where the Lipschitz constant can be controlled.\n\n- Proposed a new trainable activation function called concatenated LipSwish that preserves more information flow.\n\n- Outperforms standard baselines including Residual Flows and Flow++ when using uniform dequantization.\n\n- Interesting and useful analysis of activation functions, concatenation depth, and DenseNets vs ResNets to practically understand some key hyperparameters and architectural constraints.\n\n**Weaknesses:**\n- The actual empirical benefits over Residual Flows seems small (though it does seem to improve in all cases) so it is not clear that i-DenseNets should always be chosen over Residual Flows.\n\n- Related to above, no training time comparison was given.  Could you give wall-clock training times for Residual Flows vs i-DenseNets? This may be a key factor in determining which one to use in practice.  Also, could you provide a comparison for time for inference at test time?\n\n- The novelty is medium.  The analysis of CLipSwish is very good and may become a standard.  However, the other ideas are mainly careful synthesis of other ideas.\n\n**Other comments or questions**\n- I'd suggest emphasizing the difference of $h{n+1}$ more.  Maybe use a different symbol since it is very different then the other $h$'s.\n\n- I did not understand \"learnable weight concatenation\" until the section that gives the details.  Maybe try to describe that it is related to making the Lipschitz constant less than 1 for the dense layer.\n\n- It may be helpful to mention immediately that the CLipSwish function doubles the channel number. This is mentioned at the end of the section but having it earlier may help understanding.\n\n**Typos or other minor comments**\n- L320 \"boost performs\" -> \"boosts performance\"\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Reasonable discussion of possible negative societal impact.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.25"}, "extra_text": {}}, {"review_id": "rLObsLq4tYJ", "reviewer": "Reviewer_rhhy", "summary": "In the article, a new normalizing flow architecture, called invertible DenseNets (i-DenseNets), is introduced. It is an extension of residual flows with two new contributions. First, the invertible residual blocks are replaced by invertible dense blocks. Similar to residual blocks, the authors proof invertibility by computing the Lipschitz constant and guarantee that it is smaller than one by a suitable rescaling operation. Second, the LipSwish activation function is replaced by its concatenated cousin, which is rescaled as well to ensure invertibility.\nThe method is applied to three illustrative toy datasets as well as CIFAR10 and ImageNet32, where it outperforms competing methods and sets a new SOTA for flow models with uniform dequantization. They also use an i-DenseNet to perform hybrid modelling, where is outperform the competition as well.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "DenseNets are an important class of neural networks and modifying them to be invertible is an important contribution to machine learning. The authors describe their model very clearly and illustrate the architecture well. In secion 5, they also give interesting insights in why the individual parts of their model are important and lead a better performance in the experiments. However, their style to organize their argumentation is unusual. I would recommend moving some of the arguments made in section 5.1 to 5.3 to section 3 and show the respective experimental results at the beginning of section 4. Thereby, the reader would directly get a better understanding of the reason for the architecture choice. \nRegarding the experiments, the reported results are impressive. However, it would be interesting to apply this method to higher dimensional datasets as it is done in most articles that introduced the competing methods discussed in this paper. Also, running each experiments with multiple seeds would be desirable, although I acknowledge that this is often not the case in similar published articles. Also, the computational cost of training i-DenseNets especially compared to residual flows is not discussed, which is unfortunate since the fact that DenseNets require less computation than ResNets is stated as one main motivation of this work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation is that the model has not been applied to high resolution data and the computational cost required to train it is unclear.\nThe societal impact regarding the generation of misinformation is discussed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "h7e6PgRMpw", "reviewer": "Reviewer_Lrfi", "summary": "The paper extends Residual Flows from ResNet to DenseNet. In addition, the authors further improve the performance by modifying the Lipschitz activation. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, and the presentation is clear. However, I feel the contribution is incremental, and the performance gain is marginal. \n\n1. Since DenseNet has the same residual form F(x) = x + g(x), the authors inherit all important components (fixed-point iteration, Skilling-Hutchinson trace estimator, and \"Russian roulette\" strategy) from Residual Flow. Therefore, the extension to DenseNet is not complicated, which relies on the Lipscitzness of convex combination. In addition, the CLipSwish only modifies the original LipSwich activation slightly. \n\n2. Given the marginal performance gain (e.g., Table 3), it is beneficial to report both the memory and actual running time for both Residual Flow and i-DenseNet. Due to the sequential structure of DenseNet, I wonder if the DenseNet with the same parameters is more expensive than Residual Flow in practice.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discusses the limitations and potential societal impact in Section 5.4. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BO19snoSz", "reviewer": "Reviewer_n9rc", "summary": "The paper proposes a parameter-efficient DenseNet block and a CLipSwish (concatenated LipSwish) activation function for use with the DenseNet blocks in the residual flows setting.  The paper outperforms prior art (with uniform dequantization) in density estimation. Results are reported on CIFAR10 and ImageNet32 with rich ablation studies.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using densenet blocks and constraining its Lipschitz constant in the residual flow setting is interesting and novel. The paper is clearly written and easy to follow, with a few exceptions listed below. The work is quite self-contained and seems not to have loose ends. The main motivation for adopting densenet blocks is improving parameter efficiency and density estimation performance; both goals have been achieved according to the experiments.\n\nA few questions and remarks for improvement:\n\n- Fig.1 violet part is not clear\n- Eq.2 uses W to denote matrix (norm), while Fig.1 uses W to denote (convolutional) layers. While this is in line with the prior art (Invertible ResNet), perhaps the notation could be adjusted for clarity.\n- L76 missing subscript _2 of the norm\n- L93 could benefit from explaining how this is the same as Eq.2 from the DenseNet paper ($x_l = H_l([x_0,...,x_{l-1}]$)\n- L117,122 usage of 'dividing' is not precise\n- L134: missing {} around etas\n- L141 missing explanation for the constant\n- L162 in relation to Appendix:L584-586 could elaborate on the computation of the exact value (perhaps with a plot?)\n- L169 channel growth, as a term related to densenet blocks, perhaps could be mentioned earlier\n- L186 refers to [4] for baselines using uniform dequantization, but to introduce a reader to types of dequantizaton, a better reference is [13]\n- Sec.4.2 (in relation to CIFAR-10 and ImageNet32 experiments): how would the proposed method score in perceptual quality metrics, such as FID?\n- Sec.5.1 while the analysis of individual activation functions is interesting, it would also be interesting to evaluate the whole model using e.g., a histogram of the empirical Lipschitz constant (eLhist) [1]\n\n[1] Sanyal et al., Stable rank normalization for improved generalization in neural networks and GANs\n\n============== Post-rebuttal comment\n\nAfter reading all reviews and responses, I decided to keep my original rating (A).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Appropriately discussed in the \"Societal Impact\" section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an extension of Residual Flows to construct invertible deep neural nets. The paper features two technical contributions: The invertible residual blocks are replaced by invertible dense blocks, and the LipSwish activation function is replaced by a concatenated version. While these contributions can be considered marginal/incremental (reviewer Lrfi) and with medium novelty (reviewer psZ5), all reviewers agree the contributions offer enough value for being accepted. 

The reviewers acknowledged the author response and they engaged in committee discussions. No serious concerns surfaced during the review process, and the reviewers reached a consensus recommendation to accept the paper.
Decision: Accept (Poster)