Densely connected normalizing flows
Authors: Matej Grcić
Ivan Grubišić
Siniša Šegvić
Venue: NeurIPS 2021 Poster (2021)
Keywords: Density estimation
Normalizing flows
Image generation
DenseFlow
Densely connected couplings

Abstract: Normalizing flows are bijective mappings between inputs and latent representations with a fully factorized distribution. They are very attractive due to exact likelihood evaluation and efficient sampling. However, their effective capacity is often insufficient since the bijectivity constraint limits the model width. We address this issue by incrementally padding intermediate representations with noise. We precondition the noise in accordance with previous invertible units, which we describe as cross-unit coupling. Our invertible glow-like modules increase the model expressivity by fusing a densely connected block with Nyström self-attention. We refer to our architecture as DenseFlow since both cross-unit and intra-module couplings rely on dense connectivity. Experiments show significant improvements due to the proposed contributions and reveal state-of-the-art density estimation under moderate computing budgets.

Densely connected normalizing flows
Matej Greié, Ivan Grubišié and Siniša Segvié
Faculty Electrical Engineering and Computing
University of Zagreb
matej grcicffer.cee... hr ivan. bvan.grubisicdfer. hr sinisa.segvic@fer hr
Abstract
Normalizing flows are bijective mappings between inputs and latent representations with a fully factorized distribution. They are very attractive due exact likelihood evaluation and efficient sampling. However, their effective capacity is often insuffi cient since the bijectivity constraint limits the model width. We address this issue by incrementally padding intermediate representations with noise. We precondition the noise in accordance with previous invertible units, which we describe as cross- unit coupling. Our invertible glow-like modules increase the model expressivity by fusing densely connected block with Nyström self-attention. We refer to architecture DenseFlow since both cross-unit and intra-module couplings rely on dense connectivity. Experiments show significant improvements due to the proposed contributions and reveal state-of-the-ar density estimation under moderate computing budgets
Introduction
One of the main tasks of modern artificial intelligence is to generate images, audio waveforms, and natural-language symbols. To achieve the desired goal, the current state of the art uses deep compositions of non-linear transformations [ known as deep generative models
6 77.
Formally, deep generative models estimate unknown data distribution PD given by set of i.i.d. samples {æI...., }. The data distribution is approximated with a model distribution pa defined by the architecture of the model and a set of parameters 0. While the architecture is usually handcrafted, the set of parameters obtained by optimizing the likelihood across the training distribution PD:
argmin Ez-pol In po(x)]- 8E0
(1)
Properties of the model (e.g. efficient sampling, ability evaluate likelihood etc.) directly depend on the definition of ( or decision to avoid it. Early approaches consider unnormalized distribution
3 which usually requires MCMC-based sample generation 8 9 10 with long mixing times. Alter-
natively, the distribution can be autoregressively factorized 11 which allows likelihood estimation and powerful but slow sample generation. VAEs use factorized variational approximation of the latent representation, which allows to learn an autoencoder by optimizing lower bound of the likelihood. Diffussion models [12 [3 [4] learn reverse diffusion process, which is a fixed Markov chain that gradually adds noise to the data in the opposite direction of sampling until the signal is destroyed. Generative adversarial networks [5] mimic the dataset samples by competing in minimax game. This allows to efficiently produce high quality samples [15), which however often do not span the entire training distribution support [16). Additionally, the inability to "invert" the generation process in any meaningful way implies inability to evaluate the likelihood.
Contrary previous approaches, normalizing flows mapping to a predefined latent distribution p(z)) typically multivariate Gaussian. Given the bijection
model the likelihood using bijective
Code available at:
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

fe. the likelihood is defined using the change of variables formula:
Oz
pe(x) = p(=) det
= fo(x).
dæ
This approach requires computation of the Jacobian determinant (det de Therefore, during the construction of bijective transformations, a great emphasis placed on tractable determinant compu-
tation and efficient inverse computation [88 [9] Due to these constraints, invertible transformations require more parameters to achieve similar capacity compared to standard NN building blocks [20]. Still, modeling po using bijective formulation enables exact likelihood evaluation and efficient sample generation, which makes this approach convenient for various downstream tasks [2 22 23
The bijective formulation 2 implies that the input and the latent representation have the same dimensionality. Typically, convolutional units of normalizing-flow approaches [18] internally inflate the dimensionality of the input, extract useful features, and then compress them back the original dimensionality. Unfortunately, the capacity of such transformations is limited by input dimensionality
This issue he addressed by expressing the model sequence of bijective transformations 18 However, increasing the depth alone is a suboptimal approach to improve capacity of deep model [25). Recent works propose widen the flow by increasing the input dimensionality [24] 26]
24
We propose an effective development of that idea which further improves the performance while relaxing computational requirements.
We increase the expressiveness of normalizing flows by incremental augmentation of intermediate latent representations with Gaussian noise. The proposed cross-unit coupling applies affine transformation to the noise, where the scaling and translation are computed from a set of previ- ous intermediate representations. In addition, improve intra-module coupling by proposing transformation which fuses the global spatial context with local correlations. The proposed image- oriented architecture improves expressiveness and computational efficiency. Our models set the new state-of-the-at result in likelihood evaluation on ImageNet32 and ImageNet64.
2 Densely connected normalizing flows
We present recursive view normalizing flows and propose improvements based incremental augmentation of latent representations, and densely connected coupling modules paired with self- attention. The improved framework is then used develop image-oriented architecture, which we evaluate in the experimental section.
2.1 Normalizing flows with cross-unit coupling
Normalizing flows (NF) achieve their expressiveness by stacking multiple invertible transformations
[8]. We illustrate this with the scheme (3) where each two consecutive latent variables #,-1 and 2; connected via a dedicated flow unit ft. Each flow unit fi is bijective transformation with parameters A which we omit to keep notation uncluttered. The variable 2y is typically the input æ drawn from the data distribution pz()).
2K, ZK 1O0,
(3)
Following the change of variables formula, log likelihoods of consecutive random variables z; and 2s+1 can be related through the Jacobian of the corresponding transformation J [8]
In,(z,) In p(=+4)) In det J s-mil-
(4)
This relation can be seen as a recursion. The term In p(z(41) can be recursively replaced either with another instance of (4) or evaluated under the latent distribution, which marks the termination step.
This setup characteristic for most contemporary architectures [1
18 19 27
The standard NF formulation can be expanded by augmenting the input by a noise variable e; 24 (26). The noise e, subjects some known distribution p²(e;), e.g. multivariate Gaussian. We further improve this approach by incrementally concatenating noise to each intermediate latent representation z; A tractable formulation of this idea be obtained by computing the lower bound of the likelihood p(2;) through Monte Carlo sampling of e;:
In p((())) 2 (e,r [lnp(z,, Inp"(e:)).
(5)

---

The learned joint distribution p(zz,, e;) approximates the product of the target distributions p" (z;) and p" (e;), which is explained in more detail in Appendix D We transform the introduced noise e; with glement-wise affine transformation. Parameters of this transformation are computed by learned non-linear transformation (z(ci)) of previous representations zςi = [=0....,,, 2.-1]. The resulting layer h2 can be defined as:
(aug) h Bi z<)) [2 Zi
e, + nl),
(k, g((z)))
(6)
Square brackets [:-]] denote concatenation along the features dimension. In order to compute the likelihood for (z C2), we need the determinant of the jacobian
az (ang)
0
diag(o))
2 e,
Now we can express p(z,, e,) in terms of p(z ( according to
Inp(z;, e,) In p( (mm))) + In det di)
(8)
We join equations
5 and 8 into single step:
In (2,) 2 Ee,-p" (e.) [In p(z - Inp"(e;) + In det diag(or)]]
(9)
We refer the transformation h4 as cross-unit coupling since it acts as an affine coupling layer [17 over group of previous invertible units. The latent part of the input tensor is propagated without change, while the noise part is linearly transformed. The noise transformation can be viewed reparametrization of the distribution from which we sample the noise 44. Note that can conveniently recover z; from 2; ((uu)) by removing the noise dimensions. This step is performed during model sampling.
Fig. compares the standard normalizing flow (a) normalizing flow with input augmentation [2 (b) and the proposed densely connected incremental augmentation with cross-unit coupling (c). Each flow unit fPP consists of several invertible modules mi, and cross-unit coupling hç The main novelty of our architecture is that each flow unit J increases the dimensionality with respect to its predecessor fPr Cross-unit coupling h; augments the latent variable z; with affinely transformed noise e,, Parameters of the affine noise transformation are obtained by an arbitrary function g. which accepts all previous variables ø<i- Note that reversing the direction does not require evaluating 9i since we are only interested in the value of z,. For further clarification, we show the likelihood computation for the extended framework.
p(e(x)
[x.e]
2,, (ug)
Z Z
Flow unit
Flo unit!
Flw uni
Invertible unit (m.)
invertibie unit on,)
Invertbble unit mn.)
Inventible module m._
Inverible m...
Inverible m
Invertble module m.,
Invertible module
invertible module
Z, e N(O, 1)
Flow unit
Flow unit
Cross-untt coupling (h)
[2,0]
a()
Invertible unit (mg)
Invertible untt (n.)
module m.
Invertibe module my
(1.0)
[
:
Invonible module m.
Invertible module m
(mg)
(a)
Figure 1: Standard normalizing flow 17 18 and the proposed incremental augmentation with cross-unit coupling (c). Unlike (b) which adds noise only to the input, (c) adds noise to the output of every unit except the last.
(a), normalizing flow with augmented input 24 (b),
3

---

Example 1 (Likelihood computation) Let mı and m2 be the bijective mappings from 2u to 21 and =1_
((mu)) to 22. respectively. Let b be the cross-unit coupling from to (awg) = = 0 de pi]-
Assume a and f are computed by any non-invertible neural network 91- The network accepts zo
as the inpus. We calculate log likelihood of the input 2o according to the following sequence of equations: [transformation, cross-unit coupling, transformation, termination).
In p(z)) in p(1) + In det s.
(10)
Inp(z1)= 2 (l ("me)) - In p(e)) ) + In det diagg())). (a, 11) - (z0))
(11)
In Inp(2) 11 ( In p x2) + in det s/2b. pmp(z2) = 1,,;
(12)
(13)
We approximate the expectation using MC sampling with a single sample during training and a few hundreds of samples during evaluation to reduce the variance of the likelihood. Note however that our architecture generates samples with a single pass since the inverse does not require MC sampling.
We repeatedly apply the cross-unit coupling hu2 throughout the architecture to achieve incremental augmentation of intermediate latent representations. Consequently, the data distribution is modeled in latent space of higher dimensionality than the input space (24 26]. This enables better alignment of the final latent representation with the NF prior. We materialize the proposed expansion of the normalizing flow framework by developing an image-oriented architecture which we call DenseFlow.
2.2 Image-oriented invertible module
We propose a glow-like invertible module (also known as step of flow (I9) consisting of activation normalization, 1 X 1 convolution and intra-module affine coupling layer. The attribute "intra-module" emphasizes distinction with respect to cross-unit coupling. Different than in the origi- nal glow design, our coupling network leverages advanced transformations based on dense connectivity and fast self- attention. All three layers designed to capture complex data dependencies while keeping tractable Jacobians and efficient inverse computation. For completeness, start by reviewing elements of the original glow module 199..
ActNorm
Convolution 1x1
Intra-module Caupling
X,
X,
Projection
ActNorm [9] is an invertible substitute for batch normaliza- tion 300 performs affine transformation with per-channel scale and bias parameters:
Nyström
Dense Block
SA
O
yr C Diij +
(14)
Scale and bias are calculated as the variance and mean of the initial minibatch.
Blend
Invertible 1 1 Convolution is a generalization of chan- nel permutation 19). Convolutions with 1 x kernel are not invertible by construction. Instead, combination of orthogonal initialization and the loss function keeps the ker- nel inverse numerically stable. The normalizing flow loss maximizes In det JJ which is equivalent to maximizing E, In (Ai), where X; are cigenvalues of the Jacobian. Main- taining a relatively large amplitude of the eigenvalues en- sures stable inversion. The Jacobian of this transformation can be efficiently computed by LU LU-decomposition [19].
(s.t)
Affine
Y,
A glow-like module m consist of ActNorm, 1xl convolution and intra-module affine coupling. The proposed intra-module coupling fuses the global context recovered by fast self-attention (28) and local correla- tions extracted by densely connected convolutions 29
Figure 2:
Affine Coupling [18] splits the input tensor æ channel-wise into two halves 21 and æ2. The first half is propagated without changes, while the second half is linearly trans- formed (5). The parameters of the linear transformation are calculated from the first half. Finally, the two results are
concatenated as shown in Fig. 2
y1 Æ1 y2
x2 t, (8,t) coupling_net(z;).
(15)

---

Parameters s and are calculated using a trainable network which is typically implemented as residual block [18]. However, this setting can only capture local correlations. Motivated by
recent advances in discriminative architectures 22
31
32
we design our coupling network to
fuse both global context and local correlations as shown Fig. 2 First, we project the input into a low-dimensional manifold. Next, we feed the projected tensor to densely-connected block [29) and self-attention module [31 33). The densely connected block captures the local correlations 34], while the self-attention module captures the global spatial context. Outputs of these two branches are concatenated and blended through a BN-ReLU-Conv unit. As usual, the obtained output parameterizes the affine coupling transformation (5] Differences between the proposed coupling network and other network designs are detailed in related work.
It is well known that full- full-fledged self-attention layers have very large computational complexity. This is especially true in the case normalizing flows which require many coupling layers and large latent dimensionallttes. We alleviate this issue by approximating the keys and queries with their low-rank approximations according to the Nystrom method 28
2.3 Multi-scale architecture
We propose an image-oriented architecture which extends multi-scale Glow [9] with incremental augmentation through cross-unit coupling. Each DenseFlow block consists several DenseFlow units and resolves a portion of the latent representation according to decoupled normal distribution [8]. Each DenseFlow unit SPF consists of N glow-like modules (m, m, ..... o m,,1) and cross-unit coupling (h,). Recall that each invertible module ma, contains the affine coupling network from Fig. as described Section 2.2
The input to each DenseFlow unit is the output of the previous unit augmented with the noise and transformed in the cross-unit coupling fashion. The number of introduced noise channels is defined as the growth-rate hyperparameter. Generally, the number of invertible modules in latter DenseFlow units should increase due enlarged latent representation. We stack M DenseFlow units to form a DenseFlow block. The last invertible unit in the block does not have the corresponding cross-unit coupling. We stack multiple DenseFlow blocks form a normalizing flow with large capacity. We decrease the spatial resolution and compress the latent representation by introducing a squeeze-and-droy modules [18] between each two blocks. A squeeze-ande and-drop module applics space-to- ppace-to-channel reshaping and resolves half of the dimensions according to the prior distribution. We denote the developed architecture as PenseFloowll modules while k denotes the growth rate. The developed architecture uses two independent levels of skip connections. The first level (intra-module, is formed of skip connections inside every coupling network. The second level (cross-und unit) connects DenseFlow units at the top level of the architecture.
where is the total number of invertible
Fig. 3 shows the final architecture of the proposed model. Gray squares represent DenseFlow units. Cross-unit coupling is represented with blue dots and dashed skip connections. Finally, squeeze- and-drop operations between successive DenseFlow blocks are represented by dotted squares. The proposed DenseFlow design applies invertible but less powerful transformations (e.g. convolution 1 x 1) on tensors of larger dimensionality. On the other hand, powerful non-invertible transformations
Cross-unit coupling (h)
DenseFlow Unit (lT))
Glow-like module (m,)
Invertible unit (m)
(1FF)
Squeeze&Drop
z.~((O I)
DenseFlow Block
(drop) N(O, I)
Figure 3: The proposed DenseFlow architecture. DenseFlow blocks consist of DenseFlow units (f.F) and a Squeeze-and-Drog module [8]. DenseFlow units are densely connected through cross-unit coupling (h₂). Each DenseFlow unit includes multiple invertible modules (m4,) from Fig. 2
5

---

such as coupling networks perform most of their operations on lower-dimensional tensors. This leads to resource-efficient training and inference.
3 Experiments
Our experiments compare the proposed DenseFlow architecture with the state of the art. Quantitative experiments measure the accuracy of density estimation and quality of generated samples, analyze the computational complexity of model training, as well as ablate the proposed contributions. Qualitative experiments present generated samples.
3.1 Density estimation
We study the accuracy of density estimation on CIFAR-10 [35,, ImageNet [36] resized 32 x 32 and 64 x 64 pixels and CelebA (37. Tab. compares generative performance of various contemporary models. Models are grouped into four categories based on factorization of the probability density. Among these, autoregressive models have been achieving the best performance. Image Transformer 38 has been the best on ImageNet32, while Routing transformer (39) has been the best on Ima- geNet64. The fifth category contains hybrid architectures which combine multiple approaches into single model. Hybrid models have succeeded to outperform many factorization-spei spplcoffissssiiiieeeieeeeeiiiiie architectures.
The bottom row of the table presents the proposed DenseFlow architecture. We use the same DenseFlow-74-10 model in all experiments except ablations in order to illustrate the general applica- bility of our concepts. The first block of DenscFlow-74-10 uses 6 units with 5 glow-like modules in each DenseFlow unit, the second block uses units with modules, while the third block uses a single unit with 20 modules. We use the growth rate of 10 in all units. Each intra-module coupling starts with a projection to 48 channels. Subsequently, it includes a dense block with densely connected layers, and the Nyström self-attention module with a single head. Since the natural images are discretized, we apply variational dequantization 22 to obtain continuous data which suitable for normalizing flows.
On CIFAR-10, DenseFlow reaches the best recognition performance among normalizing flows, which equals to 2.98 bpd. Models trained on ImageNet32 and ImageNet64 achieve state-of-the-at density estimation corresponding 3.63 and 3.35 bpd respectively. The obtained recognition performance is significantly better than the previous state of the art (3.77 and 3.43 bpd). Finally, our model achieves competetive results on the CelebA dataset, which corresponds to 1.99 bpd. The likelihood is computed using 1000 MC samples for CIFAR- 10 and 200 samples for CelebA and ImageNet. The reported results are averaged over three runs with different random seeds. One MC sample is enough for accurate liglliieldoood estimation since the per-example standard deviation is already about 0.01 bpd and a validation dataset size N additionally divides it by vN. The reported results are averaged over seven runs with different random seeds. Training details are available in Appendix
3.2
Computational complexity
Deep generative models require an extraordinary amount of time and computation to reach of the-art performance. Moreover, contemporary architectures have scaling issues. For example, VFlow requires 16 GPUs and two months to be trained on the ImageNet32 dataset, while the NVAE 56 requires GPUs and about days. This limits downstream applications of developed models and slows down the rate of innovation in the field. In contrast, the proposed DenseFlow design places a great emphasis on the efficiency and scalability.
24
Tab. compares the time and memory consumption of the proposed model with respect to competing architectures. We compare our model with VFlow 24 and NVAE [56] due similar generative performance on CIFAR-10 10 and CelebA, respectively. We note that RTX 3090 and Tesla V100 deliver similar performance, while RTX2080Ti has slightly lower performance compared the previous two. However, since we model relatively small images, GPU utilization is limited by I/O performance. In our experiments, training the model for one epoch on any of the aforementioned GPUs had similar duration. Therefore, we can still make fair comparison. Please note that we are unable to include approaches based on transformers (3 model training.
58
since they do not report the computational effort for

---

Table 1: Likelihood evaluation (in bits/dim) on standard datasets.
C-FAR-10
ImageNet
CelebA
ImageNet
Method
32x32
32x32
64x64
64x64
Conv Draw 40
3.58
4,40
4.10
+VAE+
3,38
Variational
IAF-VAE
142
3.11
Autoencoders
BIVA 443
3.08
3.96
2.48
CR-NVAE
2.51
1.86
DDPM [13
3,70
Diffusion
UDM (RVE) ST
45
3.04
1.93
models
Imp. DDPM [46
2.94
3.53
VDM 47
2.65
3.72
3.40
Gated PixeICNN
48
3.03
3.83
3.57
PixeIRNN
3.00
3.86
3.63
PixeICNN+
2.92
Autoregressive
Image Transformer
38
2.90
3.77
2.61
Models
PiXelSNAIL 49
2.85
3.80
SPN 50
3.85
3,53
Routing transformer
39
2.95
3.43
Real NVP [8
3.49
4.28
3.02
3,98
GLOW [9
3.35
4.09
3.81
Wavelet Flow 51
4.08
3.78
Normalizing
Residual Flow 52
3.28
4.01
3.78
Flows
i-DenseNet 53
3.25
3.98
Flow++ 27
3.08
3.86
3.69
ANF (26
3.05
3.92
3.66
VFlow 24
2.98
3.83
3.66
mAR-SCF [54]
3.22
3,99
3,80
MaCow 55
3.16
3.69
Hybrid
SurVAE Flow
34
3.08
4.00
3.70
Architectures
NVAE [56]
2.91
3.92
2.03
PiXeIVEE+ 57
2.90
8-VAE [58
2.83
3.77
DenseFlow-744 74-10 (ours)
2.98
3.63
1.99
3.35
Table 2: Comparative analysis of the computational budget for training contemporary methods.
DenseFlow decreases the training complexity by an order magnitude.
Dataset
Model
Params
GPU type
GPUs
Duration (h)
BPD
Vllow 24
38M
RTX 2080Ti
16
~500
2.98
CIFAR-10
NVAE
56
257M
Tesla V100
55
2.91
DenseFlow-744
130M
RTX 3090
250
2.98
VFlow 24
38M
Tesla V100
16
1440
3.83
ImageNet32
NVAE 56
Tesla V100
24
70
3.92
DenseFlow. 74-10
130M
Tesla V100
310
3.63
VFlow 24
n/a
n/a
n/a
CelebA
NVAE 56
153M
Tesla V100
8
92
2.03
DenseFlow-
130M
Tesla V100
224
1.99
3.3 Image generation
Normalizing flows can efficiently generate samples. The generation performed in two steps. We first sample from the latent distribution and then transform the obtained latent tensor through the inverse mapping. Fig. 4 shows unconditionally generated images with the model trained on ImageNet64. Fig. Shshows generated images using the model trained on CelebA. In this case, we modify the latent
distribution by temperature scaling with factor 0.8 38
56 Generated images show diverse
hairstyles, skin tones and backgrounds. More generated samples can be found in Appendix G The

---

developed DenseFlow- 10nseFloo-74-10 model generates minibatch of 128 CIFAR-10 10 samples for 0.96 sec. The result averaged over 10 runs on RTX 3090.
Figure 4: Samples from DenseFlow-74-10 trained on ImageNet 64 x 64.
Figure 5: Samples from DenseFlow-744 1001- trained on CelebA
3.4
Visual quality
The ability to generate high fidelity samples is crucial for real-world applications generative models. We measure the quality of generated samples using the FID score [39]. The FID score requires large corpus of generated samples in order to provide an unbiased estimate. Hence, we generate 50k samples for CIFAR-10 10, and CelebA, and 200k samples for ImageNet. The samples are generated using the model described in Sec. 3.1 The generated ImageNet32 samples achieve FID score of 38.8, the CelebA samples achieve 17.1 and CIFAR-10 10 samples achieve 34.9 when compared to the corresponding training dataset. When compared with corresponding validation datasets, we achieve 37.1 CIFARIO and 38.5 on ImageNet32.
Tab. shows comparison with FID scores of other generative models. Our model outperforms contemporary autoregressive models [7] [00] and the majority of normalizing flows [61 52 19]. Our FID score is comparable with the first generation GANs. Similar other NF models, the achieved
FID score is still an order of magnitude higher than current state of the art 62] The results for PixeICNN, DCGAN, and WGAN-GP are taken from 60
3.5 Ablations
Tab, Jebplorss the contributions of incremental augmentation and dense connectivity in cross-unit and mnra--mduul coupling transforms. We decompose cross-unie coupling into incremental augmentation of the flow dimensionality (column 1) and affine noise transformation (column 2). Column 3 ablates the proposed intra-module coupling network based on fusion of fast self-attention and a densely connected convolutional block with the original Glow coupling [19].
The bottom row of the table corresponds to DenscFlow-45-6 model. The first DenseFlow block has 5 DenseFlow units with 3 invertible modules per unit. The second DenseFlow block has 3 units with modules. while the final block has 15 modules top row of the table corresponds to the standard normalized flow [18 19 with three blocks and 15 modules per block. Consequently, all models have the same number invertible glow-like modules. All models are trained on CIFAR-10 10 for 300 epochs and then fine-tuned for 10 epochs. We use the same training hyperparameters for all models. The proposed cross-unit coupling improves the density estimation from 3.42 bpd (row 1) to 3.37 bpd (row 3) starting from model with the standard glow modules. When a model equipped with our intra-module coupling, cross-unit coupling leads to improvement from 3.14 bpd (row 4) to 3.07 bpd (row 6). Hence, the proposed cross-unit coupling
single unit. We use the growth rate of 6. The

---

Table 3: Evaluation of FID score on CIFAR-10.
Model
FID
Autoregressive
PixeICNN [7]
65.93
Models
PixelIQN 660
49.46
i-ResNet 61
65.01
Normalizing
Glow [9
46.90
Flows
Residual flow
52
46.37
ANF [26
30.60
DCGAN 15
60
37.11
GANs
WGAN-GP 63
60
36.40
DA-StyleGAN V2 62
5.79
VDM 47
4.00
Diffusion models
DDPM [3
3.17
UDM (RVE) + ST 45
2.33
SurVAF- flow
49.03
Hybrid
mAR-SCF [54]
33.06
Architectures
VAEBM [64]
12.19
DenseFlow 1o0--llow-----11-1 (ours)
34.90
improves the density estimation in all experiments. Both components of cross-unit coupling are important. Models with preconditioned noise outperform models with simple white noise (row vs row 3, and row vs row 6). comparison of rows 1-3 with rows 4-6 reveals that the proposed intra-module coupling network also yields significant improvements. We have performed two further ablation experiments with the same model. Densely connected cross-coupling contributes 0.01 bpd in comparison preconditioning noise with respect to the previous representation only. Self-attention module contributes 0.01 bpd with respect to the model with only DenseBlock coupling on ImageNet 32 32.
Table 4: Ablations on the CIFAR- 10 dataset with DenscFlow-45-6.
Latent variable augmentation
Pre-conditioned
Intra-module coupling with two-way fusion
#
BPD
noise
X
3.42
2
3.40
3
X
3.37
4
X
3.14
5
X
V
3.08
6
/
3.07
4 Related work
VFlow
increases the dimensionality of normalizing flow by concatenating input with random variable drawn from p(e x). The resulting optimization maximizes the lower bound (e)p"( [In p(x, e) (elx)), where each term implemented by a separate normalizing flow. Similarly, ANF 26 draws connection between maximizing the joint density p(æ;, e) and Both approaches augment only the input variable æ while we augment latent representations many times throughout our models.
24
lower-bound optimization
Surjective flows 34 decrease the computational complexity of the flow by reducing the dimensional- ity of deep layers. However, this also reduces the generative capacity. Our approach achieves better generative performance under affordable computational budget due to gradual increase of the latent dimensionality and efficient coupling.
Invertible DenseNets 65
apply skip connections within invertible residual blocks 66
However, this approach lacks closed-form inverse, and therefore can generate data only through slow iterative algorithms. Our approach leverages skip connections both in module couplings, and supports fast analytical inverse by construction.
cross-unit and intra-

---

Models with an analytical inverse allocate most of their capacity to coupling networks [7]] Early coupling networks were implemented residual blocks [18]. Recent work [27] increases the coupling capacity by stacking convolutional and multihead self-attention layers into gated residual [66 49]
However, heavy usage of self-attention radically increases the computational complexity. Contrary to stacking convolutional and self-attention layers in alternating fashion, the design of our network uses two parallel modules. Outputs the these two modules are fused into a single output. SurVAE 34] expresses the coupling network as a densely connected block [29] with residual connection. In comparison with (34), our intra-module coupling omits residual connectivity, decreases the number of densely connected layers and introduces parallel branch with Nyström self-attention. Thus, intra-module coupling fuses local cues with the global context.
Normalizing flow capacity can be further increased by adding complexity to the latent prior p(z). Autoregressive prior (54) may deliver better density estimation and improved visual quality of the generated samples. However, the computational cost of sample generation grows linearly with spatial dimensionality. Joining this approach with the proposed incremental latent variable augmentation could be a suitable direction for future work.
5
Conclusion
Normalizing flows allow principled recovery of the likelihood by evaluating factorized latent acti- vations. However, their efficiency is hampered by the bijectivity constraint since determines the model width. We propose to address this issue by incremental augmentation of intermediate latent representations. The introduced noise is preconditioned with respect preceding representations throughout cross-unit affine coupling. We also propose an improved design of intra-module coupling transformations within glow-like invertible modules. We express these transformations local correlations and the global context captured by self-attention. The resulting DenseFlow archi- tecture sets the new state-of-the-ard in likelihood evaluation on ImageNet while requiring a relatively small computational budget. Our results imply that the expressiveness of a NF does not only depend on latent dimensionality but also on its distribution across the model depth. Moreover, expressiveness of a NF can be further improved by conditioning the introduced noise with the proposed densely connected cross-umd unit coupling.
fusion of
Broader impact
This paper introduces new generative model called DenseFlow, which can be trained to achieve state- of-the-art density evaluation under moderate computational budget. Fast convergence and modest memory footprint lead to relatively small environmental impact of training and favor applicability to many downstream tasks. Technical contributions this paper do not raise any particular ethical challenges. However, image generation has known issues related to bias and faimess [67]. In particular, samples generated by our method will reflect any kind of bias from the training dataset.
Acknowledgements
This work has been supported by Croatian Science Foundation, grant 1P-2020-02-588 ADEPT. The first two authors have been employed on research projects KK.01.2.1.02 1........2.....1....... 1.0.00119919 DATACROSS and KK.01.2.1.02.
A-Unit funded by European Regional Development Fund and Gideon Brothers Itd. This work has also been supported by VSITE College for Information Technologies who provided access to 2 GPU Tesla-V 100 32GB. We thank Marin Oršić, Julije Ozegović, Josip Šaric as well as Jakob Verbeek for insightful discussions during early stages of this work.
10

---

References
[1] Yann LcCun, Yoshua Bengio, al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
//wuw. www.eedepplearniggooo...oo
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 448-455, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 2009. PMLR.
[4] Diederik P. Kingma and Max Welling. Auto-cncoding variational bayes. In 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[5] lan J. Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. Commun. ACM, 63(11):139 144, 2020.
[6] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceed- ings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1530-1530 1538. JMLR.org, 2015.
[7] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747-1756. PMLR, 2016.
[8] Rémi Bardenet, Arnaud Doucet, and Christopher C. Holmes. On markov chain monte carlo methods for tall data. J. Mach. Learn. Res., 18:47:1-47:433 2017.
[9] Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 177811777) 2002.
[10] Radford M Neal ct al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
[11] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn+++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017.
[12] Jascha Sohl-Dickstein, Fric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 2256-2265. JMLR.org. 2015.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-1 2020, virtual, 2020.
[14] Jiaming Song, Chenlin Meng, and Stefano Ermon. abs/2010.02502, 2020.
Denoising diffusion implicit models. CoRR,
[15] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convo- lutional generative adversarial networks. In 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[16] Thomas Lucas, Konstantin Shmelkov, Karteek Alahari, Cordelia Schmid, and Jakob Verbeek. Adaptive density estimation for generative models. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 11993-12003, 2019.
[17] Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-lincar independent components estimation. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015.
[18] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[19] Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Informaionn Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pages 2018.

---

[20] Jömn-Henrik Jacobsen, Amoll W. M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In 6th International Conference on Learning Representations. ICLR 2018, Vancouver, BC, Canada, April May 3, 2018, Conference Truck Proceedings, 2018.
[21] Jie Ren, Peter Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark DePristo, Joshua V. Dillon, and Balaji Lakshninarayanan. Likelihood ratios for out-of- distriibint detection. In Advances in Neural Information P Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 14680-14691, 2019.
[22] Eric T. Nalisnick. Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan. Hybrid models with deep and invertible features. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume of Proceedings of Machine Learning Research, pages 4723-4732. PMIR, 2019.
[23] Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák, Neural importance sampling. ACM Trans. Graph.. 38(5): 14:11:45:::114 2019.
[24) Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. Vflow: More expressive generative flows with variational data augmentation. In International Conference on Machine Learning, pages 1660-1669. PMLR, 2020.
[25] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 Proceedings of Machine Learning Research, pages 6105-6114. PMLR, 2019.
[26] Chin- Wei Huang, Laurent Dinh, and Aaron Courville, Augmented normalizing flows: Bridging the gap between generative flows and latent variable models. arXiv preprint arXiv:2002.07710 2020.
[27] Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pages 2722 2730. PMLR, 2019.
[28] Yunyang Xiong, Zhanpeng Zeng. Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: nyström-based algorithm for approximating self-attention. CoRR, abs/2102.03902, 2021.
[29] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pages 2261-2269, IEEE Computer Society, 2017.
[30] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 448-456. oog. 2015.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 6008, 2017.
[32] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 7794-7803, 2018.
[33] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010.1 19929, 2020.
[34] Didrik Niclsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling. Survae flows: Surjections to bridge the gap between vaes and flows. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020. virtual, 2020.
[35] Alex Krizhevsky, Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (ICC),, 115(3)211--552 2015.
[37] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
12

---

[38] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran, Image transformer. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stuckhelmsmüsssn, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4052 1552-0061. PMLR, 2018.
[39] Aurko Roy. Mohammad Saffar. Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Trans. Assoc. Comput. Linguistics, 9:53-66 68, 2021.
[40] Karol Gregor, Frederic Besse. Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. In Advances in Neural Information Processing Systems 29; Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3549-3557, 2016.
[41] Arash Vahdat, William G. Macready, Zhengbing Bian, Amir Khoshaman, and Evgeny Andriyash. DVAE++: discrete variational autocncoders with overlapping transformations. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 5042-5051, PMLR, 2018.
[42] Diederik Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow. arXiv preprint arXiv: 04944, 2016.
[43] Lars Maalee, Marco Fraccaro, Valentin Liévin, and Ole Winther, BIVA: A very deep hicrarchy of latent variables for generative modeling. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 6548-6558, 2019.
[44] Samarth Sinha and Adji B. Dieng. Consistency regularization for variational auto-encoders. CoRR, abs/2105.1488 14859, 2021.
[45] Dongjun Kim, Seungjac Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Score matching model for unbounded data score. CoRR, abs/2106.05527, 2021.
[46] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. CoRR, abs/2102.09672, 2021.
[47] Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. CoRR, abs/2107.000 10.600 2021.
[48] Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4790-4795 2016.
[49] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autoregressive generative model. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmäxsan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 863-871. PMLR, 2018.
[50] Jacob Mcnick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019, 2019.
[51] Jason J. Yu, Konstantinos G. Derpanis, and Marcus A. Brubaker. Wavelet flow: Fast training of high reso- lution normalizing flows. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[52] Tian Qi Chen, Jens Behrmann, David Duvenaud, and Jöm-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9913-9923, 2019.
[53] Yura Perugachi-DDazz Jakub M Tomezak, and Sandjai Bhulai. Invertible densenets with concatenated lipswish. arXiv preprint arXiv:2102.0266 02..029.. 2021.
[54] Apratim Bhattacharyya, Shweta Mahajan, Mario Fritz, Bernt Schiele, and Stefan Roth. Normalizing flows with multi-scale autoregressive priors. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 8412-8421. Computer Vision Foundation IEEE, 2020.
[55] Xuezhe Ma. Xiang Kong, Shanghang Zhang, and Eduard H. Hovy. Macow: Masked convolutional generative flow. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pages 5891-5901 2019.
[56] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 12, 2020, virtual, 2020.
13

---

[57] Hossein Sadeghi, Evgeny Andriyash. Walter Vinci. Lorenzo Buffoni, and Mohammad H. Amin. Pixelvae++: Improved pixelvae with discrete prior. CoRR, abs/1908.09994 2019.
[58] Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes, In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
[59] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6626-6637, 2017.
[60] Georg Ostrovski, Will Dabney, and Rémi Munos. Autoregressive quantile networks for generative modeling. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmäxsson, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1933-3942, PMLR, 2018
[61] Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference Machine Learning, pages 573-582. PMLR. 2019.
[62] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data- efficient GAN training. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 2020, virtual, 2020.
[63] Ishaan Guirajani, Faruk Ahmed, Martín Arjovsky. Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5767-5777, 2017.
[64] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. VAEBM: A symbiosis between variational autoencoders and energy-based models. CoRR, abs/2010.000654 2020.
[65] Yura Perugachi-Diic Diaz, Jakub M. Tomezak, and Sandjai Bhulai. Invertible densenets. In 3rd Symposium on Advances in Approximate Bayesian Inference, pages 1-11, 2020.
[66]
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In 6th International Conference on Learning Representations. ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Truck Proceedings, 2018.
[67] Ramya Srinivasan and Ajay Chander. Biases in Al systems. Commun. ACM, 64(8):44 49, 2021.
[68] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning, arXiv preprint arXiv:1910.0999 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes] See Abstract and last paragraph of Section
(b) Did you describe the limitations your work? [Yes] See Appendix A
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Broader impact (Section and Environmental impact (Appendix) [B)
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in supplemental material or URL)? [Yes] Code available at latpss Searccic/Deeeeoo..
//github.
(b) Did you specify the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section Appendix(C
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] The results are averaged over multiple seeds. The variance is reported in Section [
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs,
internal cluster, cloud provider)? [Yes] See Section 3 Tab 2
Appendix B
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) If your work uses existing asscts, did you cite the creators? [Yes] See Appendix
(b) Did you mention the license of the assets? [Yes] See Appendix F
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See the official code release.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're us- ing/eurating? [N/A]
(e) Did discuss whether the data you are using/curating contains personally identifiable informa- tion offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A]
(b) Did you descrihe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "sWmVbsKyU_S", "reviewer": "Reviewer_RyGz", "summary": "The paper proposes to use incremental augmentation, gradually increasing dimensionality towards the latent space, in normalizing flows. The augmentation distribution at each level is conditioned on previous feature maps. This is referred to as \"cross-unit coupling\". Additionally, the authors introduce a new neural architecture for use in the usual coupling layers, referred to as \"intra-unit coupling\".\nThe result is an efficiently trained flow with state-of-the-art likelihoods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper extends the augmentation in [1,2] to multiple layers where the augmentation distributions are conditioned on previous feature maps in a similar fashion to DenseNets. Furthermore, a novel neural architecture is proposed for the coupling layers.\n\nThe proposed improvements are straightforward extensions of existing methodology, but with surprisingly good performance. The proposed methodology is clearly laid out, first introducing the \"cross-unit coupling\", before introducing the image-oriented neural architecture with two-way fusion using both convolutions and self-attention.\nExperiments are quite extensive: Comparisons on likelihood performance on 4 image datasets, as well as comparisons of computational complexity and sample quality. Finally, there is also an ablation study on the different model component's effect on likelihood. \n\nThe main strong point of the paper is the state-of-the-art likelihood performance achieved using modest computational resources compared to earlier work. Given this, I am recommending acceptance. However, the paper does have some rough edges and points that could be improved. I outline a few suggestions for this below.\n\nSome suggestions for fixing/improving the paper:\n- Line 14: \"One of the uttermost goals [...]\": This statement is maybe a bit strong and could be rephrased.\n- Line 27: Although the connections you point out are correct, I'm not sure I'd call diffusion models as VAEs since there is no \"autoencoding\" going on.\n- Line 29: \"[...] GANs ignore the factorization of the likelihood\": What is meant by this? I think the description of GANs could be improved. The inability to evaluate the likelihood comes mainly from the inability to \"invert\" the generation process in any meaningful way.\n- Eq. 11: $e_i \\rightarrow e_1$ and $\\log p(e_1) \\rightarrow \\log p^*(e_1)$.\n- Line 108: \"[...] a few hundred of samples during evaluation to reduce the variance of the likelihood\": From the code, it seems the importance weighted bound was used which also provides a tighter bound on the likelihood, not just lower variance.\n- Table 2: Do you also have access to the number of parameters? If so, this would improve the table.\n- Line 241: While one could argue about which approach is best, it is my understanding that common practice is to compute FID by comparing the training data distribution. \n- Line 263: bdp -> bpd\n- Line 276: $\\mathbb{E}_{\\boldsymbol{e} \\sim p(\\boldsymbol{e}|\\boldsymbol{x})} \\left[ \\log p(\\boldsymbol{e}, \\boldsymbol{x}) - \\log p(\\boldsymbol{e}|\\boldsymbol{x}) \\right] = \\log p(\\boldsymbol{x})$ and so is not a lower bound.  \n  - I believe you want to write $\\mathbb{E}_{\\boldsymbol{e} \\sim p^*(\\boldsymbol{e}|\\boldsymbol{x})} \\left[ \\log p(\\boldsymbol{e}, \\boldsymbol{x}) - \\log p^*(\\boldsymbol{e}|\\boldsymbol{x}) \\right]$\n\n\n\n[1] Huang et al. 2020, Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models  \n[2] Chen et al. 2020, VFlow: More Expressive Generative Flows with Variational Data Augmentation", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limititations are adequately discussed in App. A and environmental impact in App. B. \nAs for negative societal imapct, the authors mention that their research could be used for malicious purposes, they do not discuss any particular examples.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "_lEpAVMBcm-", "reviewer": "Reviewer_EGVd", "summary": "This work addresses the limitation posed by the bijectivity constraints in normalizing flows by padding  noise at each layer of the flow. \nMC sampling is used to approximate the likelihood where 1 MC sample is used for training and multiple MC samples are used for likelihood evaluation. Experiments for density estimation are performed on CIFAR-10, ImageNet-32 ,CelebA -64, ImageNet-64 datasets where the method yields state of the art density estimates. The method is shown to be computationally efficient with better sample quality.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* The work is incremental as the idea of padding with the noise dimensions to overcome the constraints associated with the dimensionality preservation have been studied in INNs and VFlow. \n\n* Overall the paper is well written and the experiments are detailed and easy to follow. Relevant details wrt to the experimental setup are included.  Tables 1, 2 and 4 include relevant comparisons to the prior work for density estimation, computational complexity and ablations of the proposed work respectively. The FID scores in table 3 should be expanded with comparison to other flow models that yield better sample quality e.g. [1,2].\n\n* The work proposes an intra-coupling module in section 2.2 . The details of the same can be elaborated as the architecture and the operations performed are difficult to follow. The self attention module is similar to that of Flow++. A discussion regarding the differences of the proposed approach with that of the existing self-attention module (of Flow++) would be helpful. \n\n* Choice of the number of MC samples for different datasets in line 211 can be expanded and the variance thereof discussed. How is the training effected due to 1 MC sample?\n\n* The related work section seems to be minimal and should be expanded to include relevant literature on normalizing flows and explicit generative models in general. The related work of [1,2,3] can be referred for details. \n\n\n[1]. Bhattacharyya et al. Normalizing Flows with Multi-Scale Autoregressive Priors. CVPR 2020.\n[2]. Ma et al. MaCaw: masked convolutional generative flow. NeurIPS 2019. \n[3]. Yu et al. Wavelet Flow: Fast Training of High Resolution Normalizing Flows. NeurIPS 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and the societal impact of the work are appropriately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "16"}, "extra_text": {}}, {"review_id": "ZQJBk4INpJ0", "reviewer": "Reviewer_5hYR", "summary": "This paper presents dense flow that are normalizing flows with additional latent variable after each transformation. This new architecture is tested on generative modelling of images and yield results close or better than concurrent methods whereas dense flows require lesss computational power for training and are relatively fast at generation time.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe method suggested in this paper of adding latent variables to NFs has already been proposed. Authors say the difference with their methods is that they add latent after each transformation steps but I don’t believe this is significantly different from this other work. However this paper demonstrates those additional latent combined with attention mechanism and multi-scale like NF architectures are useful for reaching great generative performance. \n\nQuality:\nExperiments show the model is well performing in terms of likelihood scores, sample quality and computation speed. However results regarding speed are not benchmarked very seriously as methods are using very different hardware which makes it hard to have an exact feeling on where each method stands. In particular it is very difficult to assess the memory usage of different methods and how the proposed method stands with respect to others.\nIt is not clear why table 3 is only providing FID scores for CIFAR10 and not other Imagenet and Celeba. \nI have also the following remarks:\n• L47-49: I don’t agree with your claim about the expressivity of NFs. You can always increase the capacity of the neural networks used to do the transformation although those transformation are one to one.\n• Some indices not correct in example 1.\n• Section 2.2: You should maybe describe multi scale architectures.\n\nClarity:\nOverall the paper is well written however I have the following remarks. \n• 148-156: this is not very clear.\n\nSignificance:\nIt is not clear whether the proposed method is solving any existing problem as many methods exist for creating good generative models. In particular the complexity of the architecture is relatively high and it is not clear if the nice results achieved are really coming from any new idea or just a very nice combination of existing work. If it is only the latter I would expect to see stronger results.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "/", "review_text": "", "extra_scores": {"ethical_concerns": "/", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "DEFP-8Uo8oL", "reviewer": "Reviewer_BeKy", "summary": "This paper proposes an approach to improve the modelling capacity of normalising flow based models. The proposed approach  incrementally pads intermediate representations with noise, which is conditioned on the output of previous invertible units using affine transformations. The paper reports promising results density estimation results on ImageNet and CelebA.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach for noise augmentation is interesting and novel. However, the main concerns are,\n* Motivation -- the paper does not clearly state the advantage of their approach over VFlows. In other words, why does the proposed approach perform better than VFlows? L43-51 only establishes the advantage of augmenting extra dimensions and Fig 1 only illustrates the difference to VFlows.\n\n* Architecture of glow like modules -- The paper proposes to include Nyström attention modules, however, Flow++ already includes multi-head attention (Section 2.2 should therefore discuss the difference to Flow++ in more detail). The main advantage of Nyström attention lies in computational complexity. However, there are no ablations to study this advantage in detail. Therefore, the architectural advantage of the proposed approach over Flow++ is unclear.\n\n* Density estimation -- the paper does not motivate why the results on CIFAR-10 do not show any advantage over VFlow (same bpd).\n\n* Efficiency of likelihood computation -- the paper should include a detailed analysis of speed of likelihood computation as 1000 MCMC samples are needed versus flow based models which provide likelihood estimates in a single pass.\n\n* Sample quality -- the FID score on CIFAR-10 is competitive with other flow based approaches such as, [Normalizing Flows with Multi-Scale Autoregressive Priors, CVPR 2020] which achieves an FID of 33.6 on CIFAR-10. Moreover, there is no comparison in terms of image quality to VFlow. The paper should motivate why the proposed approach fails to improve in terms of image quality compared to prior flow based models.\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "* The paper does not motivate the novelty in term of architecture over prior methods such as Flow++.\n* The computational complexity for likelihood estimation should be discussed in more detail e.g. the trade-off of using higher number of samples vs bpd.\n* The poor performance in terms of image quality must be better motivated and comparison to prior work such as [Normalizing Flows with Multi-Scale Autoregressive Priors, CVPR 2020] must be included.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a normalizing-flow architecture that includes latent variables and is geared towards image modelling. The paper demonstrates good empirical performance of the proposed architecture at modest computational budgets.

There is a clear consensus among reviewers about the strengths and weaknesses of the paper: the main strength is the good empirical performance of the proposed architecture; the main drawback is the incremental novelty. Overall, the paper seems a useful contribution for generative-modelling practitioners, so I'm happy to recommend acceptance.
Decision: Accept (Poster)