Invertible Hierarchical Generative Model for Images
Authors: Heikki Timonen
Miika Aittala
Jaakko Lehtinen
Venue: Accepted by TMLR ()

Abstract: Normalizing flows (NFs) as generative models enjoy desirable properties such as exact invertibility and exact likelihood evaluation, while being efficient to sample from. These properties, however, come at the cost of heavy restrictions on the architecture. Due to these limitations, modeling multi-modal probability distributions can yield poor results even with low-dimensional data. Additionally, typical flow architectures employed on real image datasets produce samples with visible aliasing artifacts and limited variation. The latent decomposition of flow-models also falls short on that of competing methods, with uneven contribution to a decoded image. In this work we build an invertible generative model using conditional normalizing flows in a hierarchical fashion to circumvent the aforementioned limitations. We show that we can achieve superior sample quality among flow-based models with fewer parameters compared to the state of the art. We demonstrate ability to control individual levels of detail via the latent decomposition of our model.

Published in Transactions on Machine Learning Research (11/2023)
Invertible Hierarchical Generative Model for Images
Heikki Timonen
heikki. heikki. www.nenmooooo.o.o.....ooooo
Deportment of Computer Science, Aalto University
Miika Aittala
NVIDIA
Jaakko Lehtinen
Department of Computer Science, Aalto University
NVIDIA
Reviewed OpenReview: https: //
https://openrevveuu openreview. net/ per/formm? id= 4d=4rkKN4 tM63
Abstract
Normalizing flows (NFs) generative models enjoy desirable properties such exact in- vertibility and exact likelihood evaluation, while being efficient to sample from. These properties, however, come at the cost of heavy restrictions on the architecture. Due to these limitations, modeling multi-modal probability distributions can yield poor results even with low-dimensional data. Additionally, typical flow architectures employed on real image datasets produce samples with visible aliasing artifacts and limited variation. The latent decomposition of flow-models also falls short on that of competing methods, with uneven contribution to decoded image. In this work we build invertible generative model using conditional normalizing flows in hierarchical fashion to circumvent the aforementioned limitations. We show that we can achieve superior sample quality among flow- based models with fewer parameters compared to the state of the art We demonstrate ability to control individual levels detail via the latent decomposition of our model. Project source code is available at https: //tt::///ittbt com http://mmome///looo//////
1 Introduction
Generative models for image data have taken large leaps of progress in terms sample quality. interpretability and other performance metrics. Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) were ahead of other types of models terms of sample quality until only very recently. They, however, optimize different loss from other types of generative models that operate on maximizing the likelihood (or bound thercof) of the training data. They also require separate inference network optimization procodure for
inferring latent. variables for real data (Creswell & Bharath 2018). Recently Denoising Diffusion Probabilis- tic Models (DDPMs) (Sohl-Dickstein et al.
2015 Ho et al. 2020) have caught up with GANs in terms of sample quality (Dhariwal & Nichol 2021] In their standard form, however, they suffer from lack of semantic structure of the latent space (Preechakul et al. 2022), and have a trade-off between sample quality and sam- pling speed due to the iterative nature of the denoising process. Variational Autoencoders (VAEs) (Kingmal & Welling, 2013 Rezende et al. 2014) have robust inference procedure with smooth and semantically retrieve encoded image with reasonable reconstruction error
meaningful latent space and allow well as compute lower bound for data likelihood. VAËs can, however, be notoriously difficult to train with the goal of optimizing sample quality and casily give overly smooth and blurry samples or alternativoly exhibit poor latent. structure (Higgins el al.) 2016). Recent work on VAEs has moved the focus on very deep VAEs (Vahdat & Kautz 2020 Child 2020) in the search for dita-likeihhooss that exceed autoregressive models (Van den Oord et al. 2016). whose depth scales lincarly with the data-dimennionallty, Normalizing flows (Tbbak Turner 20133) offer exact and fast inference as well as exact likelihood instead of a bound offered by VAEs and DDPMs. Flow-models map data into latent space deterministically and invertibly,

---

Published in Transactions on Machine Learning Research (11/2023)
Ous J
Glow +
Figure Curated 256 x 256 samples from our model (left) trained with CelebA-HQ. Samples from Glow (Kingma & Dhariwal 2018) (right) are generated using the official implementation and pre-trained weights. Our model produces samples of significantly higher quality in terms of sample variance and spatial consis- tency, but at much lower model capacity. The reader is encouraged to zoom for a detailed view.
which sets heavy restrictions on their architectures. Enforced invertibility is problematic also for model- ing multi-modal distributions (Cornish et al. 2020) due to the copology-preservve nature of NFs. Lastly, Flow-models tend to focus on local pixel correlations while optimizing the data likelihood and disregard the semantic content of an image (Kirichenko et al. 2020).
In this work, study the performance problems of normalizing flows purely from the perspective of sample quality with image datasets. Past work has mostly focused on finding more expressive flow-architecturee to improve a test-set likelihood-score @Dinh et al., 2016 Ho et al.] 2019 Hoogeboom et al. 2019 Behrmann
et al. 2019 Chen et. al., 2019). Despite yielding competitive likelihood scores, pure flow-models yield relatively poor samples compared with other likelihood- based generative models. Even very deep flows fail produce spatially consistent images and lack variation due heavy truncation, as seen Figure fcaturing samples from Glow (Kingma & Dhariwal 2018). Here we investigale whether we can construct an invertible generative model yet avoid the issues arising from enforced one-to-one invertibility. We show that we achieve good sample quality with high-resolution natural images with model that replaces depth with width.
Multi-resolution coarso-to-fine processing is a key principle behind many generative models. While pure flow models such as Glow do employ a resolution hierarchy, we find that latents from different resolutions fail to impact the detail of the associated scale in the expected way. Furthermore, we hypothesize that the invertible squezze-and-sllit image resizing operations employed by Glow are not ideally suited for image generation. Indeed, from an image processing perspective they can be likened to highly aliased filters, known to bias the generation towards regular grid and checkerboard artifacis (Karras et al. 2021) such those apparent. in Figure ! (right) when viewed at high zoom. Conversely, such grid artifacts may be difficuut to smooth out using the invertible scalc-and-shift convolution layers. These observations and the general difficultics of flow models motivate us to introduce a flexible multi-scale model that employs individual shallow Glow models where needed. We lose the ability to evaluate the exact likelihood, but retain the exact invertibility through pair of encoder and decoder pipelines.
Inspired by the success of image super-resolution problem with conditional normalizing flows (Lggmayr et al. 2020 Liang et al. 2021), we construct hierarchical stack of shallow Glow-like flows that models different levels of detail via a conditioning mechanism. An image is encoded into abstract decreasing- resolution features by a sequence of general-purpose ionniinvettbble) networks, discarding detail at each step. Conversely, an image is generated by rebuilding this detail using a sequence of separate Glow-like models, interleaved with general-purpose CNNs that process and upsample the conditioning signal. The system is jointly trained the weighted sum of the conditional flow-losses, inducing the encoder and decoder to find the appropriate decomposition.
We observe significant increase in sample quality when compared with deep normalizing flow models, decreasing the Fréchet Inception Distance (FID) metric (Heusel et al. 2017) of Glow 51.5 to 27.3 on the Haeeb---Q-dataset KKaras et al., 2017) at 256 x 256 resolution, using a significantly smaller model with a much shorter training time. While there are flow-models with better likelihood-scores, we choose Glow as a reference since few other flow-models have been shown generate significantly better samples at 256 256- resolution. For instance, while Residual Flows of Chen et al. (2019) achieve a better likelihood than Glow,
2

---

Published in Transactions on Machine Learning Research (11/2023)
the samples are of very similar quality in terms of subjective visual quality and the FID. We discover hierarchical latent structure where we can control elements of various levels-of-detail separately. We show that images generated by interpolations in the latent space are smooth and remain close to the manifold of real images. Finally, we notice that individual parts of our model do not need to be very deep and instead we can trade model depth for width.
Our contribution In summary, in this work we show that we can construct an invertible model that is trained only with the normalizing flow maximum likelihood-loss and has superior sample quality with 256 256-resolution CelebA-HQ compared against Glow while being much faster to train. Our model has smooth latent space and allows both fast sampling and inference of latent variables.
2 Method
We first introduce the general idea of a hierarchical conditional normalizing flow (Section 2.1 and associated practical design choices (Section 2.2) We then introduce our architecture by generalizing the idea to a multi-scale hierarchy, and discuss connections to related methods (Section 2.3
2.1
Hierarchical Conditional Normalizing Flows
A normalizing flow f yields an exact value for the model probability density function via the change of variables formula.
p(g)(x)) = Pbase (J())) + log |det (xx))
(1)
where (x)) is the Jacobian of f with respect to æ. The function f is constructed to be efficiently invert- ible and to have a Jacobian determinant with a computational timo-complexity preferably lincar or better
with respect to the data dimensionality. Pbase is usually set to a multivariate unil Gaussian distribution. this work, we follow the architectural choices of NEs presented in Dinh et al. (2016); Kingma & Dhariwal (2018) consisting of sequence of affine coupling layers, invertible 1 1x1 convolutions and spatially broadcast learnable scales and biases with data-dependent initialization "actnorms". We introduce conditioning to flow by giving an additional input e to the affine coupling layers h, which invertibly transform a variable a via
h( c) = [2a, 26 O NN.( e) + NNs(
(2)
where Ca Vy is some split of the input tensor x and NN, and NN, are neural networks.
Due to the innate limitations of normalizing flows we want to offload as much work as possible from the invertible neural networks. Inspired by solving the image super-resolution task with flow-models, we train conditional flow-model in such a way that the conditioning input c is in itself function of the data, and jointly learned in the process of maximizing the likelihood. Essentially this forms an autoencoder whose "reconstruction loss" is the flow loss. The training loss for such network is given by:
Leand (x) - log pe (æly).
(3)
where is the distribution induced by a conditional normalizing flow
x Scond Zcondi y, 0):
(4)
with unit Gaussian prior. More precisely, pe is the density induced by the fush-forward Px = fcond# Pz of the Gaussian prior Pzoaaa y ~ ()(())))))))))))))))))()) is stochastic encoder with learned components (convolutions) with parameters o. The encoder is trained jointly with the normalizing flow. This learning objective which has maximal mutual information 1(X, Y)
leads into finding the latent variable representation with the original data. If the complex conditional normalizing flow parametrizing Po was reduced into a Gaussian likelihood conditioned by y via the mean and covariance, we would recover the standard VAE reconstruction-loss. Like in VAF, for generation we need to additionally model the "prior" distribution
3

---

Published in Transactions on Machine Learning Research (11/2023)
of Y. However, unlike in standard VAE, we do not directly enforce Gaussian prior for each ((ylz)) Instead, we approximate the aggregated posterior in a separate step with distribution induced by another flow-model
y sprior (Zprior; sp),
(5)
of density Pe of the respective push-forward between Z prior and Y pelor, using the flow-loss of Equation !
Lprior log (y)
where y is sampled via x ~ (a), y ~ ((y/ar). Note that the "prior" label here denotes that the flow forior operates on y and is different from the unit Gaussian base distribution of a flow introduced in Equation In practice, we minimize the following expression
min E. Ba- (r [eoudd + L Pprior (SG [y]) 8,000
(7)
where SG[ is the stop-gradiens operator with SG(x) x but which has vanishing partial derivatives; we discuss the reason for stopping the gradients below. Alternatively, this can be thought of as two-step training procedure, with the stop-gradient operator decoupling the prior-flow from rest of the model. Figure illustrates the model setup, as well as shows the two-step -nature. From now on, we omit the flow and encoder parameters 0, o, op from the notation for clarity.
Stop-gradient operations The loss resembles that used in VAËs, only lacking the negative entropy term of the encoder in the KL-divergence between the approximate posterior and the prior. This intro- duces loophole where- in the case of a determin- istic encoder-the prior flow loss can be arbitrarily improved by concentrating the distribution of y's, js leading to degenerate solution. A related form of this degenerate solution with vanishing variance was described by Hoffman et al. (2017) in the context of the 8- VAE (Higgins et al.] 2016) and an implicit use of data-dependent prior. Xiao et al. (2019) also described similar degeneracy in their closely related work. For stochastic encoders with Gaussian noise (as used by us; described later), related loophole exists through intentionally decreasing the signal-to-noise ratio of y. making it uninformative akin to poste- rior collapse in VAF.s. We employ the stop-gradient operations to prevent the prior flow Sprior from directly impacting the distribution of y during training. Adding the entropy-terms into the loss (and bringing the loss even closer to the VAE-loss) can make the tendency for reaching degenerate state even stronger as it directly encourages g(y(æ)) have an entropy matching that of the prior.
f
Zrxioe
encoder
decoder
t cond.
foord
7ood
Figure 2: Hierarchical conditional normalizing flow structure. The input æ is encoded into the latent [ Aprior] by general-purpose CNNs (blue blocks) and NFs (green blocks) via the intermediate y. The stop-gradient operation is denoted by the red dashed line.
Separation of modeling tasks The most important design point of the described model is the choice of the form of the stochastic encoder ((y)ar) subject to the limitations of the normalizing flows. A deterministic ((y(æ) (y) (the identity) only pushes the modeling work to the flow modeling the prior, Sprior, while too strong a bottleneck (either by strong noise aggressive blurring) does not provide enough conditioning information for the conditional flow. We deliberately avoid using squeeze-and-split operations in both the conditional flow and the prior-flow. That is, none of the operations within flow change the spatial resolution from that of the respective flow's input. Instead, we design the stochastic encoder q to greatly reduce the spatial resolution with non-aliasing filters, but also allow for learned components (convolutions). Ideally, all perceptually meaningful, spatially long-range correlations that the conditional flow might not be able consistently capture can be routed via the encoder q(y) /z). Finally, the resulting distribution of y should be with another flow.
such that it can be approximated without squeezz-and--plli-opprrrti
4

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm Inference and sampling for the model in Figure 2
Require: Data point æ, noise scale procedure INFERENCE
Require: Sampling standard deviation sampling
procedure SAMPLING
y + Encoder(x)
prioo NO o sampling I)
0 < ox
D. Defined in Eq.
10
eonnd N0 2 samplling
E ~ N(0,a20011
y Sprior ( (Zprior)
y + E
a
- fcond fonnd(Zconul) (zcond; Decoder(y))
prior + Sprior
return x end procedure
Zcond f- Foond (x;; Decoder(y))
2 - xpriors ccond]
return
end procedure
In the limit of the encoder being only fixed deterministic filtering and downsampling we recover the image super-resolution task with a learned prior the row-resolutio images. The purpose of the learned encoder- components is to allow for finding more compact representation for y than merely downsampled image. Conversely, we want to avoid near-perfect auto-encoding via the y-variable. We require the conditional normalizing flow to be able to generate an appropriate level of spatially coherent detail, such that the capacity of the conditioning y-variable be used to only model global, high-level features. Otherwise, the capacity of the conditional flow to produce variation will essentially be wasted if each image can be generated using only higher-level representation.
Invertibility and sampling Our construction remains invertible as each datapoint can be encoded into latent with
y ~ (3y
(8)
2 Ixpioor Zcond] = [forior (y) ff (x; (e:y)]
(9)
that is, the datapoint lifted into a space that has more dimensions than the original datapoint. Any encoded datapoint can deterministically be recovered with zero roconstruction-crror via Equations 4 and since Sprior and foond are by construction bijective. This holds for any lype of encoder ((y)ar)). The inference
and sampling procedures for the model in Figure are summarized in Algorithm !
2.2 Individual design choices
Stochastic encoders While the encoder g could be deterministic, empirically find it useful to inject Gaussian noise to the outputs of the encoders. That is, we have q(ylx)) a() where
ox 1/dimy! (Var(((X))) T 1):
(10)
where y is parametrized as a neural network (CNN with downsampling in Figure 3 the encoder in Figure 2. We choose the a of the noise be proportional the variance (with parameter a) to prevent. the model from scaling the signal up to improve signal-to-noise ratio and effectively ignoring the noise. The mean is taken in order have isotropic noise so that the model cannot corrupt only part of elements of and squeeze an almost clean signal in the remaining elements. Assumptions of similar spirit of data corrupted with isotropic (instead of anisotropic) noise have been found to yicld higher-quality samples within the VAE-literature (Rybkin et al. 2021). In practice, the variance is computed as a Monte Carlo estimate over the current minibatch by computing (i, taking the empirical element-wise standard deviation over the minibatch and normalizing with the dimensionality y.
The noise can be thought of a regularizer or as augmentation like in the work of Ghosh al. (2019) An alternative view is to consider the noise as mollification for the prior distribution, rendering
5

---

Published in Transactions on Machine Learning Research (11/2023)
reasonable to model with the prior-flow. The purpose of the noise is hence not the same in VAEs (we could choose to have deterministic encoder), where it is strictly construction for allowing optimization of the Evidence Lower Bound (ELBO). Here, a of the Gaussian ((y(æ) does not appear in the loss and hence our optimization target is not a bound for the data-likelihood, missing the entropy-tern of the KI-divergence of the VAE-loss. We can, however, compute a Monte Carlo estimate of the ELBO by adding the missing entropy-term, which can readily be computed for the Gaussian-conditonaal stochastic encoder.
Multi-scale architecture We introduce a multi-scale representation for y to enforce more hierarchy into the generation process. That is, instead of encoding æ into single (Figure 2), we create multi-scale reprosentation y Vy, UR...... VI Fprioel, where R₂ are the resolutions of the y-variables up to Y and i = yR. The hierarchical encoder decoder structure is illustrated in Figure g The conditional part of loss of the training-loss now becomes
Leond (x) =-
Σ log (V R
pog prior))
(11)
where each p is induced by separate conditional normalizing flow. In training, we weight each term in Equation 11 by a separate weight wy. Sampling follows the same procedure as in the two-level model of Figure 2 with independent Gaussian samples drawn from the base distribution of each of the flows followed by their conditional inversion using the results from the higher levels of hierarchy.
The multi-level hierarchical framework quite general and does not enforce any particular form for the normalizing flows used to model con- ditional distributions of Equation [ One could also use probabilistic models of other types instead of normalizing flows. However, with other types of model, one may lose strict invertibility or suf- fer in terms of sampling efficiency. For example, Preechakul et al. (2022) introduce a two-stage model which can be thought of as similar hierarchical construction but with normalizing flows replaced with denoising diffusion models. We choose to re- main close to the Glow-architecture with the choice of model for the conditional distribution to high- light the benefits replacing the limited resolution- hierarchy of the baseline Glow-model with our pro- posed hierarchical construction. Modifications are justified with ablation studies in Section 3.2
fact
Flow
piror
decoder
Yosoo
cand J
Flow
encoder
split
Yi
concat
noise
CNN whtpssmple
CNN widoownsappe
cond.
Flow
-ig
+ noise
Yas
concat
IN
CNN wiupsample
1x1 conv
cond.
CNN wideorsamppe
Flow
Zga
Yea
Z:36
Trading depth for width We drastically re- duce the number of latent variables when compared with deep VAEs, by only using ~ levels for Deep VAEs have an order of magnitude more depth (Child 2020 Vahdat & Kautz] (220)) Instead, we move capacity the normalizing flows (which them- selves shallow compared to Glow) and ensure that each flow-prior models meaningful variation in the data, instcad of merely adding Gaussian noise to the output of the previous decoder. There has been evidence in previous work that replacing depth in VAEs with the ability to handle long distance inter- dependencics with attention within the latent code yields competitive likelihood-scores (Apostolopouion et al.] 2021). However, it is not clear this also translates into better image quality in high-resolution images.
V23G
Figure 3: Multi-scale architecture. The encoder (left) and decoder (right) pipelines consist of repeating per- resolution blocks (light blue) with each intermediate representation RR, further encoded into Gaussianizod latents #R, via normalizing flows (green). Refer Appendix F for detailed breakdown of the CNN and flow layers.

---

Published in Transactions on Machine Learning Research (11/2023)
U-Nets in Normalizing Flows We find it necessary to use U-Nets (Ronneberger et al 2015 in the coupling layers of the normalizing flows despite the desire for routing semantic information via lower- resolution latent. Low-level detail that is not encoded into the latent might still have spatially long-range dependencies (long strands of hair, color of the eyes) and honce equipping the flows with tools to model these dependencies is required. If the conditional normalizing flows operate only on very local detail, the model has to route these low-level features via the high-level latent, potentially stealing capacity from other useful high-level features and violating our design principles. Bottle-necking the encoder decoder pipeline with e.g. blurring might also render this impossible and the aforementioned details might simply be lost. Within flows using U-Nets, we split the tensor spatially in 8 x 8, 4 x 4 and 2 x 2 checkerboard-patterns to differentiate from the Glow-like split in the channel-direction While an equivalent split can be achieved with the
squeeze operations and pure channel-direct ion splits, we do not observe Glow-like grid artifacts in any of our samples.
2.3 Related Work
Our model can be seen as special case of a VAE, where instead of Gaussian likelihood we have more complex normalizing flow. Our approach also models the aggregated posterior (ampas>a(a) (rie [x]] and disregards the negative entropy-term of the approximate posterior- prior type KL-divergence terms in the optimization process. There is no pressure from the perspective of the optimization loss to have cach (y][z) be zero-mean unit Gaussian, preventing the phenomenon known as "posterior-collapse"" "information preference" or "optimization challenges" of VAFs (Bowman et al., 2015; Chen al.] (2016). We see the addition of noise more as a regularization technique rather than a necessity for a variational bound. Even more generally, the difference between VAEs and NFs is not always very clear. A flow-model trained on noise-augmented data can be seen VAE and vice versa (Huang et al. 2020 Some VAEs also use normalizing flows as components for more expressive posterior approximation or likelihood-modd (Rezende & Mohamed| 2015 Kingma et al., 2016 Agrawal & Dukkipati 2016). Finally, while normalizing flows are usually described as being able to yield exact likelihoods, they often employ "dequantization" to render discrete data continuous. However, dequantization, or the addition of uniform (Theis et al., 2015) or more complex noise (Ho et al. 2019) to lift the data distribution to a continuous space renders a model to only give bounds of likelihoods rather than an exact likelihood.
VAEs
Flows with noiso-augmented data Huang et al. (2020); Chen et al. (2020): : Greié et al. (2021) all share the same idea of lifting the data distribution into a dimenssai space via padding of noise for fixing issues with multi-modal data and invertibility with normalizing flows. Their work focuses mostly improvements in the likelihood score and for inages with resolutions less than 256 x 256. Our construction of the padding is more delicate. Rather than padding the original data with noise, we pad the data with a slightly noisy (or even noiseless), maximally informative compression of the data which is specifically designed to combat issues of flow-models on images.
Wavelet Flow Yu et al. (020) build a similar hicrarchy of conditional flows, but with dircct one-to-one invertibility and exact likelihood-evalaation using Haar-wavelet transforms as a fully invertible encoder decoder structure. The conditional flows generate the the detail coefficients conditioned on the mean. Compared with their work, we see benefit in using an overcomplete representation to allow a more informative high-level latent (that is, one that is not necessarily fow-rosolutions image). We also avoid aliasing due to not using box-filtering We do however, lose the ability to measure exact. likelihood.
Flows on manifolds Instead of defining tion, one can choose to parametrize manifold using points in a lower-dimensional space and model density only on this manifold. As an image is unlikely to lic exactly on the manifold, one needs to work with the projections of images onto the manifold. The training process hence comprises of two sleps: finding the manifold that is on average closest to the training data, and modeling the density of the projected data on the manifold. Several pieces of prior work explore ways of achicving this: Kothari et al. (2021); Brehmer & Cranmer (2020). Our work is in spirit very similar, yet we do not limit the density modeling a manifold but work in the full space of RGB- RGB-images. In particular, do not simply train a deterministic autoencoder
for the space of all RGB-images of a fixed resolu-
7

---

Published in Transactions on Machine Learning Research (11/2023)
Figure 4: Curated samples from our model (Config A) Irained with CelebA-HQ 256 x 256 using truncated Osampling = 0.7 (reduced-tennl temperature sampling as in Kingma & Dhariwal (2018) for latent resolutions greater than
to approximate the manifold of real images and then fit flow-model into the latent space of the autoencoder. Our encoders and decoders are trained jointly with the conditional flows, using their likelihood-loss as the training signal.
3
Results
Our model greatly improves the FID for Flow-based modols using the ColebA-HQ (Karras et al, 2017 dataset in 256 x 256: our model reaches a score of 27.3 against Glow's 51.5 with only about 36% of Glow's parameters Furthermore, we measure throughput of around 50 samples / second on an NVIDIA RTX 3090 GPU, which is around times the throughput of Glow on the same hardware. Next, we show qualitatively that our model constructs a latent decomposition which allows controlling individual levels of detail independently and in more uniform fashion than Glow. We also present ablation studies, showing how different. architectural choices within our model change its behavior and performance in terms of FID. Finally, we train baseline Glow-like flow-moode models with similar capacity to ours using the church and bedroom classes of the LSUN- dataset YYu et al. 2015). We present a simple toy-case on a 2D mixture of Gaussians, showing the ability of our model to capture multimodal distributions, Appendix E Likelhhood-scores the models are tabulated
in Appendix G Details the training parameters are aggregated in Appendix H
3.1 Qualitative model behavior
Samples from model trained with CelebA-HQ 256 256 exhibit good spatial coherency and variation both in high and low-level detail (Figure q Samples from model also lack checkerboard-liie aliasing artifacts that can be seen in the Glow counterparts.
Though the effect 2256 is subtle, each part of the latent has an observable effect on the decoded image, which not true for Glow. Figure 5 shows pixel-wise standard deviations for an encoded image when sampling a part of the full latent from the prior but fixing the others. We perform the same operation for Glow. With our model, the latent codes of increasing resolution change increasingly high-frequency details in the image. For example 21 changes the identity of the person and the background, while 2128 mostly affects the fine structure of the hair. Compared with Glow, our latent structure has more uniform effect on decoded image, with the high-resolution latent codes also yielding visible changes in the image. We encourage the reader to also look at Video 3 from the supplementary material for another visualization of the variance.
Figure i presents another view to the latent decomposition by showing how various levels of detail are encoded into the latent space. We encode real image, and cunulatively set zR; to zero starting from high resolutions. This process gradually removes detail from the image with the 2prior and æı-variables only containing very high-level information like the orientation of the head and the hair color. Interestingly, the eye-color is stored in Æ32, showing that being able to model long-range correlations within the normalizing flows is still required despite the encoder decoder -mechanism within our model. Otherwise, there would be inconsistencies in the samples of our model, like mismatching eyes.
We computed the parameter count Glow using the official pre-trained madel.

---

Published in Transactions on Machine Learning Research (11/2023)
Ours
prrior
232
1128
Glow
42
162
32
64
1282
Figure Pixel-wise standard deviations while re-sampling of latents from the prior one resolution at a time (columns) with 32 samples and no truncation. The latent code of each resolution is responsible for changing detail of the corresponding level of detail. Compared with Glow, our method yields more uniform effect of latents of different resolutions. Note that the spatial dimensions of the latents differ between the corresponding colums.
Zprior f- (mean image)
Original
2256 <-
1128 f-
Z32 f-
28 <
21
visualize the contributions of individual latents by inverting real image (left) and cumulatively zeroing them starting from the finest resolution. This causes a progressive loss of detail at larger and larger seales. The rightmost image is the the result of setting all latents to zero.
Figure 6: To complement Figure

---

Published in Transactions on Machine Learning Research (11/2023)
Interpolations in the latent space of our model result in smooth changes in the decoded images, but also yield sharp images. In contrast to Glow, our interpolations also lack strong aliasing-artifacts. Videos on interpolations in the latent space for random samples and real images can be found in the supplementary material.
3.2 Ablations
We train our model using several variations in the configuration. We identify important design-elements that affect sample quality, compared by FID-values in Figure 7a We refer the reader Appendix A for a supporting visualization for the ablations, similar to Figure
Noise in Encoders From Figure 7a notice that adding a non-zero amount noise to the y-variables is beneficial in terms of sample quality. If no noise is added (Config B) rendering the encoder deterministic we notice that the model tries to encode an increasing amount of low-level detail into the high-level, low-resolution latents. Conversely, for a high amount of noise (Config C), images after the aforementioned procedure become increasingly blurry. Hence we need to specifically tune the noise-level for optimal sample- quality. We hypothesize that the added noise is more destructive renders it more favorable to encode global features into higher-level y-variables. The decoder can tolerate noisy y imputs to limit, but too heavy noise likely starts to degrade the results, causing blurring. In the limit of very strong noise, the signal of VR. is lost and the task of the respective normalizing flow becomes trivial (a "posterior collapse"), due to vR, already being almost Gaussian.
sigh-frequency details of y variable and
U-Nets in Normalizing Flows Employing U-Nets in the normalizing flows is beneficial in terms of FID, A model with no U-Nets in the normalizing flows (Config D) has a stronger preference to attempt to encode this into the high-level latent code improve the flow-losses. We attribute the this failure to the disability of the model to generate this content using the high-resolution y modcling normalizing flows. With only limited receptive field the model has to use the capacity of the high-level latent for these features. We also see traces of very now-frequency noise which may be an attempt of the model to represent the low frequencies of the slight. Gaussian noise used in data augmentation via the low-level latents.
Flows at additional resolutions In the best-performing Config A, we do not take constant-size steps down in resolution within the decoder but decompose y = 2y256 9/28: Y32: Us V1. Mprioel- Surprisingly, adding additional resolutions levels to the encoder- decoder (Config E) to model the missing resolutions Yos and Vıo: (while reducing capacity at other flows, encoders and decoders, to have approximately similar number of parameters), renders the results considerably worsc We again observe that high-frequency details are again encoded more aggressively into the low-resolution latents. While the model might have lower-capacity encoders and decoders, it is clearly misusing the given resources.
Deeper Normalizing Flows Making the normalizing flows longer (Config F, two times the affine coupling blocks, with less feature maps keep the parameter count constant) has small negative impact on the FID. The model hence does not seem to be limited by the length of the normalizing flows (width of the model), but rather by the inductive bias of how the image is encoded into the hierarchical y.
3.3 Other datasets
We train our model also using the LSUN churches and bedrooms datascts at 128 X 128 resolution. For comparison we also train a Glow-like model from scratch using similar capacity and the same compulations resources as was used to train our own model. From Figure ( we see that our model consistently yields much better FID than Glow. We did not experiment tuning the parameters of our model with the lower-resolution expeeriee
data. Because of the limitation in the parameter budget, the Glow-like model not as deep as in Kingma & Dhariwal (018)). Uncurated samples from the models described above are found in Appendix
10

---

Published in Transactions on Machine Learning Research (11/2023)
Config A
Glow/Bedroom
Config B No-noise
Ours/Bedroom
Config C Heavy-noise
Glow/Church
Config D No-UNets
250-
Ours/Church
70
Config E Add-y-res.
60
Config Longer-flows
200
150
100
30
20
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Shown images million images
Shown images million images
(a)
(b)
Figure (a) Ablations with CelebA-HQ 256 256. Effect of model strucure and parameters on the achieved FID measured during training. The FID is computed using 25k samples with all the training data-augmentations enabled. Each model has the same learning rate and approximately the number of parameters. A proper level of added noise to y-variables improves the FID drastically. Addition of U-Nets into the flows also has a large effect. Interestingly, the FID is also very sensitive to the mumber of resolutions modeled with flows in the hierarchy. (b) FIDs during the training of our model on LSUN church and LSUN bedroom at 128 x 128-resolution, compared against a Glow-like model with similar capacity. The FID is computed using 25000 random samples from the entire trainset (which is million images for LSUN bedrooms), causing the noise in the measurements.
4 Discussion
Flow-based models have many useful properties, but have suffered from poor sample quality relative to many other families of generative models. While not on par with the current best GAN and diffusion models, the exactly invertible generative model presented in this work yiclds much higher-quality samples than the previous stale-of-the-arl invertible models. Moreover, we have shown that by constructing hierarchical stack of conditional normalizing flows we can separate high and low-level features and model them conditionally each other to resolve some of the issues concerning standard normalizing flow models. We find that data augmentation or regularization with noise is essential for our model to perform well. Studying different regularization methods and their effect on the latent decomposition as well the applicability of the latent space of our model to downstream tasks would be interesting avenues for future research.
Limitations. Our model has relatively large number of hyperparameters, such as the noise levels that drastically affect its performance. While we have presented empirical evidence why certain choices might be better that others, we have no principled method of optimizing those values. Adding them as optimization parameters and employing the VAF loss is hardly an option, since the VAE loss also requires complex paramctor-tuning for good sample quality. While our model yiclds better samples than other flow models, they still lag behind GANs and DDPMs. Like other flow models, ours also has difficulties modeling righutesolution, highly variable data, extend to conditional tasks like inpainting or denoising in principled Bayesian way due to the model not yielding an exact likelihood but only a bound.
in the LSUN results. Finally, our model does not directly
Acknowledgments
We thank Pauli Kemppinen and Erik Härkönen for help with the code release. This work was partially sup- ported by the European Research Council (ERC Consolidator Grant 866435), and made use of computational resources provided by the Aalto Science-IT project.
11

---

Published in Transactions on Machine Learning Research (11/2023)
References
Siddharth Agrawal and Ambedkar Dukkipati. Deep variational inference without pixel-wise roconstruction. arXiv preprint orXic:611105500, 2016.
Ifigeneia Apostolopoulon, lan Char, Elan Rosenfeld, and Artur Dubrawski. Deep attentive variational infer- ence. In International Conference on Learning Representations, 2021.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen, Invertible residual networks. In International Conference on Machine Learning, pp. 573 582. PMLR, 2019.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Gener- ating sentences from contimuous space. arXiv preprint arXie:1511.063/ 2015.
Johann Brehmer and Kyle Cranmer. Flows for simultancous manifold learning and density estimation. Advances Neural Information Processing Systems, 33:442-455 -533, 2020.
Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. Vflow: More expressive generative flows with variational data augmentation. In International Conference Machine Learning, pp. 1660-1669. PMLR, 2020.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. Advances in Newral Information Processing Systems, 32, 2019.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Nya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv:1611.02991, 2016.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. ar preprint XrXi::2011. 10650, 2020.
Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with contimuously indexed normalising flows. In International conference on machine learning, pp. 2133 2143. PMLR, 2020.
Antonia Creswell and Anil Anthony Bharath. Inverting the generator of a generative adversarial network. IEEE transactions on newral nelworks and learning systems, 30(7):1967 1974, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8747 8794, 2021.
Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.0888 2016.
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. Advances in neural information processing systems, 32, 2019.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv: 12436, 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672 2680, 2014.
Matej Grcié, Ivan Grubišié, and Siniša Segvić. Densely connected normalizing flows. Advances in Neural Information Processing Systems, 34:23968 229082, 2021.
Martin Housel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nesslor, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017.
12

---

Published in Transactions on Machine Learning Research (11/2023)
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Picter Abbeel. +low++: Improving flow-based generalive models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722-272222 27300. PMLR, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
Matthew D Hoffman, Carlos Riquelme, and Matthew J Johnson. The f-vac's implicit prior. In Workshop on Bayesian Deep Learning, NIPS, pp. 1-5, 2017.
Emiel Hoogeboom, Rianne Van Den Berg, and Max Welling, Emerging convolutions for generative normal- izing flows. International Conference on Machine Learning, pp. 2771-2788 PMLR, 2019.
Chin-Wei Huang, Laurent Dinh, and Aaron Courville. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models. arXiv preprint arXiv:2002.07001, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved 2017.
quality, stability, and variation. arXie preprint arXiv:
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34: 852 863, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6900 12868, 2014.
arXiv preprint
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ar preprint drXiw:1312.6114 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Advances in neural information processing systems, 31, 2018.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Tlya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016.
Polina Kirichenko, Pavel Iznailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of- distribution data. Advances in neural information processing systems, 33:20578-2085 2020.
Konik Kothari, AmirEhsan Khorashadizadeh, Maarten de Hoop, and Ivan Dokmanić. Trumpets: Injective flows for inference and inverse problems. In Uncertainty in Artificial Intelligence, pp. 1269-1278. PMLR, 2021.
Jingyun Liang, Andreas Lugmayr, Kai Zhang, Martin Danelljan, Lue Van Gool, and Radu Timofte. Hierar- chical conditional flow: A unified framework for image super-resolution and image rescaling. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4076 4085, 2021.
Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte, Srflow: Learning the super- resolution space with normalizing flow. Tn European conference on computer vision, pp. 715-732, Springer, 2020.
Konpat Precchakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2022.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International con- ference on machine learning, pp. 1530-1538, PMT.R, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp- 1278-128h 1286.
PMLR, 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assen assiisedd inferven- lion, pp. 234 241. Springer, 2015.
Oleh Rybkin, Kostas Daniilidis, and Sergey Levine. Simple and effective vae training with calibrated decoders. In International Conference on Machine Learning, pp. 9179 9189. PMLR, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathann, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256 2265. PMLR, 2015.
Esteban G Tabak and Cristina V Turner. A family of nonparametric density ostimation algorithms. Com- municalions on Pure and Ayplied Mathemalics, 66(2):145 164, 2013.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models.
arXiv preprint 1rXim:111.00844 2015.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Newral Information Processing Systems, 33:19667-19679, 2020.
Aaron Van den Oord, Nal Kalehbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in newral information processing systems, 29, 2016.
Zhisheng Xiao, Qing Yan, and Yali Amit. Generative latent flow. arXiv preprint arXiv:1905.10485, 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.0335 2015.
Jason J Yu, Konstantinos G Derpanis, and Marcus A Brubaker. Wavelet flow: Fast training of high resolution normalizing flows. Advances in Neural Information Processing Systems, 13:6184-6199 2020.
14

---

Published in Transactions on Machine Learning Research (11/2023)
A
Cumulative zeroing of the latent code for ablations
Config A (ours)
Config B
Config D
Config E
Zprior f
Original
2256 f- 0
1128 f- 0
Ø32 +-
23 0
#1 +
(mean image)
Figure 8: Cumulatively setting the latent-code of a real image to zero starting from high-resolution latents. Highlighting the failure cases of the ablations. Config B / no noisc: Part the low-rcsolution latent code is used for attempting to model hair texture while the color of the hair lost after setting žy to zero. Config D No U-Nets: Same problem as with Config B, but with coarser hair-texture encoded to the X I-resolution latent. In fact, the hair texture might be from learned constant since it visible in the mean image as well. The 1 latent merely modulates this texture. Low-frequency noise-artifacts be clearly seen after ø32 is set to zero. Config E different y-repr.: Note that since Config E also has 264 and 216, those are also cumulative set zero, between the 2128 Z32 and 232 / Zg columns, respectively. The intermediate results are not viualized here. Very fine hair-texture is carried down to 32 x 32-resolution. Some high-level features such as the coloring is completely lost by the x latent.
also
15

---

Published in Transactions on Machine Learning Research (11/2023)
B
Does the model memorize the dataset?
ME
FIL
Figure 9: 5 Lg-closest training images (columns the right) for generated images with Config A (rows)
with the CelebA-HQ dataset. We see that the sampled images do not appear in the dataset.
16

---

Published in Transactions on Machine Learning Research (11/2023)
C
Comparison with a Glow with similar capacity
Ours
Glow
LSUUN church 128
LSUN bedroom 128
Figure 10: Uncurated samples from our model and from Glow-like model with similar capacity and the same training time. All models use truncated sampling with = 0.875.
17

---

Published in Transactions on Machine Learning Research (11/2023)
D
Uncurated FFHQ 256 x 256 samples with model Config A
Figure 11: Uncurated samples from model with Config A trained with FFHQ 256 256. Sampled with truncation 0.7 for latent resolutions larger than 1.
18

---

Published in Transactions on Machine Learning Research (11/2023)
E
2D Toymodel
Target PDF
RNVP-based flow-model PDF
Ours (kernel-estimate)
1.5
1,0
1.0
0.5
0.0
-0.5
00.5
0.5
-1.0
-1.52
-1.5
-1
0
-1.22 -1
Figure 12: Real NVP-based normalizing flow DDnh al. 2016) and our model trained on a 2-dimensional mixture of Gaussians target distribution (left), The models have similar capacity. The standard normal- izing flow (middle panel) suffers from the well-documetted problem of failing to separate the modes due to the invertibility constraint of the architecture. While our model (right panel) does not capture the relative weights of the modes correctly, it captures the multi-modality better than the reference. The dimensionality the latent in our model is dim(y) 1, modeled by a neural spline flow (Durkan et al. 2019) as the prior Speior- In this experiment fcond conditional real NVP and the encoder and decoder are small fully-connected neural networks.
19

---

Published in Transactions on Machine Learning Research (11/2023)
F
Detailed network architecture
CNN w/ upsample
Encoder and decoder
(lowest decoder resolution)
Flow
Synv
decoder
nonlin
umd
4 xx4x 512
encoder
Flow
Jearned const.
AdaiN
split
ouneal
Bonlan
CNN vikpazple
upsample
CNN widowssszpls
crrd
affine
AdaIN
Hlow
conv nonlin
affine
AdaiN
noise
Y
cancat
conv nomlin
CNN whgnoaapk
Fxl cony
sclonassagll
Flow
CNN w/ downsample (all encoder resolutions)
CNN w/ upsample (other decoder resolutions)
U-Net
oonv nonlin
cOnY nonlin
upsample
conv nonlin
downsample
nonlin
COnY noolin
Normalizing flow
Affine coupling
imput
conditioning
imput
mask
affine compling
K
actnorm
conditioning
affine coupling
N *
cony nonlin
Ixl inv. cony.
OR
N.Net
affune coupling
split
Iatent
conv
conv
clip
clip
Figure 13: Detailed architecture. IN denotes instance normalization, and also appears in the U-Nots and among the affine coupling convolutions. The non- linelriiait are leakyReLUss apart from the conditioning of baockerboard-maskeee U-Net -type affine coupling blocks and the first of the N convolutions of convolutional- type affine coupling blocks. Multiple units of CNN up/donssample -blocks are concatenated if there is a change of resolution differing from (c.g. from y32 to ys)- In case of this stacking, the U-Net at the beginning of the CNN-upscaler is omitted from blocks other than the first. The last and first. affine coupling-layers of a normalizing flow-ble block use U-Nets (apart from Ys and }256) and completely mask out the flow-input. That is, the scales and biases are computed only using the conditioning signal.
20

---

Published in Transactions on Machine Learning Research (11/2023)
G Additional metrics
Table 1: Negative log-likelihood (NLL, lower values are better) from our model and Glow. The likelihood- bound for our model computed as the VAE ELBO as discussed in Section 2.2 which not directly the optimization target our model, partially explaining the performance difference with Glow, which directly optimizes for likelihood. All values are mcasured by us using our own implementations apart from Glow CelebaHQ-256,, which taken from Kingma & Dhariwal (2018). As our focus is on improving FID, which is not necessarily computed against specific test-set, we do not have separate test-set and all our values are computed against the train-set.
Model / dataset
NLL / beri-pee-iimnnsoon
Ours CelaballQ 256 (5bit)
< 1.3
Glow CelabaHQ 256 (5bit)
1.03
Ours LSUN-Church 128 (8bit)
S 4.0
Glow LSUUN-Chree 128 (8bit)
= 3.6
Ours
LSUN- LSUNNBeeroom 128 (8bit)
3.8
Glow
LSUN-Bedroom 128 (8bit)
3.4
21

---

Published in Transactions on Machine Learning Research (11/2023)
H
Hyperparameters and training details
Table Training details for Config A with ColcbA-HO/FFHQ 256 256
Name
Values
Batch size
16
Batch size Var(X)
4
Optimizer
Adam (Kingma & Ba
2014 with 31 0.9, 83 = 0.999
LR (encoders/ (encoders/deecodess//ef /wtp///eee/o/ooeeeeee
x 10 2 x 10-3, 5 x 10-3
LR. docay
Multiplicative (encoders decoders+ +flows) 0.92/0.95
Encoder parameter freeze at
60 epochs
Gradient L2 clipping
50.0
GPUs
2x V100 16 GB 96 h
Train time
Total parameter count
80.15 M
Table 3: Training details for Config A (LSUN) with LSUN church / bedroom 128 x 128
Name
Values
Batch size
16
Batch size Var(X)
8
Optimizer
Adam with B1 0.9, =
LR
3 x 10-4 1 x 10 10-3, x 10-3
LR decay
Multiplicative (encoders flooderssflloss) 0.92/0.95
Encoder parameter freeze at
60 epochs
Gradient L2 clipping
50.0
# GPUs
1x V100 16 GB 96 h
Train time
Total parameter count
74.58 M
Data Preprocessing We augment each dataset by adding uniform 1/255 noise 8-bit images normalized to [0, on top of which also add slight zero-mean Gaussian noise with standard deviation 10 During training, apply random horizontal flips with probability p 0.5.
FID Measurement When comparing to Glow, we compute the FID using 30000 samples (the full CelebA- HQ -dataset). We use 5-bit dequantization (and the tiny Gaussian noise-augmentation mentioned in the previous paragraph), when computing the value for Glow, with the result being few points weaker (56.8) for 8-bit data. We use truncated sampling with 0.8 when generating images with Glow. Our model uses 8-bit images.
22

---

Published in Transactions on Machine Learning Research (11/2023)
Table 4: Model hyperparameters for Config A. The LSUN-128 models are trained with the same configura- tion, but the Flows at resolutions -parameter (R,) set to [128,64,32,8,1, prior] instead. The affine-coupling split lype uses format M x split lype, where C' denotes splits along the channel dimension and Sk a spatial checkerboard split with K- K-pixel alternation. Coupling types are listed starting from the side of the input (o.g. the channel-splits are in general closer to the input than the latent). Affine coupling blocks with spatial splits (denoted with Sk) use U-Nets as their forward neural norworks. There is no additional source of noise for the flow at the highest resolution a Y256 and hence a is not defined there. LeakyReI. LeakyRel.U-noniinegg n-onninearrts use slope 0.1.
Name
Values
Flows at resolutions (R:)
[256, 128, 32, 8, 1, prior]
Number of channels at flows
[3, 4, 8, 8, 408, 4]
Noise scale to flow (a)
[n/a, 0.4, 0.05, 0.05, 0.05, 0.05]
Noise scale to decoder (a)
[n/a, 0.4, 0.05, 0.05, 0.075, 0.075
Flow-loss- weight (w:)
[1/(25xx26xx3) (0/(128x12xx)) 10/ 10/(32x3x8)), 1/29, 1/29, 1/2")
Flow-lengths (K)
[4, 8, 8, 8, 8.8]
A Affine-coulling-sliittyyee
|2C,252] 28, 254, 25..) [2C, 252, 25,,222,25 25,, 28p (8C), [8C]. [8C]
A fffine-cuppling-ente length (N)
[4, 4, 4, 4, 4, 4|
1 x 1 invertible convolution kernel
frree-form free-form, free-form, free-form, unitary, unitary]
Total latent space dimensionality y A Affinc-coupligg conditioning channels Affine-coupling hidden layer channels Encoder hidden layer channels Decoder hidden layer channels
271360
[16, 32, 64, 128]
[32, 64, 128, 128
[64, 256, 256, 512)
(64, 256, 512, 512)
Table 5: Model hyperparameters for our reference Glow implementation using notation of Kingma & Dhari- 20188. We use the Adamax variant of Adam for optimization, with learning rate 5 x 10-3 and batch size 16. The data is dequantized to 8 bits. Gradient magnitude is clipped at 50.0.
wal
Name
Values
Levels (L)
5
Depth per level (K)
24
Coupling type
Additive
Hidden channels coupling layers
256
23


Reviews:
[{"review_id": "jGCSjBNjN9", "reviewer": "Reviewer_xzyC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n\n1) The authors improve the sampling quality of flow-based generative models.\n\n2) They demonstrate qualitatively that i) lower resolution layers are capable of capturing global features of the image. ii) higher resolution stochastic layers do still yield variations in the image at finer details. this is in contrast to glow models where these layers do not affect the sampled images.\n\nweaknesses:\n\n1) albeit the proposed model outperforms the glow baseline, it might be good if the authors give a picture of where their model stands in terms of other generative models like vaes. \n\n2) they should also compare with glow in terms of sampling time and model size (trainable parameters)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a generative model that is based on conditional normalizing flows. The conditioning factor enables a multi-scale architecture where each layer in the hierarchy generates the conditioning factor of the next layer with latent space at each layer having a different spatial resolution. The model resembles hierarchical VAEs with normalizing flows for the encoder and prior distributions of the latent space and a normalizing flow in the decoder but with an alternative regularization scheme.\n\nThe authors test the suggested architecture on CelebA-HQ and LSUN churches and bedrooms in terms of FID compared to glow. More importantly, they qualitatively assess the behavior of the latent space at different resolutions showing its capability to capture finer detail at higher resolutions (a  weakness of existing flow-based models).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "****** regarding the prior regularization scheme in equation 4. *****\n\n1) Do the authors think that such a loss can have a probabilistic/ information theoretic interpretation? \n\n2) Could a regularization coefficient achieve a better trade-off between reconstruction loss and regularization?\n\n3) The prior portion of the model resembles prior networks [1] albeit in [1] the prior network is not trainable (hence implicitly avoiding degenerate solutions). I think the authors should connect these two concepts\n\n4) Slightly improve writing for better comprehensibility: lacking the entropy term originating from the KL--> lacking the negative entropy term of the encoder.\n\n5) The risk of degenerate encoders is also discussed in [2] (prior to  cited Xiao's work) and solved by beta-VAEs (beta<1). For completeness, I also think the authors should cite and comment on [2]\n\n\n****** regarding the architectural depth  *****\n\n1) I think the authors at this point should also report number of layers (flows) in the architecture and number of transformations per flow for a fair comparison with layers in the VAEs. report of trainable architectures for both model categories might their argument stronger.\n2) there are actually more recent deep vaes with shallower architectures [3] which are also not commented [3].\n\n\n***** notational issues *****\n\n1) there seems to be a contradiction in the notation used:\n\nright above equation 4: \n\n$z_{prior}=f_{prior}(y;\\phi)$\n\nin equation 7, they define $z_{prior}$ as the inverse of f_{prior}\n\n2) minor: in section 2.2. The noise can be though**t**\n\nReferences\n\n[1] Osband, I., Aslanides, J., and Cassirer, A. (2018). Randomized prior functions for deep reinforcement learning. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems 31, pages 8617–8629. CurranAssociates, Inc.]\n\n[2]Hoffman MD, Riquelme C, Johnson MJ. The β-vae’s implicit prior. InWorkshop on Bayesian Deep Learning, NIPS 2017 (pp. 1-5).\n\n[3] Apostolopoulou I, Char I, Rosenfeld E, Dubrawski A. Deep Attentive Variational Inference. InInternational Conference on Learning Representations 2021 Oct 6.", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BQnuMf669V", "reviewer": "Reviewer_ugi5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Claim and Evidence**\n\nIf I understand correctly, this paper points out four problems of Glow and proposes a method to remedy them:\n1. difficult to model multi-modal distribution\n2. aliasing artifacts in generated images\n3. limited variations of generated images\n4. uneven contribution of decomposed latent representations\n\n- For the first problem, if I get all the information, this paper does not verify it, at least directly. However, I think the fact that FID scores are improved in image generation tasks on two image datasets (CelebA-HQ 256x256 and LSUN) can be considered as evidence to support this claim.\n- For the second problem, Figure 1 qualitatively evaluates the improvement.\n- For the third and fourth problems, Figures 5 and 6 evaluate the effect of $\\boldsymbol{z}$'s changes on generated images. Although both are qualitative evaluations of a single image, they are consistent with their claims, at least for the example presented.\n\n\n**Audience**\n\nThe Flow-based model is one of the most promising image generation models, along with the score-based, GAN-based, and Transformer-based models. Its improvement is an important issue since the aliasing problem of Glow is directly related to the impression of the generated images. Therefore, this paper should be of interest to TMLR readers.\n\n\n**Weaknesses**\n\nThe proposed method certainly improves over Glow by the comparisons from various aspects. However, as pointed out in this paper, this paper does not compare the proposed models with other generative models, such as GAN-based and score-based models. In fact, according to [1], the SOTA of CelebA-HQ 256x256 FID is 3.25 of the GAN-based model, and the FID of SOTA of the score-based model is also significantly different from the proposed method.\n\n[1] https://paperswithcode.com/sota/image-generation-on-celeba-hq-256x256", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new invertible generative model using hierarchical normalizing flows. The U-Net-based encoder-decoder model creates an intermediate representation $\\boldsymbol{y}$, which has fine-to-coarse spatial hierarchical structures. Then, the model transforms an image and intermediate representations into latent representations $\\boldsymbol{z}$ using invertible flow models at each level. This paper applies the proposed method to the CelebA-HQ 256x256 and LSUN datasets and quantitatively evaluates its image generation performance using FID. For CelebA-HQ 256x256, the proposed method is quantitatively evaluated by visualizing generated images and latent representations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- P.3, Eq. (3): *... where $p\\_\\theta$ is the distribution induced by a conditional normalizing flow $\\boldsymbol{z}\\_{\\text{cond}} = f\\_{\\text{cond}}(\\boldsymbol{x}; \\boldsymbol{y}, \\theta)$ with a unit Gaussian prior.*: It may not be easy to understand this part mathematically. Is $p\\_\\theta$ a probability distribution constructed by push-forwarding $\\boldsymbol{z}\\_{\\text{cond}}\\sim \\mathcal{N}(0, I)$ using $f^{-1}\\_{\\text{cond}}(\\cdot; \\boldsymbol{y}, \\theta)$\n- P.3, Eq. (4): Similarly, is $p_\\varphi$ a probability distribution constructed by push-forwarding $\\boldsymbol{z}\\_{\\text{prior}}\\sim \\mathcal{N}(0, I)$ using $f^{-1}\\_{\\text{prior}}$?\n- P.4: [...] we optimize minimize [...] -> [...] we optimize to minimize [...]\n- P.5: The noise can be though as [...] -> The noise can be thought of as [...]\n- P.8, Figure 4: What exactly does the term \"truncated sampling\" used in image generation (e.g., Figure 4) refer to? Also, how does the parameter $\\sigma$ work there?", "broader_impact_concerns": "This paper does not explicitly mention Broader Impact. However, similarly to other image generation technology,  we may carefully consider the social negative impacts of its misuse, such as fake images.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uSKoKD8gYW", "reviewer": "Reviewer_WMXD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ This is a well written paper that is relatively easy to follow. The clarity can be improved further by adding an algorithm box for sampling and inference in the model, though this is minor.\n\n+ The experimental evaluation and hyper-parameter analysis is sufficiently thorough, and it is clear that this approach works and can improve over GLOW.\n\n+ I appreciate that the authors are clear about the current limitations of this work in Section 4, which is important.\n\n- The main improvement of the proposed is to GLOW, which is 5 years old and hasn’t been SOTA for image generation purposes in a long time. However, for the purposes of closing the gap of flow-based to diffusion/gan-based models, this work is certainly valuable.\n\n- The model is quite complex and sensitive to hyper-parameter values (for which no good heuristics are available), though the authors clearly acknowledge this.\n\n- No literature from 2023 is covered and only a single paper from 2022. While I am not very familiar with the latest state of flow-based generative models, it would make sense to me to try include a discussion of more recent advances in generative (flow-based) models more generally.\n\n - The main selling point of using a flow-based model seems to be the ability to perform exact inference and compute exact likelihoods. However, the paper is mainly concerned with improvements in terms of FID. While this is understandable, as the comparison is to another flow-based model, it would have also made sense to include some results that better demonstrate the benefit of flow-based models compared to diffusion/gan-based models even though those are not directly compared to.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel flow-based hierarchical generative model. It works by organizing multiple shallow conditional normalizing flows in a hierarchy. A deterministic encoder (with added noise) computes the conditioning information for each level of the hierarchy. The decoder leverages conditioning normalizing flows at each “layer” to generate the image. A learned prior (also a flow-based model) models the aggregate posterior for sampling the initial latent.\n\nThe proposed model mainly presents improvements compared to GLOW (Kingma & Dhariwal, 2018) in terms of FID. It is also shown how the hierarchical structure of the model is better able to capture variations to the generated images at multiple levels of abstraction. The proposed model is quite complex, and a hyper-parameter analysis reveals that it is also quite sensitive to some of the hyperparameter choices, especially the amount of encoder noise added. Though once properly tuned the model performs is able to improve substantially compared to GLOW.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper is sufficiently polished and ready for publication, though I have listed some suggestions for improving this work in the comments above.  ", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an interesting new invertible generative model using hierarchical normalizing flows and demonstrates performance improvements of the suggested approach over Glow.  All reviewers were positive about the paper and recommended acceptance.  

As there were not any notable issues outstanding after the rebuttal, I am happy to recommend the acceptance of the paper "as is".  I would still encourage the authors to add the sampling times requested by Reviewer xzyC in the final version.
Decision: Accept (poster)