Multi-Resolution Continuous Normalizing Flows
Authors: Vikram Voleti
Chris Finlay
Adam M Oberman
Christopher Pal
Venue: ICLR 2022 Submitted (2022)
Keywords: Normalizing flows
generative models
neural ode
continuous normalizing flows
computer vision

Abstract: Recent work has shown that Neural Ordinary Differential Equations (ODEs) can serve as generative models of images using the perspective of Continuous Normalizing Flows (CNFs). Such models offer exact likelihood calculation, and invertible generation/density estimation. In this work we introduce a Multi-Resolution variant of such models (MRCNF), by characterizing the conditional distribution over the additional information required to generate a fine image that is consistent with the coarse image. We introduce a transformation between resolutions that allows for no change in the log likelihood. We show that this approach yields comparable likelihood values for various image datasets, using orders of magnitude fewer parameters than the prior methods, in significantly less training time, using only one GPU.

Under review conference paper at ICLR 2022
MULTI-RESOLUTION CONTINUOUS NORMALIZING FLOWS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has shown that Neural Ordinary Differential Equations (ODEs) can serve as generative models of images using the perspective of Continuous Normalizing Flows (CNFs). Such models offer exact likelihood calculation, and invertible generation/density estimation. In this work we introduce Multi-Resolution variant of such models (MRCNF), by characterizing the conditional distribution over the additional information required to generate fine image that is consistent with the coarse image. We introduce a transformation between resolutions that allows for no change in the log likelihood. We show that this approach yields comparable likelihood values for various image datasets, using orders of magnitude fewer parameters than the prior methods, in significantly less training time, using only one GPU.
INTRODUCTION
Reversible generative models derived through the use of the change of variables technique (Dinh et al. 2017, Kingma & Dhariwal 2018: Ho al., 2019a| Yu et al. 2020) are growing in interest as alternatives to generative models based on Generative Adversarial Networks (GANs) (Goodfellow et al., (2066) and Variational Autoencoders (VAEs) (Kingma & Welling 2013). While GANs and VAEs have been able to produce visually impressive samples of images, they have a number of limitations. A change of variables approach facilitates the transformation of a simple base probability distribution into a more complex model distribution. Reversible generative models using this technique attractive because they enable efficient density estimation, efficient sampling, and computation of exact likelihoods.
A promising variation of the change-of of- variable approach is based on the of continuous time variant of normalizing flows (Chen et al. 2018a] Grathwohl et al. 2019)] Finlay et al. 2020). which uses an integral over continuous time dynamics to transform base distribution into the model distribution, called Continuous Normalizing Flows (CNF). This approach uses ordinary differential equations (ODEs) specified by a neural network, Neural ODEs. CNFs have been shown to be capable of modelling complex distributions such those associated with images.
While this new paradigm for the generative modelling of images not as mature as GANs or VAEs in terms of the generated image quality, it is a promising direction of research as it does not have some key shortcomings associated with GANs and VAEs. Specifically, GANs are known suffer from mode-collapse (Lin et 2018), and are notoriously difficult to train (Arjovsky & Bottou| (017) compared to feed forward networks because their adversarial loss seeks a saddle point instead of local minimum (Berard et al.) 2020). CNFs are trained by mapping images to noise, and their reversible architecture allows images to be generated by going in reverse, from noise to images. This leads to fewer issues related to mode collapse, since any input example in the dataset can be recovered from the flow using the reverse of the transformation learned during training. VAEs only provide a lower bound on the marginal likelihood whereas CNFs provide exact likelihoods. Despite the many advantages of reversible generative models built with CNFs, quantitatively such methods still do not match the widely used Fréchet Inception Distance (FID) scores of GANs or VAEs. However their other advantages motivate us explore them further.

---

Under review conference paper at ICLR 2022
NOISE
IMAGE
A Ni. = ( 23
X3
Y2
W(O, fo0,)or11) Z2
92
M
X2
Y1
M
co00.o01)- Z1
X1
The architecture our viewed in color). Continuous normalizing flows (CNFs) Is are used to generate images x, from noise z, at each resolution, with those at finer resolutions conditioned (dashed lines) on the coarser image one level above X,+1. except the coarsest level where it unconditional. Every finer CNF produces intermediate image ys, which is then combined with the immediate coarser image x,+1 using a linear map M to form X,-
Figure 1:
Continuous Normalizing Flow (MRCNF) method (best
Furthermore, state-of-the the GANs and VAEs exploit the multi-resolution properties of images, and recent top-performing methods also inject noise at each resolution (Brock et al. 2019 Shaham et al. 2019 Karras et al. 2020 Vahdat & Kautz 2020). While shaping noise fundamental to normalizing flows, only recently have normalizing flows exploited the multi-resolution properties of images. For example, WaveletFlow (Yu jet al.] 2020) splits an image into multiple resolutions using the Discrete Wavelet Transform, and models the average image each resolution using normalizing flow. While this method has advantages, it suffers from many issues such as high parameter count and long training time.
In this work, consider non-trivial multi-resolution approach continuous normalizing flows, which fixes many these issues. A high-level view of our approach shown in Figure I
Our main contributions are:
1. We propose multi-resolution transformation that does not add cost in terms of likelihood.
2. We introduce musti-Resolution Continuous Normalizing Flows (MRCNF).
3. We achieve comparable Bits-per-dimension (BPD) (negative log likelihood per pixel) on image datasets using fewer model parameters and significantly less training time with only one GPU.
2 BACKGROUND
2.1 NORMALIZING FLOWSS
Normalizing flows Tabak & Turner 2013 Jimenez Rezende & Mohamed, 2015 Dinh et al. 2017 Papa makarios al. 2019 Kobyzev et al. (2000) are generative models that map a complex data distribution, such real images, to known noise distribution. They are trained by maximizing the log likelihood of their input images. Suppose a normalizing flow g produces output z from an input x i.e. z = g(x). The change-of-var variiables formula provides the likelihood of the image under this transformation as:
dg dx + pz)
log p(x) log dct
(1)
The first term the right (log determinant of the Jacobian) is often intractable, however, previous works normalizing flows have found ways to estimate this efficiently. The second term, log p(z). is computed as the log probability of under known noise distribution, typically the standard Gaussian N0, I).
2

---

Under review conference paper at ICLR 2022
2.2 WAVELET FLOW (YUE) AL. 2020
WaveletFlow decomposes image using the Discrete Wavelet Transformation, and maps the average image at each resolution to noise using normalizing flow. The WaveletFlow model builds on the Glow (Kingma & Dhariwal 2018) architecture, with high parameter count and a long training time. It uses an orthogonal transformation, which does not preserve the range of the image, and adds a constant term to the log likelihood each resolution. We aim to fix these issues using MRCNF.
2.3 CONTINUOUS NORMALIZING FLOWS
Continuous Normalizing Flows (CNF) (Chen et al. 2018a Grathwohl et al. 2019; Finlay et al. 2020) are a variant of normalizing flows that operate the continuous domain. CNF creates geometric flow between the input and target (noise) distributions, by assuming that the state transition is governed by an Ordinary Differential Equation (ODE). further assumes that the differential function parameterized by a neural network, this model is called Neural ODE (Chen et al.) 201 Suppose CNF y transforms its state v(!) using Neural ODE, with neural network defining the differential. Here, v44)) is, say, an image, and at the final time step v44) is sample from known noise distribution.
dv(() f(v(1,4) = v(4) g(((())) = v(lo) + 1." I(v()),)) di dt
(2)
This integration is typically performed by an ODE solver. Since this integration can be run backwards as well obtain the same v(to) from v(t1), CNF is reversible model.
Equation can be used to compute the change in log prebaabbbiity induced by the CNF. However, Chen et al. (2018a) and Grathwohl et al. (2019) proposed more efficient variant in the context of CNFs, called the formula:
instantanoous change-of-vvrialle
i log ((v()) -Tr at
Of @v(t)
A Pogo)((t) = /.(
Ōf (v(L)
(3)
Hence, the change in ses-rbbabblityy of the state of the Neural ODE i.e. A log Pv is expressed another differential equation. The ODE solver now solves both differential equations eq. (2) and eq. (3) by augmenting
the original state with the above. Thus, CNF provides both the final state v(11) as well the change in log probability A log Pog.)(()) (v(()) together.
Prior works (Grathwohl al. 2019 Finlay al. 2020 Ghosh al., 2020 Onken al. 2021 Huang &
Ych (021) have trained CNFs as reversible generative models of images, by maximizing the likelihood of the
images under the model:
(x)
pog)(x) A log Px-x + log
(4)
where x is an image, z and A log Px-+z computed by the CNF using eq. (2) and eq. 3 and p(z) is the likelihood of the computed z under a known noise distribution, typically the standard Gaussian N (O, I). Novel images generated by sampling z from the known noise distribution, and running it through the CNF
in reverse.
3
OUR METHOD
Our method a reversible generative model images that builds on top of CNFs. We introduce the notion of multiple resolutions in images, and connect the different resolutions in autoregressive fashion. This helps generate images faster, with better likelihood values at higher resolutions, using only one GPU in all experiments. We call this model Multi-Resolutie Continuous Normalizing Flow (MRCNF).
3

---

Under review conference paper at ICLR 2022
3.1 MULTI-RESOLUTION IMAGE REPRESENTATION
Multi-resolution representations of images have been explored in computer vision for decades (Burt 1981 Marr 2010; Witkin 1987; Burt & Adelson 1983 Mallat 1989 Lindeberg 1990). This implies that much of the content of an image at a resolution is composition of low-level information captured at coarser resolutions, and some high-level information not present the coarser images. We take advantage of this image in resolution space i.e. by expressing it series of S images
property by first decomposing at decreasing resolutions: x - Xx,, .x)). where X1 = x is the finest image, Xş is the coarsest, and every X is the average image of x,, This called an image pyramid, upsampling-dowssappiinn ct al. 1984, Witkin 1987; Lindeberg 1990). In this work, we obtain coarser image simply by averaging pixels in every 2x2 patch, thereby halving the width and height.
Gaussian Pyramid if the 1983] Adelson
operations include a Gaussian filter (Burt 1981 Burt & Adelson
However, this representation is redundant since much of the information in x is contained X Instead, we express x as a series of high-level information y, not present in the immediate coarser images X and a final coarse image Xg:
(y1, X2) + (y1. -
(5)
Our overall method is to map these S terms to S noise samples using S CNFs.
3.2 DEFINING THE HIGH-LEVEL INFORMATION
We choose to design a linear transformation with the following properties: 1) invertible i.c. it should be possible to deterministically obtain x, from ys and x and vice versa 2) volume preserving i.e. determinant is 1, change in log-likelihood is 0 3) angle preserving and 4) range preserving (under the notion of the maximum principle (Varga 1966)).
(-
(- (-2:
2 c -4)
Consider the simplest case of 2 resolutions where X is a 2x2 im- age with pixel values I1, T2, F3, 24. and X2 is a 1x1 image with pixel value F = { + I2 + + x1). We require three values (V1, V2, 9/) yı that contain information not present X2, such that X1 is obtained when y1 and X2 are combined.
Figure 2: Tetrahedron in 3D space with 4 corners
This could be viewed as a problem of finding matrix M such that: [x1, 22, 23, x. M [y1, 1/2 Vs a) We fix the last column of M as [1, 1, 1, 1] since every pixel value in X1 depends on E. Finding the rest of the parameters can be viewed as requiring four 3D vectors that are spaced such that they do not degenerate the number of dimensions of their span. These can be considered as the four corners of tetrahedron in 3D space, under any configuration (rotated 3D space), and any scaling of the vectors (see Figure 2}.
Out of the many possibilities for this tetrahedron, we could choose the matrix that performs the Discrete Haar Wavelet Transform [Mallat 1989; Mallat & Peyre 2009):
21
7
1
y/
y/1
?
.22
y2
1/2
7
(6)
23
V3
1/3
Ø33
[4_
[44_
However, this has log |det(M- 1) log(1/2) (cq. (6)) and is therefore not volume preserving. Other simple
scaling of eq. (6) has been used in the past, for example multiplying the last row of eq. (6) by 2, yielding orthogonal transformation, such as in WaveletFlow (Yu et al. 2020). However, this transformation neither preserves the volume i.e. the log determinant is not O, nor the maximum i.e. the range of x, changes.
4

---

Under review conference paper at ICLR 2022
We wish to find transformation M where: one of the results is the average of the inputs, क; it is unit determinant; the columns are orthogonal; and it preserves the range of ε. Fortunately such matrix exists although we have not seen it discussed in prior literature. It can be seen as a variant of the Discrete Haar Wavelet Transformation matrix that is unimodular, i.e. has a determinant of 1 (and is therefore volume preserving), while also preserving the range of the images for the input and its average:
Z1
c c a
y/1
y/1
c
-c -c
22
c
C -c a
1/2
9/2
c
Z2
(7)
23
a
C
1/3
1/3
c-
-c
-c-
33
244
C
e
0
aa
a a
[44_
where c 22/3, a = 4. Hence, log [det(M-))] log(1) = 0. This can be scaled up to larger spatial regions
by performing the same calculation for each 2x2 patch. Let M be the function that uses matrix M from above and combines every pixel in X with the three corresponding pixels in y, to make the 2x2 patch at that location in x, using eq. 7
X, (y.; - ys, X = (x.)
(8)
Equation can be used to compute the change in log likelihood from this transformation x, - (y,, Xs+1):
log p(,) = A log Px, (y,.x,.). + log p(y..
log /dt(
poy, xx+1) + log p(x++1)
(9)
where log ( = dims (1)2 in the case of eq. 6). where "dims" is the number of pixels
times the number of channels (typically 3) in the image, and log det(M
0 for eq. (7). Although a
model could learn to compensate for the additional log-likelihood due to eq. (6). we find that using eq.
improves log likelihood values and decreases training time.
3.3 MULTI- CONTINUOUS NORMALIZING FLOWS
Using the multi-resolltion image representation in eq. (5) we characterize the conditional distribution over the additional information (y,) required to generate higher resolution image (x,) that is consistent with the average (X++1) over the pixel space. This framework ensures that one coarse image could generate several potential fine images, but these fine images have the same coarse image as their average. This fact preserved across resolutions. Note that the 3 additional pixels in ys per pixel in X are generated conditioned on the entire coarser image x,+1, thus maintaining consistency using the full context.
At each resolution s, we use CNF to reversibly map between ys (or Xs when S) and sample z, from a known noise distribution.
In principle, any generative model could be used to map between the multi-resolution image and noise. Normalizing flows are good candidates for this as they are probabilistic generative models that perform exact likelihood estimates, and can be run in reverse to generate novel data from the model's distribution. This allows model comparison and measurement of generalization to unseen data. We choose the CNF variant of normalizing flows at each resolution. CNFs have recently been shown to be effective in modeling image distributions using a fraction of the number of parameters typically used in normalizing flows (and non flow-based approaches), and their underlying framework of Neural ODEs have been shown to be more robust than convolutional layers (Yan et al. 2020).
Training:
We train an MRCNF by maximizing the average log-likelihood of the images in the training dataset under the model. The log probability of each image log p(x) can be estimated recursively from eq. as:
log p(x)= = A log Pxi x2)) + log p(y1, x2) = log Pxp -())x)) + log x2) + log p(x))
$-1
A log Px, x + plys xx+1)) + log logs(x))
(10)
5

---

Under review conference paper at ICLR 2022
where A log Px, (y,. is log [de((M plys xs+1) and logp(xs) are given by eq.
(4)
Zg = gy Xs+1)
log p(y, Xs+1) = A log [x,)== log p(zs)
(11)
3. = gs (xs)
logp(xs) = A log -xg + logp(zs)
(12)
The coarsest resolution S can be chosen such that the last CNF operates on the image distribution small enough resolution that is easy to model unconditionally. All other CNFs are conditioned on the immediate courser image. The conditioning itself is achieved by concatenating the input image of the CNF with the coarser image. This model could be seen as a stack of CNFs connected in an autoregressive fashion.
Typically, likelihood-basee generative models are compared using the metric of per--dieensiooon (BPD), i.c. the negative log likelihood per pixel in the image. Hence, we train our MRCNF to minimize the average BPD of the images in the training dataset, computed using eq. (13):
BPD(x) - log p(x) /dims(x)
(13)
We use FFJORD (Grathwohl et al.) 2019) as the baseline model for our CNFs. In addition, we use to two regularization terms introduced by RNODE (Finlay et al.) (020) to speed up the training of FFIORD models by stabilizing the learnt dynamics: the kinetic energy of the flow K((O, and the Jacobian norm B(0):
[(()) - 1 (1 de; x(()=
B(0) /' ||e ()) de, c0,
(14)
Parallel training: Note that although the final log likelihood p(x) involves sequentially summing over values returned by all S CNFs, the log likelihood term of each CNF is independent of the others. Conditioning is done using ground truth images. Hence, each CNF can be trained independently, in parallel.
Given S-resolution model, we first sample 2 s The CNF ga at resolution transforms the noise sample z, to high-level information y, conditioned on the immediate coarse image X (except Is which is unconditioned). y, and X,+1 then combined to form x, using M from eq. (7). This process is repeated progressively from coarser to finer resolutions, until the finest resolution image X1 is computed (see Figure I). It is to be noted that the generated image at one resolution is used to condition the CNF at the finer resolution.
Generation:
L...., S from the latent noise distributions.
Xs 95 (zs)
$ S
(15)
y = 9. (z. xs+1);
x, = 3.=M(y.: Xs+1)
8 = S-1 +1
3.4 MULTI-RESOLUTION NOISE
We further decompose the noise image as well into its respective coarser components. This means that ultimately we use only one noise image at the finest level, but it is decomposed into multiple resolutions using eq. 77. Xs+1 mapped to noise of a quarter variance, while y, is mapped to noise of c-factored variance (see fig. Although this is optional, preserves interpretation between the single- and multi-resolution models.
4 RELATED WORK
Multi-resolution approaches already serve key component of state-of the GAN (Denton et al. 2015
Karras et al., 2018 Karnewar& & Wang] 2020) and VAE (Razavi et al. 2019) Vahdat & Kautz 2020) based
deep generative models. Deconvolutional CNNs (Long et al. 2015 Radford et al. 2015)
upsampling
layers to generate images more effectively. Modern sfate-of-the. the-art generative models have also injected noise
different levels improve sample quality (Brock al. 2019
Karras et al. 2020 Vahdat & Kautz 2020
6

---

Under review conference paper at ICLR 2022
Several prior works on normalizing flows (Kingma & Dhariwal 2018 Hoogeboom al. 201 Song iet al. 2019: Ma et al. 2019; Durkan et al.. 2019; Chen et al. 2020 Ho et al. 2019a Lee et al., 2020 Yu et al. 2020 build on RealNVP (Dinh et al. 2017). Although they achieve great results terms of BPD and image quality, they nonetheless report results from significantly higher number of parameters (some with 100x!). and several times GPU hours of training. However, many prior works don't report these metrics.
STEER (Ghosh et al. 2020) introduced temporal regularization to CNFs by making the final time of integration stochastic. However, we found that this increased training time without significant BPD improvement.
Comparison to WaveletFlow: We emphasize that there are important and crucial differences between our MRCNF and WaveletFlow. We generalize the notion a multi-resolution image representation (section 3.2). and show that Wavelets are one case of this general formulation. WaveletFlow builds on the Glow (Kingma & Dhariwal 2018) architecture, while ours builds on CNFs (Grathwohl et al.] 2019. Finlay et al. 2020). We also make use of the notion of mulli-rrsolutiion decomposition of the noise, which is optional, but not taken into account by WaveletFlow. WaveletFlow uses an orthogonal transformation which does not preserve range: our MRCNF uses eq. 7 which is wolume-preserving and range- preeeerrsevvi...... Finally, WaveletFlow applies special sampling techniques to obtain better samples from its model. We have so far not used such techniques for generation, but we believe they can potentially help our models as well, By making these important changes, we fix many of the previously discussed issues with WaveletFlow. For more detailed ablation study, please check (subsection 5.1
"Multiple scales" in prior normalizing flows: Normalizing flows (Dinh et al. 2017 Kingma & Dhariwal 2018, Grathwohl et al., 2019) try be "multi-scale" by transforming the input smart way (squeezing operation) such that the width of the features progressively reduces in the direction of image to noise, while maintaining the total dimensions. This happens while operating at a single resolution. In contrast, our model stacks normalizing flows at multiple resolutions in an autoregressive fashion by conditioning the images coarser resolutions.
5 EXPERIMENTAL RESULTS
We train MRCNF models on the CIFARIO (Kizhevssyy al. 2009) dataset at finest resolution of 32x32, and the ImageNet (Deng et al.) 2009) dataset at 32x32, 64x64, 128x128. We build on top of the code provided in Finlay et al. (2020 Our code is attached in the supplementary material, In all cases, we train using only one NVIDIA RTX 20280 Ti GPU with LIGB.
we compare our results with prior work in terms of (lower is better in all cases) the BPD of the images of the test datasets under the trained models, the number of parameters used by the model, and the number of GPU hours taken to train. The most relevant models for comparison are the -rresolution FFJORD (Grathwohl& et al., 2019) models, and their regularized version RNODE (Finlay et al., 2020), since our model directly converts their architecture into multi multi-resolution. Other relevant comparisons are previous flow-based methods (Dinh et al.) [017)] Kingma& & Dhariwal 2018 (Sog al., 2019 Ho et al., 2019a Yu et al. 2020), however their core architecture (RealNVP (Dinh et al. 2017) is quite different from FFJORD.
Table 1,
BPD: At lower resolution spaces, achieve comparable BPDs in lesser time with far fewer parameters than previous normalizing flows (and non flow-based approaches). However, the power of the multi-resolution formulation is more evident at higher resolutions: achieve better BPD for ImageNet64 and ImageNetl 28 TTblle 2) with significantly fewer parameters and lower time using only one GPU.
It is be noted that we were not able to reproduce the same BPD as provided by STEER (Ghosh et al. 2020), we report the results of our re-implementation in Table 1 A more complete table can be found in the appendix. We also include analysis of out-of-distribution properties in the appendix.
7

---

Under review conference paper at ICLR 2022
Table Bits- cits-per-dimensoon (lower is better) of images in the corresponding evaluation for CIFARIO, ImageNet 32x32, and ImageNet 64x64. We also report the number of parameters in the models, and the time taken to train (in GPU hours). All our models were trained on only one GPU.
Blank spaces indicate unreported values. x':
+As reported in Ghosh et al. (2020). &Re-inplemented by us.
Fails to train. *RNODE (Finlay et al. 2020) used GPUs to train on ImageNe164.
CIFARIO
IMAGENET32
IMAGENET64
BPD PARAM TIME
BPD PARAM TIME
BPD PARAM TIME
Non Flow-based Prior Work
Gated PixcICNN (Van den Oord ct
2016
3.03
3.83
60
3,57
60
SPN (Menick & Kaichbrennert 2011
3.85
150.0M
3.53
150.0M
Sparse Transfornt Chuldred al 2019
2,80
59.0M
3,44
152.0M
7days
NVAE (Vahdat (020
2.91
55
3.92
70
DistAug af
2.56
152.0M
3.42
152.0M
Flow-based Prior Work
RealN (Dinh et al. (217))
(017))
3,49
4.28 46.0M
3.98
96.0M
Glow
(Kingma Phariwal
2018)
3.35
44.0M
4.09 66.1M
3.81
111.IM
MaCow Ma (0119)
3.16
43.5M
3.69
122.5M
Flow++ ffo et 201
3.08
31.4M
3.86
169.0M
3.69
73.5M
Wavelet Prow YYu al
202
4.08 64.0M
3.78
96.0M
822
DenseFlow (Greic et 2021
2.98
250
3.63
310
3.35
224
I-Resolution Continuous Normalizing Flow
FFIORD Grathwohl al.
2019)
3.40
0.9M
25dyys
3.96
1.0M
>5dass
RNODE Finlay et al.
2020
3,38
1.4M
31.8
2.36
2.0M
300.
*3.83
2.0M 2556.4
$3.49
$1.6M
$40.4
FFIORD STEER
Ghosh al.
2020
3.40
1.4M
86.3
3.84
2.0M
>days
RNODE + STEER
Ghosh et al.
2020)
3.397
1.4M
22.2
2.35
2.0M
24.9
$3.49
$1.6M
$30.1
(OURS) Multi-Resolutiin Continuous Normalizing Flow (MRCNF)
2-resolution MRCNF (small)
3.65
1.3M
19.8
3.77
1.3M
18.2
3,44
2.0M
42.3
2-resolution MRCNF (big)
3.54
3.3M
36.5
3.78
6.7M
18.0
6.7M
X
3-resolution MRCNF (small)
3.79
1.5M
17.4
3.97
1.5M
13.8
3.55
2.0M
35.4
3-resolution MRCNF (big)
3.60
5.1M
38.3
3.93
10.2M
41.2
7.6M
Train time: All our experiments used only one GPU, and took significantly less time to train than I-resolution CNFs, and all prior works including flow-basee and non-flow-basee els. Since all the CNFs can be trained in par- allel, the actual training time in practice could be much lower than reported.
GT
mod-
32x32
Gen
64x64
Super-resolution: Our formulation also al- lows for super-resolution of images (Figure 3 free of cost since our framework autoregres-
64x64
Figure 3: Examples of super-resolving from ImageNet 16x16
64x64. GT: ground truth, Gen: generated.
sive in resolution.
Progressive training: We trained an MRCNF model on ImageNet 28 by training only the finest resolution (128x128) conditioned on the immediate coarser (64x64) images, and at- tached it 3-resolution model trained on ImageNet64. The resultant 4-resolution Ima- geNet128 model gives BPD of 3.31 (Table 2
Table 2: Metrics unconditional ImageNet128 generation.
IMAGENET128
BPD
PARAM TIME
Parallel Multiscale (Reed 2017
3.55
SPN (Menick Kalchbrenner 2019
3.08
250.00M
(OurS) 4-resolution MRCNF
3.31
2.74M
58,59
with just 2.74M parameters in 260 GPU hours.
8

---

Under review conference paper at ICLR 2022
5.1 ABLATION STUDY
Our MRCNF method differs from WaveletFlow in three respects: 1. CNFs, 2. eq. 7 instead
of eq. (6) as used by WaveletFlow, 3. we use multi-resolution noise. We check the individual effects of these changes an ablation study in Table 3 and conclude that:
Table 3: Ablation study across using Wavelet in eq. (6. and multi- noise formulation in 3.4
CIFARIO
IMAGENET64
BPD PARAM TIME
BPD
PARAM
TIME
WaveletFlow (Yu et al. 2020
3.78
98.0M
822.00
I-resolution CNF (RNODE) Finlay et al. 2020
3.38
1.4M
31.84
3.83
2.0M
256.40
2-resolution
eq.
WaveletFlow, but with CNFs, w/o multi-res noise
3.68
1.3M
27.25
X
2.0M
X
eq.
WaveletFlow, but with CNFs, w/ multi-res noise
3.69
1.3M
25.88
X
2.0M
X
eq.
MRCNF w/o multi-res noise
3.66
1.3M
19.79
3.48
2.0M
42.33
eq.
MRCNF w/ multi-res noise (Ours)
3.65
1.3M
19.69
3.44
2.0M
42.30
3-resolution
eq.
WaveletFlow, but with CNFs, w/o multi-res noise
3.82
1.5M
22.99
3.62
2.0M
43.37
eq.
WaveletFlow, but with CNFs, w/ multi-res noise
3.82
1.5M
25.28
3.62
2.0M
44.21
MRCNF w/o multi-res noise
3.79
1.5M
17.25
3.57
2.0M
35.42
eq.
MRCNF w/ multi-res noise (Ours)
3.79
1.5M
17.44
3.55
2.0M
35.39
1. Simply replacing the normalizing flows in WaveletFlow with CNFs does not produce the best results. It does improve the BPD and training time compared to WaveletFlow.
2. Using our unimodular transformation in eq. 7 instead of the original Wavelet Transformation of
eq. (6) not only improves the BPD, it also consistently decreases training time. Hence, although model could learn to compensate for the additional BPD in eq. (6) we find that using eq. 7 makes training better and faster.
3. As expected, the use of multi-resolution noise does not have a critical impact on either BPD or training time. We use it anyway so to retain interpretation with I-resolution models.
Thus, our MRCNF model not a trivial replacement of normalizing flows with CNFs in WaveletFlow. We generalize the notion of multi-resolution image representation, in which the Discrete Wavelet Transform is one of many possibilities. then derived unimodular transformation that adds no change in likelihood.
6 CONCLUSION
We have presented a Multi-Resolution approach Continuous Normalizing Flows (MRCNF). MRCNF models achieve comparable or better performance in significantly less training time, training on a single GPU, with fraction the number parameters other competitive models. We achieved this by designing a general formulation for a multi-resolution image representation, and derived unimodular transformation that adds no cost terms of log- likelihoooo.. Although the likelihood values for 32x32 resolution datasets such CIFARI0 and ImageNet32 do not improve significantly over the baseline. ImageNet64 and above see marked improvement. The performance better for higher resolutions, as seen in the case of ImageNet128 28.
We also conducted ablation study note the effects of each change introduced in the formulation. We found that our unimodular transformation has a significant contribution towards improving BPD and decreasing training time.
9

---

Under review conference paper at ICLR 2022
ETHICS STATEMENT
In terms of broader social impacts of this work, generative models of images can be used to generate so-called fake images, and this issue has been discussed length in other works. We have tried to stay away from particularly concerning use of our work. We have not performed experiments on face-specific datasets. Although the ImageNet dataset contains human face images, we have only tested our method on an unconditional model, and have made no effort towards controllable generation. An increasing concern in the machine learning community that of increased use of computational resources and its impact on the environment. We emphasize lower computational budgets, and achieve comparable performance with far fewer parameters and significantly less training time.
REPRODUCIBILITY STATEMENT
To ensure reproducibility of our work, we have attached anonymized version of our code repository in the supplementary material. We have mentioned that our code builds on top of public code repository of the most relevant prior method: https//gitub..om//fllayy//
We have included the exact mathematical equations governing our method Section 3.3, which can be verified in our code. We have mentioned other details of our models in the appendix number of channels, number of blocks, corresponding total number of parameters. We have also mentioned in the appendix that we perform gradient clipping, and added relevant details there. Our code logs and plots the BPD, number of parameters, and training time for every experiment, and saves samples of generated images at regular intervals. also plots other values of interest such as change in log likelihood, log probability of mapped noise samples, etc. The code also includes options for launching experiments with configurations of prior methods including FFJORD (Grathwohl et al. (2019).
RNODE (Finlay et al. 2020), STEER (Ghosh al. 2020 and WaveletFlow (Yu et al. 2020) using CNFs.
REFERENCES
Edward H Adelson, Charles H Anderson, James R Bergen, Peter Burt, and Joan M Ogden. Pyramid methods in image processing. RCA engineer, 29(6):33-41, 1984.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint 1rXiv:1701.04800 2017.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference Machine Learning, pp. 573-582, 2019.
Hugo Berard, Gauthier Gidel, Amjad Almahairi, Pascal Vincent, and Simon Lacoste-Julien. A closer look at the optimization landscapes of generative adversarial networks. In International Conference on Machine Learning, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019.
Peter Burt and Edward Adelson. The laplacian pyramid as compact image code. IEEE Transactions on communications, 3144)533-5540 1983.
Peter Burt. Fast filter transform for image processing. Computer graphics and image processing, 16(1): 20-51, 1981.
Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. Vflow: More expressive generative flows with variational data augmentation. In International Conference Machine Learning, 2020.
10

---

Under review conference paper at ICLR 2022
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018a.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances Neural Information Processing Systems, pp. 9916-9926, 2019.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autoregressive generative model. In International Conference on Machine Learning, pp. 864-872. PMLR, 2018b.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: /904. /0509, 2019.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv:1810.01392 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. IEEE, 2009.
Emily I. Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. In Advances in neural information processing systems, pp. 1486-1494, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference Learned Representations, 2017.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems. volume 32, pp. 7511- 2019.
7522,
URL /tttps://proceedincss nwwrsss.eriipp.ooeeeee /ttp:riip..cpppppe//////////
.pdf]
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. International Conference on Machine Learning, 2020.
Arnab Ghosh, Harkirat Singh Behl, Emilien Dupont, Philip HS Torr, and Vinay Namboodiri. Steer: Simple temporal regularization for neural odes. In Advances in Neural Information Processing Systems, 2020.
lan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. International Conference on Learning Representations, 2019.
Matej Grcié, Ivan Grubišié, and Siniša Šegvić. Densely connected normalizing flows. arXiv preprint, 2021. URL 2010 O 02502
Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribttion examples in neural networks. In International Conference on Learning Representations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++? Improving flow- generative models with variational dequantization and architecture design. In International Conference on Machine Learning, 2019a
11

---

Under review conference paper at ICLR 2022
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans, Axial attention in multidimennionall transformers. arXiv preprint arxi::://2./2//820 2019b.
Emiel Hoogeboom, Rianne van den Berg, and Max Welling. Emerging convolutions for generative normalizing flows. In International Conference on Machine Learning. 2019a.
Emiel Hoogeboom, Jorn Peters, Rianne van den Berg, and Max Welling, Integer discrete flows and lossless compression. In Advances in Neural Information Processing Systems, volume 32, pp. 12134-12144, 2019b. URL https: lattps::/proceecinngg 9e933007444 9d07d81 50w8cc333cccc00 wtwp:e/wwiiinnn...iiis......o www.ipeu.i...m http:/pps.../pppee/////////L
Emiel Hoogeboom, Victor Garcia Satorras, Jakub Tomczak, and Max Welling. The convolution exponential and generalized sylvester flows. In Advances in Neural Information Processing Systems, 2020.
Anders Høst-Madsen, Elyas Sabeti, and Chad Walton. Data discovery and anomaly detection using atypicality: Theory. IEEE Transactions on information Theory, 65(9):5302 -32-2322222-2,,,-- 2019.
Chin-Wei Huang, Laurent Dinh, and Aaron Courville. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models. arXiv preprint arXiv:2002.07101, 2020.
Han-Hsien Huang and Mi-Yen Yeh. Accelerating continuous normalizing flow with trajectory polynomial regularization. AAAI Conference on Artificial Intelligence, 2021.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference Machine Learning, pp. 1530- 2015.
Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and Ilya Sutskever. Distribution augmentation for generative modeling. In International Conference on Machine Learning, pp. 10563-10576, 2020.
Mahdi Karami, Dale Schuurmans, Jascha Sohl-Dickstein, Laurent Dinh, and Daniel Duckworth. In- vertible convolutional flow. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https: hctps::/prooeeinng nwwrips. http:/pppp..cpppppp///////////// Paper--appr...o pddf
pp. 5635-5645,
Animesh Karnewar and Oliver Wang. Msg-gan: Multi-scale gradients for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7799-7808, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learned Representations, 2018.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 10-8119, 2020.
Jacob Kelly, Jesse Bettencourt, Matthew James Johnson, and David Duvenaud. Learning differential equations that are easy to solve. Advances in Neural Information Processing Systems, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6144 2013.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow. 2016. URL httlhhttp///xxii www./rgg///.//ooo////..oo..e 04934 cite arxiv: 1606.649364
12

---

Under review conference paper at ICLR 2022
Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in neural information processing systems, pp. 10215-10224, 2018.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail to detect out-of-distribution data. Advances in neural information processing systems, volume 33, 2020.
Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Re- port, University of Toronto, 2009. URL https://www
toronto. wwt.:/.ddduu......e cttrii/ccfaar.. html
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pp. 7167-7177, 2018.
Sang-gil Lee, Sungwon Kim, and Sungroh Yoon. Nanoflow: Scalable normalizing flows with sublinear parameter complexity. In Advances in Neural Information Processing Systems, 2020.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh, Pacgan:
The power of two samples in generative adversarial networks. Advances in neural information processing systems, pp. 1498-1507, 2018.
Tony Lindeberg. Scale-space for discrete signals. IEEE transactions on pattern analysis and machine intelligence, 12(3):234-25 254,, 1990.
Jonathan Long, Evan Shelhamer, and Trevor Darrell, Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3431-3440, 2015.
You Lu and Bert Huang. Woodbury transformations for deep generative flows. In Advances in Neural Information Processing Systems, 2020.
Xuezhe Ma, Xiang Kong, Shanghang Zhang, and Eduard Hovy. Macow: Masked convolutional generative flow. In Advances in Neural Information Processing Systems, pp. 5893-5902, 2019.
Stephane G Mallat. A theory for multiresolution signal decomposition: the wavelet representation. IEEE transactions pattern analysis and machine intelligence, 11(7):674-693, 1989.
Stéphane G. Mallat and Gabriel Peyré. A wavelet tour of signal processing: the sparse way. Elsevier, 2009.
David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. International Conference on Learning Representations, 2019.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? International Conference on Learning Representations, 2019a.
Eric Nalisnick, Akihiro Matsukawa, Yee whye Teh, and Balaji Lakshninarayanan. Detecting out-of- distribution inputs to deep generative models using test for typicality. arXiv preprint arXiv:1906.00 029944 5, 2019b.
13

---

Under review conference paper at ICLR 2022
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Didrik Nielsen and Ole Winther. Closing the dequantization gap: pixelenn as a single-layer flow. In Advances in Neural Information Processing Systems, 2020.
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. Ot-flow: Fast and accurate continuous normalizing flows via optimal transport. AAAI Conference on Artificial Intelligence, 2021.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. Interna- tional Conference on Machine Learning, 2016.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02762, 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference Machine Learning. 2018.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, 2013.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. arXiv preprint arXiv: 15| 1.06434, 2015.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14866-14876, 2019.
Scott Reed, Aäron den Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Dan Belov, and Nando De Freitas. Parallel multiscale autoregressive density estimation. In International Conference on Machine Learning, 2017.
Elyas Sabeti and Anders Høst-Madsen, Data discovery and anomaly detection using atypicality for real-valued data. Entropy, 21(3):219, 2019.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José Núñez, and Jordi Luque. Input complexity and out-of distriibutiinnnonnnnnnn detection with likelihood- based generative models. In International Conference on Learning Representations, 2020.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning a generative model from a single natural image. In Proceedings of the IEEE International Conference Computer Vision, pp. 4570-4580, 2019.
Jon Sneyers and Pieter Wuille. Flif: Free lossless image format based on maniac compression. 2016 IEEE International Conference on Image Processing (ICIP), pp. 66-70. IEEE, 2016.
Yang Song, Chenlin Meng, and Stefano Ermon. Mintnet: Building invertible neural networks with masked convolutions. In Advances in Neural Information Processing Systems, pp. 1 11004-11011 1014, 2019.
Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algorithms. Communi- cations on Pure and Applied Mathematics, 66(2):145- 164, 2013.
Lucas Theis, Aäron van den Oord, and Matthias Bethge. note on the evaluation of generative models. In International Conference Learning Representations, 2016.
14

---

Under review conference paper at ICLR 2022
Arash Vahdat and Jan Kautz. Nvac: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In Advances in neural information processing systems, pp. 4790-4798, 2016.
Richard S Varga. On discrete maximum principle. SIAM Journal Numerical Analysis, 3(2):355-359, 1966.
Andrew P Witkin, Scale-space filtering. In Readings in Computer Vision, pp. 329-332. Elsevier, 1987.
Changyi Xiao and Ligang Liu. Generative flows with matrix exponential. In International Conference on Machine Learning, 2020.
Hanshu Yan, Jiawei Du, Vincent Y. Tan, and Jiashi Feng. On robustness of neural ordinary differential equations. International Conference Learning Representations, 2020.
Jason Yu, Konstantinos Derpanis, and Marcus Brubaker. Wavelet flow: Fast training of high resolution normalizing flows. In Advances in Neural Information Processing Systems, 2020.
15

---

Under review conference paper at ICLR 2022
APPENDIX
A OUT-OF-DISTRIBUTION PROPERTIES OF (MR)CNF
The derivation of likelihood-based models sug- gests that the density of an image under the model is an effective measure of its likeli- hood of being in distribution. However, re-
1-resolution CNF
Normalized Empirica KDE
TinylmageNet Test
SVHN Test
Constant
cent works (Theis et al.) 2016, Nalismick et al.
CIFAR1O Test
2-resolution CNF
2019a Serrà et al.) 2020] Nalisnick et al.
2019b) have pointed out that possible that images drawn from other distributions have higher model likelihood. Examples have been shown where normalizing flow models (such as Glow) trained on CIFARIO images assign higher likelihood SVHN (Netzer et al. 2011) images. This could have serious implications on the practical applicability of these models. Some also note that likelihood-bate based models do not generate images with good sample qual- ity as they avoid assigning small probability out-of-diistibution (OoD) data points, hence using model likelihood (-BPD) for detecting OoD data is not effective.
3-resolution CNF
Log-likelihood per dimension (-BPD)
Figure 4: Histogram of log likelihood per dimension of out- of distiitritttion datasets (TinylmageNNet, SVHN, Constant) under (MR)CNF models trained on CIFARIO. As with other likelihood- based generative models such as Glow & PixeICNN, Ool datasets have higher likelihood under (MR)CNFs.
We conduct the same experiments with (MR)CNFs, and find that similar conclusions can be drawn. Figure 4 plots the histogram of log likelihood per dimension (-BPD) of OoD images (SVHN, TinyImageNet) under MRCNF models trained on CIFARTO. can be observed that the likelihood of the OoD SVHN higher than CIFARIO for MRCNF, similar to the findings for Glow, PixcICNN, VAE in carlier works (Nalisnick et al.
2019a
Choi et al. 2018 Serrà ct al. 2020 Nalisnick et al.
2019b Kirichenko et al. 2020
One possible explanation put forward by Nalisnick et al. (2019b) is that "typical" images are less "likely" than constant images, which is a consequence of the distribution a Gaussian in high dimensions. Indeed, as our FFiuuee 4hsows, constant images have the highest likelihood under MRCNFs, while randomly generated (uniformly distributed) pixels have the least likelihood (not shown in figure due to space constraints).
Choi al. (2018); Nalisnick et al. (2019h) suggest using "typicality"
better measure of OoD. However, Serrà et al. 2020) observe that the complexity of an image plays a significant role in the training of likelihood- based generative models. They propose new metric S an out-of-distribution detector:
S(x) = bpd x) L(x)
(16)
where [xx) is the complexity of an image x measured as the length of the best compressed version of x (we use FLIF (Sneyers & Wuille 2016) following Serro al. (2020)) normalized by the number of dimensions.
We perform a similar analysis as Serrà et al. (2020) to test how S compares with -bpd for OoD detection. For different MRCNF models trained on CIFAR1O, we compute the area under the receiver operating characteristic curve (auROC) using -bpd and S as standard evaluation for the OoD detection task (Hendrycks et al. 2019 Serrà et al.! 2020).
Table 4: auROC for OoD detection using -bpd and S (Serr al. 2020 for models trained on
CIFARIO.
CIFAR10
SVHI
TIN
(trained)
-bpd
S
-bpd
S
Glow
0.08
0.95
0.66
0.72
1-res CNF
0.07 0.16
0.48
0.60
Table 4 shows that S does perform better than -bpd in the case
2-res MRCNF
0.06
0.25
0.46
0.66
of (MR),NFs, similar to the findings in SSerà et al. (200 for
3-res MRCNF
0.05
0.25
0.46
0.66
16

---

Under review conference paper at ICLR 2022
Glow and + seems that SVHN easier to detect as OoD for Glow than MRCNFs. However, OoD detection performance is about the same for TinylmageNet. We also observe that MRCNFs are better at OoD than CNFs.
Other OoD methods (Hendrycks & Gimpel 2017 Liang et al. 2018 Lee et al. 2018 Sabeti & Høst-Madsen
2019
Høst-Madsen et al. 2019 Hendrycks al. 2019) are not suitable in our case, as identified Serra
et al.
2020).
A.1
SHUFFLED IN-DISTRIBUTION IMAGES
Kirichenko et al. 2020) conclude that normalizing flows do not represent images based on their semantic contents, but rather directly encode their visual appearance. We verify this for continuous normalizing flows by estimating the density of in-distribution test images, but with patches of pixels randomly shuffled. Figure 5 (a) shows example of images of shuffled patches of varying size. Figure (b) shows the graph of the their log-likelihoods.
That shuffling pixel patches would render the image semantically meaningless is reflected in the Fréchet Inception Distance (FID) between CIFAR 10- Train and these sets of shuffled images - 1x1: 340.42, 2x2: 299.99, 4x4: 235.22, 8x8: 101.36, 16x16: 33.06, 32x32 (i.e. CIFARIO-Tess images with large pixel patches shuffled are quite close in likelihood to the unshuffled images, suggesting that since their visual content has not changed much they are almost as likely as unshuffled images under MRCNFs.
3.15. However, we see that
Nolse Shuffled 1x1
Shuffled 4x4
25C
Shuffled 8x8
Normalized aed Empin
Shuffled
Constant
CIFAR10 Test
45C
Lag likelihood per dimension
(a)
(b)
Figure 5: (a) Example of shuffling different-sized patches of 32x32 image: (left to right, to bottom) 1x1, 2x2, 4x4, 8x8, 16x16, 32x32 (unshuffled) (b) Bits-per-dim Epoch each resolution for different MRCNF models trained CIFAR10.
FULL TABLE
17

---

Under review conference paper at ICLR 2022
Table 5: Unconditional image generation metrics (lower is better in all cases): number of parameters in the model, bits-per- per-dimension, time (in hours). Most previous models use multiple GPUs for training, all our models were trained on only one NVIDIA V100 GPU. AAs reported in Ghosh et al. (2020). "FFJORD RNODE Finlay al. (2220) used 4 GPUs train ImageNet64. 'x': Fails to train.
CIFARIO BPD PARAM TIME
IMAGENET32
IMAGENET64 TIME BPD PARAM TIME
BPD PARAM
Non Flow-based Prior Work
Pi (Oord al (016))
3.00
3,86
3.63
Gated PixeICNN (Yan Hher Cord et
2016
3,03
3.83
60
3.57
60
Paralle! Multiscale- RReed 2017
3.95
3.70
Image Transformer Parmarata 20188
2.90
3.77
PixelSNAII. (Chen or
2,85
3.80
SPN (Menick-& Katchiut 2019)
3.85 150.0M
3.53
150.0M
Sparse (Chrlder 2112999
2.80
59.0M
3.44 152.0M 7days
Axial Transformer (Hordea 201
3.76
3.44
PixelFlow++ (Nielsen Winheer 2020
2.92
NVAE (Vahdat Kautz 2020)
2,91
55
Dist-Aug Sparse Transformer Junn al.
2020
2.56
152.0M
3.42
152.0M
Flow-based Prior Work
IAF (Kingma et al. (016))
3.11
RealN VP Dinh et 12017
3.49
4.28
46.0M
3.98
96.0M
Glow
Kinema Dhar
2018
3.35
44.0M
4.09
66.1M
3.81
111.IM
N-ResNets BB
Emerging Hongehs monog
3.34
44.7M
67.1M 4.18
4.09
67.1M
IDF
(Hoogchxxin
3.34
3,90
S-CONF Kanntti
3.34
MintNet (Sog
3.32
17.9M 5ddays
17.4M 4.01
4.06
Residual Flow
2019
3.28
3.76
MaCow (Ma et al.
2019)
3.16
43.5M
3.69
122.5M
Neural Spiine Fluss
Pnrrka et al.
2019
3.38
11.8M
3.82
15.6M
Flow++ Ho
2019.
3.08
31.4M
3.86
169.0M
73.5M
ANF Huang ct
2020
3.05
3.92
3.66
MEF
2020
3.32
37.7M
4.05
37.7M
3.73
46.6M
VFlow
Chen
(2920
2.98
3.83
Woodbury NF
aa & Huang
2020
3.47
4.20
3.87
NanoFlow (Lee al.
2020)
3.25
ConvExp
(Hongchoom
2020
3.218
Wavelet Flow
et al.
2020
64.0M
3.78
96.0M 822
TayNODE (Kelly et al.
2020)
1.039
I-resolution Continuous Normalizing Flow
FFJORD Grathwohl et al. 2019)
3.40
0.9M
2ddays
3.96
2.0M
Sdays
X
RNODE Finlay al. 2020)
3,38
1.4M
31.84
12.36
2.0M
300. "3.83
2.0M
"256.4
$3.49
$1.6M
$ 40.39
FFJORD + STEER
GGosh et al.
2020
3.40
1.4M
86.34
3.84
2.0M
Sdays
RNODE STEER
Ghosh al.
2020)
3.397
1.4M
22.24
2.35
2.0M
24.90
$3.49
$1.6M
$3.077
(OURS) Multi-Resooution Continuous Normalizing Flow (MRCNF))
2-resolution MRCNF
3.65
1.3M
19.79
3.77
1.3M
18.18
3.44
2.0M
42.30
2-resolution MRCNF
3.54
3.3M
36.47
3.78
6.7M
17.98
6.7M
3-resolution MRCNF
3.79
1.5M
17.44
3.97
1.5M
13.78
3.55
2.0M
35.39
3-resolution MRCNF
3.60
5.1M
38.27
3.93
10.2M
41.20
7.6M
X
18

---

Under review conference paper at ICLR 2022
C
QUALITATIVE SAMPLES
7
1
3
H
d
1
6
17
1
E
3
H
d
4
9
4
V
4
5
3
5
4
2
V
5
2
2
5
3
7
4
8
6
3
9 8 5
4
3
5
8
6
3
9 8 5
4
4
2
R
9
9
7
r
_ 3 L
J
4
2
R
9
9
7
r
_ 3 5
7 7 3
5
7
a
6
1 7
c
7 3
5
7
a
6
v 7
c
9
>
7
7
5
7
o
7
?
0
4
7
7
2
U
a
0
4
7
7
&
2
U
&
d 8 4
J
J
4
9
6
8
5
J
4
9
6
9
0
7
1
a
0
&
/
8
4
7
1
J
a
0
y
4
/
0
8
9
8
R
-
4
3
1
9
8
8
8
5
J
4
3
2
9
(a) Generated samples 16x16
(b) Corresponding generated samples 32x32
Figure 6: Generated samples from MNIST.
(a) Generated samples 8x8
(b) Generated samples 16x16
(c) Generated samples at 32x32
Figure 7: Generated samples from CIFAR10.
D SIMPLE EXAMPLE OF DENSITY ESTIMATION
For example, if we Fuler method
ODE solver, for density estimation Equation 2 reduces to:
v(t) v(to) (ti (v))
(17)
19

---

Under review conference paper at ICLR 2022
where fa is neural network, to represents the "time" which the state is image x, and is when the state noise 2. We start at scale S with an image sample xs, and assume ty and & are and respectively:
2s xs + {s( to xs-1) 25 = Xx + fs (xS-1+ to xx-2)
(18)
21 = x1 f1 xo)
Zo Xo fo(xo, to)
E
SIMPLE EXAMPLE OF GENERATION
For example, if we use Euler method as our ODE solver, for generation Equation 2 reduces to:
v(4o) = v(ti) + (to (v)))av((t)), c)
(19)
Le. the state is integrated backwards from t (i.e. z,) to to (i.c. x2). We start at scale with a noise sample Zo. and assume to and t are and respectively:
X Zo - fo t1)
21 f21 t xo)
(20)
X5-1 25-1 fs (2s-1, l1 x8-2)
X5 = Zg fs(2s, t xs-1)
F MODELS
We used the same neural network architecture as RNODE (Finlay et al. 2020). The CNF at each resolution consists of stack of bl blocks of 4-layer deep convolutional network comprised of 3x3 kernels and softplus activation functions, with 64 hidden dimensions, and time t concatenated to the spatial input. In addition, except at the coarsest resolution, the immediate coarser image is also concatenated with the state. The integration time of each piece is [0, I]. The number of blocks bl and the corresponding total number of parameters are given in Table
Table 6: Number of parameters for different models with different total number number of channels (ch) and number of blocks (bl) per resolution.
of resolutions (res), and the
MRCNF
resolutions ch bl
Param
64 2
0.16M
64
0.32M
64
14
1.10M
64
8
1.33M
2
64
20
3.34M
64
40
6.68M
1.53M
3
64
8
2.04M
64
20
5.10M
20

---

Under review conference paper at ICLR 2022
G GRADIENT NORM
In order to avoid exploding gradients, We clipped the norm of the gradients (Pascanu et al. 2013) by maximum value of 100.0. In case of using adversarial loss, we first clip the gradients provided by the adversarial loss by 50.0, up the gradients provided by the likeliikd loss, and then clip the summed gradients by 100.0.
H 8-BIT TO UNIFORM
The charaelof-variables formula gives the change probability due to the transformation of to v:
dv
log log p(v) + log
det
du
Specifically, the change of variables from 8-bit image image with pixel values in range [0, 1] is:
(p)
(p)
s
256
db
log p(as) pbbs) log det
da
log ((as( lobs( log 256
log)(as) = log p(bs) D3 log 256
bpd(as) = log poas(
Dg log 2
(log p(bs)
D3 log 256)
Dg log 2
log (bs))
log 256
+
Dş log 2
log
bxd(x) +
where (x) is given from Equation 13
I FID v/s TEMPERATURE
Table lists the FID values of generated images from MRCNF models trained on CIFAR1O, with different temperature settings on the Gaussian.
Table 7: FID v/s temperature for MRCNF models trained CIFARIO.
Temperature
1.0
0.9
0.8
0.7
0.6
0.5
I-resolution CNF
138.82
147.62
175.93
284.75
405.34
466.16
2-resolution MRCNF
89.55
106.21
171.53
261.64
370.38
435.17
3-resolution MRCNF
88.51
104.39
152.82
232.53
301.89
329.12
21


Reviews:
[{"review_id": "t9TLN3PPkpO", "reviewer": "Reviewer_hXQn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nThe problem considered in this paper is challenging and the approach to solve it (upscaling via conditional flows) is adequate to the problem. The improvements compared to WaveletFlow seems to work in practice what can be observed while analysing the quantitative results (BPD). It is also very beneficial for that work that the authors analyse also training time and complexity of the model during the experimental stage. The OOD analysis (unfortunately provided in Supplementary) is also interesting in terms of possible other applications of the model. It is also important to mention, that the code is provided in the suplement, so reproducing the results should be easy. \n\nWeaknesses \n\nMy major concern is about the contribution of the paper. The first two points in introduction that point out the contribution are non-informative in my opinion. The first refers to the transformation that does not add the cost to the training objective. Why is it so important for the model to not modify the likelihood cost with that transformation? If the (log) cost is negative, it can reduce the likelihood value while testing. \n\nThe second contribution is simply the name of the model and should be removed. Even for the third contribution I have the problem of understanding what components of the model are responsible for time and model capacity reduction. In my opinion the contribution of the paper is based on the different base flow model (CNF) and using unimodular transformation in eq. (7).  This contribution is rather incremental with respect to the WaveletFlow. Using CNF seems to be a good direction but is not motivated taking into account that any conditional flow can be used instead. The new transformation function does not provide a significant improvement compared to WaveletFlow and is motivated by reduction of the cost of transformation not the quality of model itself. \n\nThe second concern is about the provided results. The Table 1 is incomplete, some results are missing for reference approaches, it is sparse, and therefore, difficult to follow and analyse. The authors do not provide the qualitative results for high-resolution images as in WaveletFlow - such empirical would be beneficial to show the gain of using the approach provided in the paper. The selection of BPD criterion for quantitative evaluation is ok, but the goal of multi-resolution generative model is to generate good-quality images for larger dimensions. Using some standard measures like FID or related and comparison to GAN models would be beneficial for this case.    \n\nThe third concern is about the quality of the paper - the provided manuscript is difficult to follow, the contribution is not expressed well and the description of the method should be more clear. I do not understand, why it is important to optimize BPD  directly instead of likelihood? For me it is just the scaling issue that does not affect the training procedure. Some interesting parts like OOD analysis are moved to suplement. ", "summary_of_the_paper": "The paper describes the novel approach to generate high-resolution images in progressive manner. The approach is based on set of conditional flows that take the image generated from previous stage as conditioning factor and generate the image with the higher resolution. The quality of the method is compared to the reference approaches, mainly focusing on WaveletFlow as the most similar method.", "summary_of_the_review": "Unfortunately, the paper is rather incremental with respect to WaveletFlow, the contribution is not expressed well and qualitative analysis is missing. I would like to encourage authors for more extensive empirical evaluation and highlighting the novelty of the approach. Therefore, my current recommendation is to reject this work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qnG0_vQfPjX", "reviewer": "Reviewer_cSSK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, my general take is that the paper is relatively weak on both conceptual novelty as well as application performance. My main findings are below.\n\nCons:\n\nConceptual Novelty:\nThough the paper tried to differentiate from prior work (WaveletFlow) in various aspects (using CNF rather than RealNVP, utilizing a volume and range preserving formulation), the main idea of multi-resolution conditional modeling of images at the next resolution based on the previous, is very much the same. In fact, I don't think the authors did a good job at justifying the few differentiating design factors from WaveletFlow. For instance, the authors claimed that the volume-preserving and range-preserving transformations are a key novelty but the effects of this transformation (as opposed to the wavelet transforms) are not studies in an experiment.\n\nExperiment Results:\nFrom the experiment results (Table 1), it's not clear that (1) the multiresolution formulation is more efficient, or (2) the generative modeling quality is improved, both of which I think are central to the claims of the paper. \n- First, the BPD metric seems to get worse with more resolutions, which seems contrary to the central story?\n- Second, even though the authors claimed in the paper that the model is \"significantly faster with on-par performance\", the performance seems quite a bit worse than even very early methods like RealNVP. While the model size seems smaller, that seems more likely be due to the use of FFJORD/CNF, rather than the multi-resolution formulation, which is an existing architecture the authors adopted, rather than a contribution of this paper. \n- Third, while model size seems to be a main selling point of the story, being \"smaller/cheaper with lower performance\" does not feel very compelling. Very likely than not, when the \"large\" models are reduced using simple methods like reducing the number of feature channels / number of stacked flows, they can slim down to the same size with comparable or even performance.\n\nPros:\n\nI do like that the authors are quite describing the mathematical details of the proposed multi-resolution transformations (Sec. 3.1, 3.2) which are easy to follow and enjoyable to read.", "summary_of_the_paper": "The authors proposed a multi-resolution version continuous normalizing flows with the claim of faster training time resulting in comparable performance. The authors note a few key differences with previous work (WaveletFlow) - using CNF instead of realnvp / glow based architectures, modeling noise at multi-resolution, and utilizing a volume and range preserving formulation, ", "summary_of_the_review": "Overall, I enjoyed reading the paper and I really like the mutliresolution idea that leverages conditional probability modeling, but I think the claimed main conceptual novelties are weak and not well supported by experimental results. Therefore I cannot recommend acceptance at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bfRJNJIszai", "reviewer": "Reviewer_efZN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-The authors provide a faster multi-resolution strategy for normalizing flows.\n-The empirical results are better than the benchmarked methods, e.g. WaveletFlow [1], Glow [2], FFJORD [3].\n-The paper is clearly written and easy to read.\n-Authors provide detailed descriptions and the source code thus the work should be easier to reproduce.\n\nWeaknesses:\n-Comparisons to some of the benchmarked methods are not fair/relevant (details below).\n-Being faster to train, one would expect the method to be scaled up to larger images but no such experiments were reported. Thus, it is hard to evaluate how significant the training speedup is since the ODE based solutions are bottlenecked by training time [3], i.e. if the method does not address/alleviate the scalability issues, training faster on 64x64 images is less impactful.\n-Clarifications are needed for various parts of the submission (details below).\n\nRequired clarifications:\n-Section 3.2, the sentence: “This can be scaled up to larger spatial regions by performing the same calculation for each 2×2 patch.” I do not immediately see how. If “each” means a sliding window with stride 2, authors should explicitly write this. Otherwise some clarification is needed.\n-Section 3.2: Why does the logdet term being zero makes the training faster? More generally, the authors propose a new M matrix but other than looking nice, the value of the novelty is not clear (i.e., why should an additive log term be problematic for the training speed?). Maybe the authors can stress the training speedup a bit more and provide some intuition about the cause of this speedup.\n-Section 3.3, sentence with “...are generated conditioned on the entire coarser image...”: I do not see how. M is local, sliding it in stages (I assume) will still yield a (larger) local mapping. Could the authors expand their explanation? This is likely going to result in inconsistent textures, similar to WaveletFlow [1].\n-Section 4, “Comparison to WaveletFlow” part: It is not feasible to ask the authors to apply their innovations on top of a Glow [2] based architecture. However, as it stands, their comparison to WaveletFlow [1] does not seem fair. The ablations section does not seem relevant either (more on that below). The authors might want to add a disclaimer here that core architectures, training strategy, hardware/software are extremely different. If possible, I suggest the authors limit the qualitative comparisons to FFJORD [3] and leave the WaveletFlow comparisons to the empirical results section only. This would also inform the reader better since I believe this paper is a direct extension of FFJORD [3] with a generalized wavelet multi-resolution strategy. The other perspective of “applying CNF etc. to WaveletFlow [1]” would be misleading.\n-Table 3 and more generally Section 5.1: As I understand it, the comparisons are not between a WaveletFlow [1] modified to incorporate CNF vs. MRCNF but rather between MRCNF with Haar vs. MRCNF with the authors’ M matrix. This needs to be extremely clear.\nFurthermore, bullet point (1) is confusing to me. Do the authors replace the flow steps in WaveletFlow with a CNF? If so, what were the exact architectural changes?\nFinally, the last paragraph may be misleading (or at least unclear). The authors seem not to have modified (or even reproduced) WaveletFlow themselves, seeing as they report the exact same empirical results from the paper [1] and no mention of it in the provided code. Thus, the claim of  “MRCNF being more than WaveletFlow with CNF” is confusing - the experiments are just not there for proving/disproving this claim. I may be missing something, I ask the authors to please clarify.\n-Supplementary: Authors mention that gradient clipping for the adversarial loss being different. I could not find another mention of the adversarial loss in the paper.\n\nMinor points:\n-Introduction: I would urge the authors to abstain from blanket statements like “... notoriously difficult to train…”. With the newer normalization, loss function and architectural innovations GANs, at least in my experience, are easier to train.\n-Section 3.2, first paragraph: I understand that this is probably due to the page limit but if the authors can find the space, each bullet point should start as a new line.\n-Figure 2: I believe this figure can be cut without impacting the readability of the paper.\n\n\n[1] Jason Yu, Konstantinos Derpanis, and Marcus Brubaker. Wavelet flow: Fast training of high resolution normalizing flows. In Advances in Neural Information Processing Systems, 2020\n\n[2] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in neural information processing systems, pp. 10215–10224, 2018\n\n[3] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. International Conference on Learning Representations, 2019\n", "summary_of_the_paper": "In this work, the authors propose a multi-resolution strategy for continuous normalizing flows. The proposed approach consists of a general wavelet based decomposition/downsampling where the transformation obeys some useful mathematical properties such as volume and range preservation. Empirical results indicate likelihood estimation performance better or on par with benchmarked methods. The proposed model also trains significantly faster with fewer parameters.", "summary_of_the_review": "As the authors also point out, using multi-resolution strategies is not novel in itself. Additionally, the reported experiments indicate that the proposed method is not mature enough to be applied to larger images, limiting its usefulness. However, despite the incremental novelty of the approach and limited empirical results I believe the authors propose a method with potential to be widely adopted in the normalizing flows models. I would be willing to update my recommendation based on the authors’ response to my clarification requests listed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ViCLhHxnvzO", "reviewer": "Reviewer_K92k", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Comments/ questions: \n- The paper is written well and easy to understand.\n- The contributions do not seem to be significant enough yet from either a theoretical or practical perspective. I would request the authors to investigate further and work with larger image sizes (256 x 256 Glow paper). Perhaps more insight into why flows are worse on OOD from a multiscale perspective would be a good addition. Can we check if the problem is worse or better at coarser scales? Are constant images consistently assigned higher likelihoods at all image scales? If so, can't we regularize for this? When shuffling images, again use the multi-scale ideas and check which scales are affected most and how that is related to the tile size used in shuffling. I think these would be interesting and insightful additions to the paper.\n- The main advantage of the method seems to be training times and number of parameters in a network which is good but there are other papers out there now that claim similar advantages with injective flows (Trumpets, UAI '21 Kothari et al, M-flows, NeurIPS' 20 Cranmer et al).  Rather than being yet another paper in the same direction, I think the author's would benefit greatly from investigating their key ideas a bit more. For example, a more in-depth analysis of noise injection at various scales and how that affects image generation would be good to have. A simple analysis could show how posterior samples of $x_s$ vary given, say $y_s, y_{s-1}$  and $x_{s}$. \n\n", "summary_of_the_paper": "The paper proposes a multi-resolution variant of continuous normalizing flows for images. They show empirical results for image sizes upto 64 x 64 where they seem to be better than regular continuous normalizing flows. The key proposed benefit seems to be in number of parameters and training times.", "summary_of_the_review": "I think that the current draft of this work makes incremental contributions to the area of normalizing flows for image synthesis. While I completely agree with the core motivations of bringing multi-resolution structures and scale-separated noise injection to CNFs, I believe the authors can make a much more significant contribution by analyzing how their core motivations play out in their proposed method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2pSMVzWT1Wq", "reviewer": "Reviewer_McBP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation and execution of the paper are good and I appreciate the many experiments and ablations the authors have run. However, I have a few concerns about the discussion of some of the results in the paper as well as questions around some aspects of the model. The main strengths and weaknesses (in my eyes) are described below.\n\n**Strengths**:\n- The paper is well motivated and the ideas are sound - it makes sense to take advantage of the multi resolution structure of images.\n- The paper is well written and the figures are nice.\n- The authors make extensive ablation studies and compare their model to a large number of baselines (including reimplementations of baselines which is great). I appreciate the effort the authors have put in, in order to have thorough experiments on a variety of image datasets.\n- The results on Imagenet64 are impressive, getting low bpd with a low number of parameters and reduced training time.\n- The progressive training results on Imagenet128 are very nice. The authors are able to achieve fairly good results by only training a single refining step on top of an Imagenet64 model, which is impressive.\n- While the authors do not perform any experiments on this aspect of the model, the fact that the model can be trained in parallel is interesting.\n- Discussion of related work is thorough and the model is well situated in the literature.\n- The OoD experiments in the appendix are quite interesting.\n\n**Weaknesses**:\n- Some of the experimental results are slightly confusing. From both Table 1 and Table 3, it appears that adding more levels of resolutions (and hence more parameters) consistently _decreases_ performance, which seems very counterintuitive with the main point of the paper (that modeling multiple resolutions is important). This could suggest that it’s not the multi resolution aspect of the model that helps, but something else. I was disappointed that the authors do not seem to discuss this at all in the paper (in fact I had not realised this until I looked at the numbers in detail). This should be explicitly mentioned and discussed in the paper.\n- Experimental results on low resolution datasets are not very good. For example, on CIFAR10, for all tested resolutions (2 and 3) the model does worse than the RNODE and vanilla FFJORD baselines. Again this does not seem to be discussed or acknowledged anywhere in the paper. It is not a problem as such that this doesn’t work well for low res datasets given that it works well for high res datasets, but it would be good to discuss this and be more explicit about limitations in general.\n\n**Questions**:\n- I am confused about how the conditioning on the coarse image is done exactly. This feels like quite a significant part of the model but as far as I can tell is only described by a single sentence on page 6. How is the input image of the CNF concatenated with the coarser image? The dimension of the input image is equal to the dimension of the coarser image + the dimension of the noise variables if I understood correctly, so how can the input image and the coarse image be concatenated (they have different shapes)? Shouldn’t the noise and the coarse image be concatenated instead? It would be good to be more explicit about how this works.\n- In section 3.2, you choose $\\mathbf{y}_1$ such that it “contains information not present in $\\mathbf{x}_2$ such that $\\mathbf{x}_1$ is obtained when $\\mathbf{y}_1$ and $\\mathbf{x}_2$ are combined”. A very simple thing to do here that would satisfy this criterion is to set $\\mathbf{y}_1 = (x_1, x_2, x_3)$. Have you run this ablation?\n\n**Typos**:\n- Page 6: “we use to two regularization terms” -> “we use two regularization terms”\n", "summary_of_the_paper": "This paper proposes a new architecture for continuous normalizing flows that explicitly models images at multiple resolutions. The authors achieve this by learning an unconditional distribution of coarse images and then learning conditional distributions at progressively finer resolutions, each with a continuous normalizing flow. In contrast to the typical “squeeze” layers used in normalizing flows, where the noise is only passed at the first layer, the proposed model injects noise at several resolutions (which has already proved successful with other generative models). The authors introduce a new (invertible) transformation to go from a coarse image to a finer image which preserves the range of the image while having unit determinant (implying the transformation leaves the log likelihood unchanged). The whole model is then trained via maximum likelihood.\n\nThe authors evaluate their method on standard image datasets and perform several ablations to test the contributions of their model.\n\nThe main contributions of the paper in my eyes are then:\n- The introduction of a new architecture and layer for learning multi resolution normalizing flows\n- Demonstration that the proposed model can be trained with limited computational resources even on fairly large scale datasets (such as Imagenet128)\n- Ablation studies evaluating the improvements from each aspect of the proposed model\n", "summary_of_the_review": "This paper introduces a new architecture for multi resolution continuous normalizing flows. The paper is well written and the model is sound and the authors achieve impressive results on Imagenet at high resolutions. However, as described in the weakness sections I believe there are still some aspects of the model and experiment discussion that need to be clarified. I therefore believe this paper is currently marginally above the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ### Description

The paper enhances flow-based generative models by putting them into a coarse-to fine multi-resolution framework. The key technical challenge as I understand is designing up-scaling conditional flow modules. Since the operation needs to be invertible, the paper carefully designs what degrees of freedom need to be injected in addition to the low resolution image to compose a higher resolution one.

### Decision
The paper received 5 expert and rather detailed reviews. I have read and understood the paper and all reviews. Reviewers remark that the paper is well written, addresses a challenging problem. However reviews were in a consensus on that the contribution of the paper is marginal. The average score was 4.4. The authors did not respond to reviewers  and did not update the paper. There was no post-rebuttal discussion and or additional feedback from reviewers. Therefore, must reject.

### Comments
I have only minor comments on the writing and organization of the paper.
There are many self-repetitions in the text, restating what was already said above in same or very similar sentences. Some questions studied in appendices are not presented in the main papar.
Decision: Reject