Likelihood Contribution based Multi-scale Architecture for Generative Flows
Authors: Hari Prasanna Das
Pieter Abbeel
Costas J. Spanos
Venue:  (2020)
Keywords: Generative Flow
Normalizing Flow
Multi-scale Architecture
RealNVP
Dimension Factorization

Abstract: Deep generative modeling using flows has gained popularity owing to the tractable exact log-likelihood estimation with efficient training and synthesis process. However, flow models suffer from the challenge of having high dimensional latent space, same in dimension as the input space. An effective solution to the above challenge as proposed by Dinh et al. (2016) is a multi-scale architecture, which is based on iterative early factorization of a part of the total dimensions at regular intervals. Prior works on generative flows involving a multi-scale architecture perform the dimension factorization based on a static masking. We propose a novel multi-scale architecture that performs data dependent factorization to decide which dimensions should pass through more flow layers. To facilitate the same, we introduce a heuristic based on the contribution of each dimension to the total log-likelihood which encodes the importance of the dimensions. Our proposed heuristic is readily obtained as part of the flow training process, enabling versatile implementation of our likelihood contribution based multi-scale architecture for generic flow models. We present such an implementation for the original flow introduced in Dinh et al. (2016), and demonstrate improvements in log-likelihood score and sampling quality on standard image benchmarks. We also conduct ablation studies to compare proposed method with other options for dimension factorization.

Under review as a conference paper at ICLR 2020
LIKELIHOOD CONTRIBUTION BASED MULTI-SCALE ARCHITECTURE FOR GENERATIVE FLOWS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep generative modeling using flows has gained popularity owing to the tractable exact log-likelihood estimation with efficient training and synthesis process. How- ever, flow models suffer from the challenge of having high dimensional latent space, same in dimension as the input space. An effective solution to the above challenge as proposed by Dinh et al. (2016) is a multi-scale architecture, which is based on iterative early factorization of part of the total dimensions at regular intervals. Prior works on generative flows involving a multi-scale architecture perform the dimension factorization based novel multi-scale architecture that performs data dependent factorization to decide which dimensions should pass through more flow layers. To facilitate the same, we introduce a heuristic based on the contribution each dimension to the total log-likelihood which encodes the importance of the dimensions. Our proposed heuristic is readily obtained as part of the flow training process, enabling versa- tile implementation of our likelihood contribution based multi-scale architecture for generic flow models. We present such implementations for several state-of- the-art flow models and demonstrate improvements in log-likelihood score and sampling quality on standard image benchmarks. We also conduct ablation studies to compare proposed method with other options for dimension factorization.
static masking. We propose a
INTRODUCTION
Deep Generative Modeling aims to learn the embedded distributions and representations in input (es- pecially unlabelled) data, requiring no/minimal human labelling effort. Learning without knowledge of labels (unsupervised learning) is of increasing importance because of the abundance of unlabelled
data and the rich inherent patterns they posses. The representations learnt can then be utilized in
number of downstream tasks such as semi-supervised learning (Kingma al., [0144 Odena (0166). synthetic data augmentation and adversarial training (Cisse et al.) 2017). text analysis and model based control etc. The repository of deep generative modeling majorly includes Likelihood based
models such as autoregressive models (Oord et al. 2016b] Graves 2013). latent variable models (Kingma & Welling, 2013), flow based models (Dinh et al.] 2014|2206] Kingma & Dhariwal (018))
and implicit models such generative adversarie networks (GANs) (Goodfflloow et al.] 2014). Autoregressive models (Salimans et al. [017]] Oord et al., (0166 a Chen et 2017) achieve excep- tional log-likelihoce score on many standard datasets, indicative of their power model the inherent distribution. But, they suffer from slow sampling process, making them unacceptable to adopt in real world applications. Latent variable models such as variational autoencoders (Kingma Welling 2013) tend to better capture the global feature representation in data, but do not offer an exact density estimate. Implicit generative models such as GANs which optimize generator and a discriminator in min- max fashion have recently become popular for their ability to synthesize realistic data (Karras) jtt al. 2018 Engel et al. 2019). But, GANs do not offer a latent space suitable for further downstream tasks, nor do they perform density estimation.
Flow based generative models (Dinh et al., 2016 Kingma & Dhariwal 2018 perform exact density estimation with fast inference and sampling, due to their parallelizability They also provide an information rich latent space suitable for many applications. However, the dimension of latent space for flow based generative models is same the high-dimensional input space, by virtue of bijectivity nature of flows. This poses a bottleneck for flow models to scale with increasing input dimensions due to computational complexity. An effective solution to the above challenge is multi-scale architecture,

---

Under review as a conference paper at ICLR 2020
introduced by Dinh al. (2016), which performs iterative early gaussianization of part of the total dimensions at regular intervals of flow layers. This not only makes the model computational and memory efficient but also aids in distributing the loss function throughout the network for better training. Many prior works including Kingma & Dhariwal (2018); Atanoy et al. (2019) Durkan et al.] (2019); Kumar al. (2019) implement multi-scale architecture in their flow models, but use static masking methods for factorization of dimensions. We propose multi-scale architecture which performs data dependent factorization decide which dimensions should pass through more flow layers. For the decision making, we introduce a heuristic based on the amount of total oog-likeihood contributed by each dimension, which in turn signifies their individual importance. We lay the ground rules for quantitative estimation and qualitative sampling to be satisfied by an ideal factorization method for a multi-scale architecture. Since the proposed architecture, the heuristic is obtained as part of the flow training process, it can be universally applied to generic flow models. We present such implementations for flow models based on affine/additive coupling and ordinary differential equation (ODE) and achieve quantitative and qualitative improvements. We also perform ablation studies to confirm the novelty of our method. Summing up, the contributions of our research are,
Contributions:
1. A log-determinant based heuristic which entails the contribution by each dimensions towards the total log-likelihood in multi-scale architecture.
2. A multi-scale architecture based on the above heuristic performing data-dependent splitting of dimensions, implemented for several classes of flow models.
3. Quantitative and qualitative analysis of above implementations and an ablation study
To the best of our knowledge, we are the first to propose a data-dependent splitttnnn of dimensions in a multi-scale architecture.
2
BACKGROUND
In this section, we illustrate the functioning of flow based generative models and the multiscale architecture as introduced by Dinh et al. (2016).
2.1 FLOW-BASED GENERATIVE MODELS
Let x be high-dimensional random vector with unknown true distribution p(x). The following formulation is directly applicable to continous data, and with some pre-processing steps such as dequantization (Uria et al. 2013) Salimans et al. [017 Ho et al. (2019) to discrete data. Let z be the latent variable with a known standard distribution p(z), such as a standard multivariate gaussian. Using an i.i.d. dataset DD the target is to model po (x) with parameters O. A flow, fo is defined to be an invertible transformation that maps observed data x to the latent variable z. A flow is invertible, so the inverse function T maps z to i.e.
z fo (x) T (x) and x = T(z)) =r (x) The log-likelihood can be expressed as,
log (x)(x)) = log(p(z)) + +lg (lees (x)
Ox (
where (x))
0x is the Jacobian of fg at x.
The invertibile nature of flow allows it to be capable of being composed of other flows of compatible dimensions. In practice, flows are constructed by composing series of component flows. Let the flow fo be composed of K component flows, i.e. fo = fax fog - G ..... fo, and the intermediate x. Then the log-likelihood of the composed flow is,
variables be denoted by yK, YK " Yo
log (x)(x)) = log(p(z)) + log (laet /eet
(f(fex fox_, GxT
fg, (x))
(3)
= log(p(z))
log (et (ant(Oy/yT))) /0yT
(4)
Log. latent density
Log del
2

---

Under review as a conference paper at ICLR 2020
which follows from the fact that det(A- B) det(A) det(B). In work, we refer the first term in and the second term as log rog-determiiant (log-der). The reverse path, from to x can be written composition of inverse flows, =f (z) = fo o of o...o o fox (z). Confirming with the properties of flow as mentioned above, different types of flows can be constructed (Kingma & Dhariwal 2018 Dinh et al. 2016 2014 Behrmann et al. 2018).
Equation 4 log-latent-denae
2.2 MULTI-SCALE ARCHITECTURE
Multi-scale architecture is design choice for latent space dimensionality reduction of flow models, in which part of the dimensions are factored out/early gaussianized regular intervals, and the other part is exposed to more flow layers. The process is called dimension factorization. In the problem the factoring operation can be mathematically expressed as,
setting as introduced in Section 2.1
Yo = x
(5)
2 V1+1 (y)), IE (0,1, K K-2) 2]
ZK føx (VK-1)
z (Z1, .Zk)
(8)
The factoring of dimensions at early layers has the benefit of distributing the loss function throughout the network (Dinh et al., (016) and optimizing the amount of computation and memory used by the model. We consider the multi-scale architecture for flow based generative models as introduced by Dinh et al. (2016) (and later used by state-of-the-ar flow models such as Glow(Kingn) & Dhariwal 2018)) as the base of our research work.
3 LIKELIHOOD CONTRIBUTION BASED MULTISCALE ARCHITECTURE
In a multi-scale architecture, it is apparent that the network will better learn the distribution of variables getting exposed to more layers of flow as compared to the ones which get factored at finer scale (carlier layer). The method of dimension splitting proposed by prior works such as Dinn et al. 2016 Kingma & Dhariwal 2018 [Behrmann et al. 2018) are static in nature and do not distinguish between importance of different dimensions. In this section, we introduce heuristic to estimate the contribution of each dimension towards the total log-likelihood, and introduce a method which use the heuristic decide the dimensions be factored an earlier layer, eventually achieving preferrential splitting in multiscale architecture. Our approach builds an efficient multiscale architecture which factors the dimensions at each flow layer way such that the local variance in the input space is well captured as the flow progresses and the log-likei is maximized. We also describe how our multi-scale architecture can be implemented over several standard flow models.
Recall from Equation that the log-likelihood is composed of two terms, the log-latent-denat densityy term and the log-det term. The log-latent-densityy term depends on the choice of latent distribution whereas the log-det term depends the modeling of the flow layers, So, careful design of flow layers can lead to maximized log-determinant, eventually maximizing the likelihood. The total log-det term is nothing but the sum of log-det terms contributed by each dimension. Let the dimension of the input space x be s X $ X c, where is the image height/width and is the number of channels for image inputs. For the following formulation, let assume no dimensions that we have access to log-det term for all dimensions at each flow layer, and the dimension at all intermediate layer remains same (i.e. X c). We apply flow (fo) with K component flows to x, z pair, so that z = fe (x) fox o fāx o... o fa, (x). The intermediate variables are denoted by YK, - .o with ук = z (since early gaussianization was done) and yo = x. The log-det term at layer 1, (1()) is given by,
gaussianized early so
[T sealer
log d0t
The log-det of the jacobian term encompasses contribution by all the dimensions. We decompose it to obtain the individual contribution by variables (dimensions) towards the total log-det (~~ total ligllioel The log-det term be viewed (with slight abuse of notations) as s
3

---

Under review as a conference paper at ICLR 2020
mint5.
miat7.4)
1
2
3
4
min().3)
min(8.4)
5
6
7
8
mox(G.4
maz(L.))
max(3.4)
Figure 1:
Likelihood contribution based squeezing operation for ReaINVP: (On left) The tensor
[L lexexe representing log-det of variables a flow layer. (On right) It is squeezed to x & x 4c with local max and min pooling operation. The black (respectively white) marked pixels represent dimensions having more (respectively less) log-det locally.
tensor corresponding each of the dimensions, summed over the flow layers till 4,
[L 5x8xc Σid (d
lxxoxe where a, E {0, ,s} and y E -.. ,c]
1xxxxe:
(10)
(a.8n) log dey((0y./8ff1))
s.t.
(11)
a,P,7
The entries in [L lexexe having higher value correspond to the variables which contribute more towards the total log-likelihood, hence more valuable for better flow formulation. So, can the likelihood contribution (in the form of log-det term) by each dimension as a heuristic for deciding which variables should be gaussianized early in multi-scale architecture. Ideally, at each flow layer, the variables with more log-det term should be exposed to more layer of flow and the ones having less log-det term should be factored out. In this manner, selectively more power can be provided to variables which capture meaningful representation (and are more valuable from log-det perspective) be expressive by being exposed multiple flow layers. This formulation leads enhanced density estimation performance. Additionally, for many datasets such as images, the spatial nature should be taken into account while deciding dimensions for early gaussianization. Summarily, at every flow layer, an ideal factorization method should,
1. (Quantitative) For efficient density estimation: Early gaussianize the variables having less log-det and expose the ones having more log-det more flow layers
2. (Qualitative) For qualitative reconstruction: Capture the local variance over the flow layers, i.e. the dimensions being exposed to more flow layers should contain representative pixel variables from throughout the whole image.
Keeping the above requirements in mind, variants of hybrid techniques for factorization can be implemented for different types of flow models which involve multi- architecture, improve their density estimation and qualitative performance. The key requirement is availability of log-det contributions per dimension, which can be fulfilled by decomposition of the log-det of the jacobian. We refer to the method as Likelihood Contribution based Multi-scale Architecture (LCMA). The steps of LCMA implementation for flow models is summarized Algorithm I dimension factorization phase in algorithm 1 we group the dimensions having more/less log-det locally and then perform splitting. This preserves the local spatial variation the image in both parts of the factorization, leveraging both enhanced density estimation well as qualitative reconstruction. Another important observation is since the factorization of dimensions does not occur during the training time, and before the actual training starts, the decision of dimensions which get factored at each flow layer is fixed, the change of variables formula can be applied. This allows the use of non-invertible operations (c.g. max and min pooling) for efficient factorization with log-det heuristic.
Note that step 2 of
Step of dimension factorization phase requires computation of individual contribution of dimensions (4)
(T. sxxxx)) towards the total likelioo,,,,,, which can vary depending the original design of flow
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1: L.CA implementation for generative flow models
Pre-Training Phase: Pre-train network with no multiscale architecture (no dimensionality reduction) to obtain the log-det term at every flow layer. Dimension Factorization: In this phase, the dimensions to be factored at each flow layer is decided based on the log-det term at that layer
11 1. The individual contribution of dimensions towards likelihood [ lexexe) is computed specifically for corresponding flow models (Refer Section 3. and Section 3.2).
2. Convert lex into a 2 x X 4c shaped tensor using local max and min-pooling (= -max-pooliie -max-pooling(- pooling(-input)) operations (Figure 1 each flow layer.
3. Among the 4e channels, one half contains the dimensions having more log-det term compared with its neighbourhood pixel (Black marked in Fig. [T. while the other half contains the dimensions having less log-det (White marked in Fig. T.
4. Split the tensor along the channel dimension to two parts.
5. Forward the corresponding dimensions contributing more towards likelihood into more flow layers and early gaussianize the ones contributing less.
6. Repeat steps 1-5 for all the layers with dimensions passed to that layer till the latent space.
Training Phase: The decision of dimensions to be factored at each layer as performed in previous step remains fixed. Finally, the flow model with proposed LCMA is trained.
models. Some flow models offer direct decomposition of jacobian into per-dimension components, whereas for others, an indirect estimation method has be adopted. We now describe such methods to obtain such individual likelihood contribution of dimensions for flow models based on affine coupling (ReaINVP (Dinh et al., 2016) and Glow (Kingma & Dhariwal (018)), and flow models involving ordinary differential equation (ODE) based density estimators (i-ResNet (Behrmann et al. 2018), all of which involve a multiscale architecture.
3.1 ESTIMATION OF DER-DIMENSION LIKELIHOOD CONTRIBUTION FOR AFFINE COUPLING BASED FLOW MODELS
RealNVP DDnn et al. 2016): For RealNVP with afffine coupling layers, the logarithm of individual diagonal elements of jacobian, summed over layers till layer / provides the per-dimensional likelihood contribution components at layer I.
Glow Kingma & Dhariwal 2018): Unlike RealNVP where the log-det terms for each dimension can
be expressed as log of corresponding diagonal element of jacobian, Glow contains 1 x 1 convolution blocks having non-diagonal log-det term for channel dimensions, for s x tensor h given by,
log del dd conv2D(h;; W) ){
= 88888 log (W)l
(12)
dh
remains to decompose the log det (W) to individual contribution by each channel. As a suitable candidate, singular values of W correspond the contribution from each channel dimension, so their log value is the individual log-det contribution. So the individual log-det term for channels are obtained by.
det W) = II a (W) + log dot [) = Σ log(o,(WW (W))
(13)
where or (W) are the singular values of the weight matrix W. For affine blocks in Glow, same method as RealNVP is adopted.
3.2 ESTIMATION OF DER-DIMENSION LIKELIHOOD CONTRIBUTION FOR FLOW MODELS WITH ODE BASED DENSITY ESTIMATORS
Recent works flow models such as Behrmann al. (018);; Grathwohl et al. (2018): ; Chen ct al. (2019) employ variants of ODE based density estimators. We introduce method to find per- dimensional likelihood contribution for i-ResNNet (Behrmann al. 2018), which is a residual network
5

---

Under review as a conference paper at ICLR 2020
with invertibilitty and efficient jacobian computation properties. i-ResNet is modelled as a flow F(z), such that 2 = F(z) = (I + (2)) where g(z) is the forward propagation function. The log-likelihood expression is written with the log-det of the jacobian is expressed as power series,
In In (2) Inpz(=) + In det Jr(+)), In Jr(x)) = ir (In (T+ +,())))) = E(1)) tr )
k
where tr denotes the trace. Due to computational constraints, the power series is computed up to finite number of iterations with the tr(J,) term stochastically approximated by hutchinson's 's trace estimator, tr (A) = Bp(r) [uT Au]. Elu] = 0 and Cov(v) = I. The component corresponding to each dimension that becomes part of the log-det term is the diagonal element of J so the per-dimension contribution to the likelihood can be approximated as the diagonal elements of Jr. summed over the power series upto a finite number of iterations n. The diagonal elements are obtained with the hutchinson's 's trace estimator without any extra cost, i.e. e = [1) U2, ". Uaxaxe] {
tr(Jg) K
E Ep(o) (-1) 111
10 J
E plo [(0 Joy yl
k
k
k=1
In above equation, (v JbG is the vector-jacobian product which is multiplied again with The individual components which are summed when (v=")k multiplied with v correspond to the diagonal terms in jacobian, over the expectation Ep(r). So those terms are the contribution by the individual dimensions, to the log-likelihood and are expressed as [ Jaxxx for use in step 1 of dimension factorization step in LCMA implementation for i-ResNet.
4 RELATED WORK
Multi-scale architecture and variants have been successful in number of prior works in deep generative modeling. For invertible neural networks, Finzi et al. (2019) use a keepChannel for selective feed forward of channels analogous to multi-scaling. In the spectrum of generative flow models, multi-scale architecture has been utilized to achieve the dimensionality reduction and enhanced training because of the distribution of loss function in the network (Dinh et al., 2016 Kingma & Dhariwal, 2018). A variant of multiscale architecture has been utilized to capture local variations in auto-regressive models (Reed et al., 2017). Among GoooGodfieloow et al.] 2014)
models, Denton et al. (2015) use a multiscale variant to generate images in coarse-to-fine manner. For multi-scale architectures in generative flow models, our proposed method performs factorization of dimensions based on their likelihood contribution, which in another sense translates to determining which dimensions are important from density estimation and qualitative reconstruction point of view.
Keeping this in mind, we discuss prior works generative flow models which involve multi-scaling and/or incorporate permutation among dimensions to capture their interactions.
A number of generative flow models implement multi-scale architecture, such as Dinh et al. (2016);
Kingma & Dhariwal (2018); Atanov et al. (2019); Izmailov et al. (2019); Durkan et al. (0019) ;
Kumar et al. (2019); Behrmann et al. (2018) etc. Kingma & Dhariwal (2018) introduce an 1 X 1 convolution layer in between the actnorm and affine coupling layer in their flow architecture. The 1 1x1 1 convolution is a generalization of permutation operation which ensures that each dimension can affect every other dimension. This can be interpreted as redistributing the contribution of dimensions to total likelihood among the whole space of dimensions. So Kingma & Dhariwal (2018) treat the dimensions as equiprobable for factorization in their implementation of multi-scale architecture, and split the tensor each flow layer evenly along the channel dimension. We, on the other hand, take the next step and focus on the individuality of dimensions and their importance from the amount they contribute towards the total log-likellhne likelihood. The log-det score is available via direct/indirect decomposition of the jacobian obtained as part of computations in a flow training, so we essentially have a heuristic for free. Since our method focuses individually on the dimensions using a heuristic which is always available, it can prove to be have more versatility in being compatible with generic mclti-scale architectures. [Hoogeboom et al. (2019) extend the concept of x 1 convolutions to invertible d x d convolutions, but do not discuss about multi-scaling Dinh et al. (2016) also include a type of permutation which is equivalent to reversing the ordering of the channels, but is more restrictive and fixed. Flow models such as Behrmann et al. (2018); Grathwohl et al. 2018); Chen et al.)

---

Under review as a conference paper at ICLR 2020
(2019) involve ODE based density estimators. They also implement multi-scale architecture, but the splitting operation is a static channel wise splitting without considering importance of individual dimensions any permutations. Izmailov et al. 2019); Durkan et al. (2019); Kumar et al. (2019);
Atanov et al. (2019) use multi-scale architecture in their flow models, coherent with Dinh et al. (2016)
Kingma & Dhariwal (2018), but perform the factorization of dimensions without any consideration of the individual contribution of the dimension towards the total log-likelihood. For qualitative sampling along with efficient density estimation, we also propose that factorization methods should preserve spatiality of the image in the two splits, motivated by the spatial nature of splitting methods in Kingma Dinh & Dhariwal (2018) (channel-wiss splitting) and Dinh et al. (2016) (checkerboard and channel-wise splitting).
EXPERIMENTS
In Section 3 established that our proposed likelihood contribution based factorization of dimen- sions can be implemented for flow models involving a multi-scale architecture, in order to improve their density estimation and qualitative performance. In this section we present the detailed results of proposed LCMA adopted for the flow model of RealNVP (Dinh et al. (016)) and quantitative comparisons with Glow (Kingma & Dhariwal] 2018) and j-ResNet (Behrmann et al.) (018). For direct comparison, all the experimental settings such data pre-processing, optimizer parameters as well as flow architectural details (coupling layers, residual blocks) are kept the same, except that the factorization of dimensions at each flow layer is performed according the methods described in Section For ease of access, we also have summarized the experimental details in Appendix [A]
For RealNVP, we perform experiments on four benchmarked image datasets: CIFAR-10 (Krizhevsky 2009), Imagenet (Russakovsky et al. 2014) (downsampled to 32 x 32 and 64 x 64), and CelebFaces Artributes (CelebA) (Liu et al. 2015). The scaling in LCMA is performed once for CIFAR-10 10, thrice for Imagenet 32 x 32 and 4 times for Imagenet 64 x 64 and CelebA. We compare LCMA with conventional RealNVP and report the quantitative and qualitative results. For Glow and i-ResNet with ICMA, we perform experiments on CIFAR-10 and present improvements over baseline bits/dim. We also perform an ablation studies for LCMA vs. other possible dimension splitting options.
5.1 QUANTITATIVE COMPARISON
The bits/dim scores of RealNVP with conventional multi-scale architecture (as introduced in Dinh et al. (2016) and RealNVP with LCMA are given in Table [ It can be observed that the density estimation results using LCMA is all cases better in comparison to the baseline. We observed that the improvement for CelebA is relatively high as compared to natural image datasets. This observation was expected as facial features often contain high redundancy and the flow model learns to put more importance (reflected in terms of high log-det) on selected dimensions that define the facial features. Our proposed LCMA exposes such dimensions to more flow layers, making them more expressive and hence the significant improvement in code length (bits/dim) is observed. The improvement in bits/dim is less for natural image datasets because of the high variance among features defining them, which has been the challenge with image compression algorithms. Note that the improvement in density estimation is always relative to the original flow architecture (RealNVP in our case) over which we use our proposed LCMA, we do not alter any architecture other than the dimension factorization method. The quantitative results of LCMA implementation for RealNVP, The density estimation scores
Glow and i-ResNet with CIFAR-IO dataset is summarized in Table 2 for flows with LCMA outperform the same flow with conventional multi-scale architectures.
Table I: Improvements in density estimation (in bits/dim) using proposed method for RealNVP
Model
CelebA
CIFAR-10
ImageNet 32x32
ImageNet 64x64
VP Dinh et al. 2016
3.02
3.49
4.28
3.98
Re flow model with Likelihood
2.71
3.43
4.21
3.92
Contribution based Multiscale Architecture (ours)

---

Under review as a conference paper at ICLR 2020
Table 2: Density estimation results (in bits/dim) for RealNVP, Glow and i-ResNet with LCMA trained on CIFAR-10. "Model for i-ResNet has not fully converged
Type of multi-scale architecture
Re
Glow
i-ResNet
Conventional Multi-scalc Architecture
3.49
3.35
3.45
Likelihood Contribution based Multi-scale Architecture
3.43
3.31
3.40"
5.2 QUALITATIVE COMPARISON
An ideal dimension factorization method should capture the local variance over series of flow layers, which helps in qualitative sampling. For LCMA implementation, we introduced local max and min pooling operations log-det heuristic decide which dimensions to be gaussianized early (Section 3). Figure 2(a] shows samples from original datasets, Figure ((b) shows the samples from trained RealNVP flow model with conventional multi-scale architecture and Figure shows the samples from RealNVP with LCMA trained on various datasets. The finer facial details such as hair styles, eye-lining and facial folds in Celeba samples generated from RealNVP with LCMA were perceptually better than the baseline. The global feature representation observed is similar to that in RealNVP, as the flow architecture was kept the same. The background for natural images such as Imagenet 32 x 32 and 64 x 64 was constructed at par with the original flow model. As has been observed in different flow models such as RealNVP and Glow, the latent space holds knowledge about the feature representation in the data. We performed linear interpolations in latent space to ensure its efficient construction and generated images, as shown in Figure 3 The interpolations observed were smooth, with intermediate samples perceptibly resembling synthetic faces, signifying the efficient construction latent space. More interpolations are included in Appendix
(a) Examples from dataset
(h) Samples from trained RealNVP (Dinh et al. 2016)
(c) Samples from trained RealNVP flow model with LCMA
Figure 2: Samples from RealNVP Dinh al. 2016) and RealNVP flow model with proposed likelihood contribution based multiscale architecture (LCMA) trained on different datasets. The datasets shown in this figure are order: CIFAR-10, Imagenet(32 x 32), Imagenet (64 64) and CelebA (without oowtemmeeatuure sampling). Additional samples are included in Appendix C
8

---

Under review as a conference paper at ICLR 2020
Figure 3: Smooth linear interpolations in latent space between two images from CelebA dataset (without low-temperature sampling). The intermediate samples perceptibly resemble synthetic faces.
5.3
ABLATION STUDY
Table 3: Ablation study results for multi-scale architectures with various factorization methods
Evaluations
Randomn Pormutation
Multiscale architecture with early guussanization high log-det dimensions
NV Diuh et all, 2016
with early gaussianization low log-skt dimensions
Multiscale architecture
Quantitative
3.05
3.10
3.02
2.71
Evaluation (Bitskdim)
Qualitative
Evaluztion
We performed ablation studies to compare LCMA with other methods for dimension factorization in a multi-scale architecture. We consider 4 variants for our study, namely fixed random permutation (Case 1), multiscale architecture with early gaussianization of high log-det dimensions (Case 2). factorization method with checker-board and channel splitting as introduced in RealNVP (Case 3) and multiscale architecture with early gaussianization of low log-det dimensions, which is our proposed LCMA (Case 4). In fixed random permutation, we randomly partition the tensor into two halves, with no regard to the spatiality or log-det score. In case 2, we do the reverse of LCMA, and gaussianize the high log-det variables early. The bits/dim score and generated samples for each of the method are given Table 3 As expected from an information theoretic perspective, gaussianizing high log-det variables early provides the worst density estimation, of important information. Comparing the same with fixed random permutation, the latter has better score the probability of high log-det variable being gaussianized early reduces to half, and it gets further reduced with RealNVP due to channel-wise and checkerboard splitting. LCMA has the best score among all methods, the variables carrying more information are exposed to more flow layers. Fixed random permutation has the worst quality of sampled images, as the spatiality is lost during factorization. The sample quality improves for Case 2 and RealNVP. The sampled images perceptually best for LCMA. Summarizing, LCMA outperforms multi-scale architectures based on other factorization methods, as it improves density estimation and generates qualitative samples.
the model could not capture the high amount
CONCLUSIONS
We proposed a novel multi-scale architecture for generative flows which employs a data-dependent splitting based the individual contribution of dimensions to the total log- log-likelihood. Implementations of the proposed method for several state-of-the-an flow models such as RealNVP (Dinh et al.) 2016)). Glow(Kingma& & Dhariwal 2018) and i-ResNettBBchrann et al.) 2018) were presented. Empirical studies conducted on benchmark image datasets validate the strength of our proposed method, which improves log-likeling likelihood scores and is able to generate qualitative samples. Ablation study results confirm the power of LCMA over other options for dimension factorization.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Andrei Atanov, Alexandra Volokhova, Arsenii Ashukha, Ivan Sosnovik, and Dmitry Vetrov. Semi- conditional normalizing flows for semi-supervised learning. arXiv preprint arXiv:1905 1905.00555, 2019.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. arXiv preprint 1eXiv:///1.0099/ 2018.
Ricky TQ Chen, Jens Behrmann, David Duvenaud, and Jörn- Henrik Jacobsen. Residual flows for 2019.
invertible generative modeling. arXiv preprint arXiv:1906.022
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad. and Pieter Abbeel. Pixelsnail: An improved autore- gressive generative model. arXiv preprint arXiv: / 12.06967,, 2017.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. Proceedings of the 34th International Conference Machine Coarning-Volumme 70, pp. 854 863. JMLR. org, 2017.
Emily Denton, Soumith Chintala, Rob Fergus, et Deep generative image models using a laplacian pyramid of adversarial networks. In Advances in neural information processing systems, pp. 1486-1494, 2015.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 140.85 16, 2014.
Laurent Dinh, Jascha Sohl- Dilll-iistttin, and Samy Bengio. Density estimation using real NVP. CoRR, 0aas/605.08880 2016. URL http:::/aa//xi....... www.ivvoorr/aaae. aa///b///166 1605 08803
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Cubic-spline flows. arXiv preprint arXiv:1906.0214 2019.
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. arXiv preprint arXiv: 2019.
Marc Finzi, Pavel Izmailov, Wesley Maddox, Polina Kirichenko, and Andrew Gordon Wilson Invertible convolutional networks. In Workshop Invertible Neural Nets and Normalizing Flows, International Conference on Machine Learning, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680, 2014.
Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367, 2018
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv: arXiv:1388.08 1308.0850, 2013.
Jonathan Ho. Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. arXiv preprint arXiv: 2019.
Emiel Hoogeboom, Rianne van den Berg, and Max Welling, Emerging convolutions for generative normalizing flows. arXiv preprint 1eXiv:1901.1111 2019.
Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised learning with normalizing flows. In Workshop on Invertible Neural Nets and Normalizing Flows, International Conference on Machine Learning, 2019.
Tero Karras, Samuli Laine, and Timo Aila. A basee generator architecture for generative adversarial networks. arXiv preprint 18Xiv: 04948, 2018.
10

---

Under review as a conference paper at ICLR 2020
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv: 14, 2013.
arXiv preprint
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 27115-10224, 2018.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances neural information processing systems, pp. 3581-3589, 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: flow-based generative model for video. arXiv preprint OrXiv:1903.0144 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp- 3730-3738, 2015.
Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv:1606.01 01583, 2016.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, el. Conditional image generation with pixelcan decoders. In Advances in neural information processing systems, pp. 4790-4798, 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv: /60166661969 2016b.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density estimation. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2912-2921. JMLR. org, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei- Fei Li. Imagenet large scale visual recognition challenge. CoRR, abs/1409.0575, 2014. URL
http:://axivv wwwp://ggg/aaaaa...... 00755
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcn+++: Improving the pixel- cnn with discretized logistic mixture likelihood and other modifications. CoRR, abs/1701.055 005117,
2017. URL http: /ttp://aavv///a////////////..avl www.//g/aaaaoo 17011 C5517
Benigno Uria, lain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive density-estimator In Advances Neural Information Processing Systems, pp. 2175-2183, 2013.

---

Under review as a conference paper at ICLR 2020
A EXPERIMENTAL SETTINGS
For direct comparison with Dinh 2016). data pre-processing, optimizer parameters as well as flow architectural details (coupling layers, residual blocks) are kept the same, except that the factorization of dimensions each flow layer performed according the method described in Section 3 In this section, for the ease of access, we summarize the experimental settings.
Datasets: We perform experiments on four benchmarked image datasets: CIFAR-10 10 (Krizhevsky 2009), Imagenet (Russakovsky et al. 2014) (downsampled to 32 x 32 and 64 x 64), and CelebFaces Attribtess (CelebA) (Liu et al. 2015).
Pre-processing: For CelebA, we take central crop of 148 148 then resize to 64 x 64. For dequantization of images (whose values lies in [0, 256]2) P). the data is transformed to logit(a + (1 z ), where a = 0.05. The sample allocation for training and validation were done as per the a) O 256 official allocation for the datasets.
Flow model architecture: We use affine coupling layers as introduced (Dinh et al. 2016). A layer of flow is defined 3 coupling layers with checkerboard splits with channel splits at s/2 x 8/2 resolution, where is the resolution at the input of that layer. For datasets having resolution 32, we use such layers and for those having resolution 64, use layers. The cascade connection of the layers is followed by 4 coupling layers with checkerboard splits the final resolution, marking the end of flow composition. For CIFAR- 10, each coupling layer uses 8 residual blocks. Other datasets having images of size 32 x 32 use 4 residual blocks whereas 64 x 64 ones use 2 residual blocks. More details on architectures will be given in source code release.
resolution, 3 coupling layers
Optimization parameters: We optimize with ADAM optimizer (Kingma & Ba 2014) with default hyperparameters and use an L2 regularization on the weight scale parameters with coefficient 10-5 A batch size of 64 was used. The computations were performed in NVIDIA Tesla V100 GPUs.
Multiscale Architecture: Scaling is done once for CIFAR- 10, thrice for Imagenet x 32 and times for Imagenet 64 x 64 and CelebA.
B INTERPOLATIONS AMONG TWO IMAGES FROM CELEBA DATASET
Figure presents more interpolation examples obtained using our model between two images from CelebA dataset.
Figure 4: Linear interpolations between CelebA images (without low-temperature sampling)
12

---

Under review as a conference paper at ICLR 2020
C
ADDITIONAL SAMPLES
In this section, we present more samples from RealNVP model with likelihood contribution based multiscale architecture trained on different datasets.
Figure 5: Samples from model trained on CIFAR-1O dataset
Figure 6: Samples from model trained on Imagenet 32 x 32 dataset
13

---

Under review as a conference paper at ICLR 2020
Figure 7: Samples from model trained on Imagenet 64 x 64 dataset
Figure 8: Samples from model trained on CelebA dataset without low-temperature sampling
14


Reviews:
[{"review_id": "rkgJFSRMqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose a heuristic algorithm for deciding which random variables to be Gaussianized early in flow-based generative models. The proposed algorithm involves first training a flow without multi-scale training, for example, 32*32*c  - 32*32*c - 32*32*c. Then, it computes the logdet term for each variable at each layer. It then spatially partition the first flow block by two halves of shape 16*16*2c based on max-pooling the logdet term. Then it recursively Gaussianize one half, and partition the other half as 8*8*4c, still using the pre-computed logdet tensors (Ld in the paper). After partitioning, they train a multi-scale model with the learned partition.\n\nWhile I agree adaptive multi-scale architecture is a topic worth researching, and the paper does have some positive experimental results. I think the writing of the paper is very vague and the techniques are not sensible. \n\nWriting: the main algorithm is just depicted in the last paragraph of Page 4. There are not any equations or pseudocode on what exactly does the proposed algorithm do. Figure 1 and 2 are not detailed enough. For example, Figure 2 doesn't explain how to \"Perform splitting based on log-det heuristic and spatial constraints\". I can only guess what the algorithm is. I suggest the authors make the algorithm more clear, and avoid using large paragraphs of natural language to depict the algorithm. \n\nTechnique: \n1. While I agree partitioning based on logdet term makes some sense, I think *recursively* partitioning without updating the logdet terms is problematic. If the flow only have one layer, the proposed algorithm makes sense. However, for a multi-layer flow model. After the first partitioning, the network changes. For example, for a two layer flow 32*32*c - 32*32*c - 32*32*c, the two layers both have 3*3*c*c filters. However, after partitioning the first output layer as two 16*16*2c, the filter of the second layer should have shape 3*3*2c*2c now. It is not clear how to translate the original, single-scale model into a multi-scale one. And the logdet tensor for the second flow layer doesn't mean anything now. \n\n2. For affine coupling layer, we can indeed compute the logdet term dimension wise. However, it is not clear how to do the computation for other types of flows. For example, invertible ResNets, which estimates the log-det term for each ResBlock with an unbiased estimator.\n\n3. I am also not sure with the training algorithm after pretraining. Do we need to remember what pixel to pickup for each max-pooling operation? That has O(s*s*c*L) space complexity. Does the \"gather\" operation baesd on the memorized locations time consuming?\n\n4. Training a single-scale model has higher time complexity than training a multi-scale model. Is this time complexity too high?\n\n=================\n\nUpdate: thank for the authors for their significant effort on revising this paper.\n\nWriting is indeed much better. However there are still many typos (e.g. jabocian, algorithm). Algorithm 1 is better than the original plain-text version, it still doesn't look like even a pseudocode though. I suggest converting the bullets into actual code, e.g., (try to minimize the amount of natural language since it is vague)\n\nFor each layer l \nLd1, Ld2 <- Maxpooling(sth), Minpooling(sth)\n...\nEndFor\n\nI still don't think my concern 1 is addressed. Imagine a single-scale flow\n\ny = AffineCoupling(x)\nz = AffineCoupling(y)\nh = AffineCoupling(z)\n\nvs a multi-scale flow\n\ny = AffineCoupling(x)\ny1, y2 = LCMA-Split(y)\nz = AffineCoupling(y1)\nz1, z2 = LCMA-Split(z)\nh = AffineCoupling(z1)\n\nI don't think |dh/dz1| of model 2 is a submatrix of |dh/dz| of model 1. Because z1 of model 2 is not a part of z of model 1 in the first place. In model 2, z1 is computed with only y1, while in model 1, z1 is computed with the full y.\n\nConcern 3:\nThe authors partially addressed my question. However, the proposed algorithm is still more expensive than a fixed multi-scale architecture, right? A fixed multi-scale architecture such as RealNVP (next scale block is s/2 * s/2 * 2c) is cheaper than a single-scale architecture (next scale block is s/2 * s/2 * 4c). I guess the time complexity of the proposed approach is the letter one instead of the first one. So the improved likelihood still comes with time complexity cost (comparing with a fixed multi-scale architecture).\n\nTo summarize, I can increase my score to 3 as a positive feedback for the author's effort. But I really think this paper still has a long way to go to be complete.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgVejq6YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe paper investigates a strategy for factoring out dimensions in multi-scale generative flow models. The strategy relies on a heuristic that employs the availability of per-dimension log-determinant terms (e.g. as affine couplings) to decide which half of the dimensions to factor out. Quantitative experimental results support the proposed strategy’s efficacy.\n\nRecommendation: Weak Reject\nThe paper misses several important comparisons and baselines - both in terms of placing the proposed method in a broader context (e.g. how could the method be used in multiscale i-ResNet), as well as providing experimental comparisons that would definitively demonstrate the method’s contribution (e.g. is it the heuristic dimension splitting or the unfactored pre-training that provides the benefit.\n\nThe applicability of the proposed method appears to be overstated. The paper suggests that the method is applicable to any multi-scale flow architecture. However, it depends on the availability of per-dimension log-determinants. As the paper states, these values are readily available for methods that rely on affine couplings (e.g. RealNVP, Glow), but not for methods such as i-ResNets or FFJORD, that also employ multi-scale architectures to factor out variables.\n\n\nMajor comments:\n1. Section 2.1 introduces flows as a mapping from data x to the latent code z, and gives the log-likelihood formula (2) that can be optimized during model training. This presentation is at odds with Section 3, where the authors talk about sampling a latent variable that fixes the “log-latent density” and only leaves the log-determinant term to be optimized (e.g. “[...] the log-latent-density term depends on the choice of the latent distribution and if fixed given the sampled latent variable. [...] once the variable is sampled, maximizing the log-det term results in maximized likelihood.” The two presentations are at odds with each other and should be reconciled, ideally by altering Section 3 since it does not reflect the way flow models are trained.\n2. The authors made an implicit assumption that the log-det term is decomposable into per-dimension component. As mentioned earlier in this review, generally speaking this is not true and methods such as i-ResNets or FFJORD do not (directly?) permit such decomposition, while they do employ multi-scale architectures akin to that of RealNVP. This assumption manifests itself in several places in the paper, which are listed below. Either the authors could make this assumption and the limitations imposed by it very clear, or they could provide additional information on how their method extends to methods that do not provide per-dimension log-det contributions.\na) In Section 3 “The total log-det term is nothing by the sum of log-det terms for contributed by each dimension”. This is obvious (and possibly holds) only for affine couplings.\nb) In Section 3 “Nevertheless, such an analogy can be extended for other flow models which involve a multiscale architecture [...]”. As mentioned, I am not convinced this is true.\nc) In Section 4 “Since our method focuses individually on the dimensions using a heuristic which is always available in flow model training, it can prove to be have more versatility in being compatible with generic multi-scale architecture”. Same as above - in fact I believe that the opposite may be true.\nd) Section 4 does not mention methods that employ multi-scale architectures whil not allowing for discerning per-dimension contributions of the log-det. Mentioning these methods is important for accurately positioning the authors contribution relative to the existing literature.\ne) The method is only applied to the RealNVP architecture. The claim of it being generally applicable to multi-scale architectures would be much stronger if the comparison would include other multi-scale architectures. Here, I strongly recommend i-ResNets or a similar architecture based on log-det approximation to the entire coupling layer.\n3. Unless I missed it, the authors do not mention how they aggregate the log-det values. They are available per datapoint. Are they aggregated over the entire dataset?\n4. In Section 4 the authors imply that the contribution of a dimension towards the total likelihood is dominated by the log determinant (“[...] as the contribution by the variable (dimension) towards the total log-det (~ total log-likelihood)”). This is not obvious, and I am not sure it is always true - it probably depends on the choice of the base distribution p(z).\n5. The authors show experimentally that the use of their variable factorisation heuristic improves log-likelihoods over the baseline RealNVP model; they further perform ablation studies to confirm that preferential factorisation of low log-det dimensions provides best log-likelihoods and qualitative results on the CelebA dataset. However, several points remain unclear.\na) What are the relative contributions of pre-training without dimension factorisation and the proposed factorisation heuristic? Do the experiments shown in Table 2 all employ the pre-training scheme? If so, this should be clearly stated. If not, this set of experiments should be performed.\nb) The networks used in the experiments use between m=1 and 4 factorisation layers. So in total, there are 2^m possible decisions of which half of the variables at those layers to factorise. These numbers are not so large as to prevent us from training all possible combinations of the networks. How does the proposed heuristic compare to just training all (or a few sampled) factorisations decided a priori? This comparison should also investigate the effect of using pre-training.\n6. In the qualitative comparison (Section 5.2) the authors argue two points (i) RealNVP with the proposed method generates finer details on CelebA; and (ii) provides a reasonable latent space that can be interpolated in.\na) With regards to (i), I personally find it difficult to argue about fine details in what are effectively low-resolution images. The differences (which are already hard to see and easy to imagine) could be present due to minute changes in the training code/procedure/random seed/etc. I would be more confident about these differences if the authors could confirm that the methods were trained using the same code base, random seeds, dataset shuffle and framework versions.\nb) How do the qualitative interpolation results compare to RealNVP without the proposed factorisation heuristic?\n\nMinor comments:\n- In Section 2.1 “Let x be a high-dimensional random vector with unknown true distribution x ~ p(x) [...]”. Here x appears to refer to the data or a sample from the data distribution, and not just a random vector. This could be made clearer.\n- In Section 2.1 the “jacobian” should be written with a capital “J”\n- In Section 2.1 and elsewhere, when denoting outputs of intermediate flows f_i as y_i, it is mentioned that y_0 = x, but not that y_k = z. If the latter is true, mentioning this would improve clarity.\n- Section 3 “In a multi-scale architecture, it is apparent that the variables getting exposed to more layers of flow will be more expressive in nature [...]”. I do not find this to be an obvious fact. Perhaps the authors could provide an intuition for why this is true or demonstrate this on a toy example? My confusion may stem from the fact that I am unsure what it means for a variable (i.e. a dimension we factor out) to be more expressive.\n- Some phrasing is a little unusual (e.g. “less (more) log-det tems” instead of “smaller (larger) log-det term”; “to be have”).\n- In Section 5.1 “code length” is introduced without context. The connection between code length, log-likelihood and compression can be made more clear.\n\n------------------------------------------------\nPost rebuttal update\n------------------------------------------------\n\nI’d like to thank the authors for additional elaboration on their baseline experiments that consider their heuristic-based scheme for factoring out dimensions (the low-low-low scheme) and the “anti-heuristic” (which considers the high-high-high scheme). I agree with the authors that both schemes would have been within the 2^m possibilities I encourage them to consider, however I do not fully agree with some of the conclusions the authors draw for these experiments.\n\nFirst, given that the method proposed by the authors is a heuristic, it is not obvious that the high-high-high and the low-low-low schemes indeed correspond the worst and best possible of the 2^m schemes. Running the 2^m experiments could demonstrate this empirically and strengthen the manuscript.\n\nSecond, if we speak about the high-high-high and low-low-low factorizations, then we’ve already gone through the computationally expensive effort of pre-training a model that does not factor out any variables. It is precisely because of this that I’d like to see a baseline that doesn’t do any pre-training, but simply enumerates the 2^m schemes and selects the one with the best validation score. It is not obvious that this approach would computationally more expensive than the pre-training step -- another point I believe is important to cover.\n\nTo summarize, I do not believe that the baselines currently present in the paper are a substitute for the ones requested.\nIncluding new results requires a review of those new results, so I would not be in favour of adding new results at the camera ready stage. I therefore retain my score.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkx20h-RcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new multi-scale architecture for flow-based generative models. Unlike prior work on multi-scale flow architectures which use fixed dimension-splitting heuristics, the proposed approach learns which dimensions to process further. The features are chosen for further processing based on a heuristic motivated by each feature's contribution to the total likelihood. This contribution is given by the each features' contribution to the log-determinant term in the change of variables formula. The model is trained in a two-step process. First a flow model with no multi-scale architecture is trained. Then each feature's  importance is calculated based on its contribution to the log-determinant. Then these scores are used to rank the features for the second, multi-scale model which is retrained from scratch. The authors demonstrate the performance of their approach on density modeling on standard image datasets. They demonstrate an improvement over the standard real-nvp architecture.\n\nOverall, I would recommend to reject this paper. While the proposed method is interesting, I do not feel that the authors  experiments demonstrated its performance relative to other additions on top of coupling-layer based flow models. For example, one could view the invertible 1x1 convolutions from GLOW as a similar way to \"``learn\" how the features should be ordered. The authors do not provide a comparison with more modern flows such as Glow, Residual Flows -- both of which perform considerably better than Real-NVP with the proposed architecture. \n\nWhile I think it is possible that Glow, Residual Flows and other recent flows could benefit from the proposed architecture, the results presented here do not convince me of this fact. If the authors presented a more comprehensive ablation study comparing their architecture to the various additions of other recent work on large-scale flow-based generative models, and their model performed favorably, then I could be convinced to raise my score. \n\n__________________\nPOST REBUTTAL\n__________________\n\nI appreciate the great efforts the authors have gone through to address my initial concerns. Most of these concerns had to do with the limited scope of the paper's initial experiments. The authors have expanded their experiments to incorporate a few more flow-based generative models and have demonstrated their proposed approach can improve their likelihoods across the board. I am somewhat disappointed that they did not attempt to improve upon the current SOTA on CIFAR10 (residual flow) despite the fact that code is available for this model. If the authors had used their method to improve upon the SOTA then their experiments would have been considerably more convincing. \n\nDespite that, I find these new results much more compelling and I would like to make that clear to the area chairs. Given the harsh discretization of the scoring system (1, 3, 6, 8) I cannot in good faith increase my score from a 3 to a 6, but I would like to make it very clear that these changes to the experiments do improve my view of the paper -- moving from a weak reject to neutral, though that is a not an official option. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a multi-scale architecture for generative flows that can learn which dimensions to pass through more flow layers based on a heuristic that judges the contribution to the likelihood. The authors compare the technique to some other flow based approaches. The reviewers asked for more experiments, which the authors delivered. However, the reviewers noted that a comparison to the SOTA for CIFAR in this setting was missing. Several reviewers raised their scores, but none were willing to argue for acceptance. 
Decision: Reject