RG-Flow: A hierarchical and explainable flow model based on renormalization group and sparse prior
Authors: Hong-Ye Hu
Dian Wu
Yi-Zhuang You
Bruno Olshausen
Yubei Chen
Venue:  (2021)
Keywords: Unsupervised learning
representation learning
flow-based generative model
renormalization group
sparse encoding

Abstract: Flow-based generative models have become an important class of unsupervised learning approaches. In this work, we incorporate the key idea of renormalization group (RG) and sparse prior distribution to design a hierarchical flow-based generative model, called RG-Flow, which can separate different scale information of images with disentangle representations at each scale. We demonstrate our method mainly on the CelebA dataset and show that the disentangled representation at different scales enables semantic manipulation and style mixing of the images. To visualize the latent representation, we introduce the receptive fields for flow-based models and find receptive fields learned by RG-Flow are similar to convolutional neural networks. In addition, we replace the widely adopted Gaussian prior distribution by sparse prior distributions to further enhance the disentanglement of representations. From a theoretical perspective, the proposed method has $O(\log L)$ complexity for image inpainting compared to previous flow-based models with $O(L^2)$ complexity.

Under review as a conference paper at ICLR 2021
RG-FLOW: A HIERARCHICAL AND EXPLAINABLE FLOW MODEL BASED ON RENORMALIZATION GROUP AND SPARSE PRIOR
Anonymous authors Paper under double-bil blind review
ABSTRACT
Flow-based generative models have become an important class unsupervised learning approaches. In this work, we incorporate the key idea of renormaliza- tion group (RG) and sparse prior distribution to design a hierarchical flow-based generative model, called RG-Flow, which can separate information different scales of images with disentangled representations at each scale. We demonstrate the CelebA dataset and show that the disentangled repre- sentations at different scales enable semantic manipulation and style mixing of the images. To visualize the latent representations, we introduce receptive fields for flow-based models and find that the receptive fields learned by RG-Flow are similar those in convolutional neural networks. In addition, we replace the widely adopted Gaussian prior distribution by a sparse prior distribution to further enhance the disentanglement of representations. From a theoretical perspective. the proposed method has Oloog I.) complexity for image inpainting compared to previous generative models with (T?2) complexity.
method mainly
INTRODUCTION
One of the most important unsupervised learning tasks is to learn the data distribution and build generative models. Over the past few years, various types of generative models have been proposed. Flow-based generative models are particular family of generative models with tractable distribu-
tions (Dinh et al.
2017 Kingma Dhariwal] 2018 Chen al.) 2018b 2019: Behrmann et al.) 2019
Hoogeboom et al. 2019 Brehmer & Cranmer [2200] Rezende et al. 2020, Karami et al. 2019). 9). Yet
the latent variables are on equal footing and mixed globally. Here, we propose a new flow-based
model, RG-Flow, which is inspired by the idea of renormalization group in statistical physics. RG-
Flow imposes locality and hierarchical structure in bijective transformations. It allows us to access information different scales in original images by latent variables at different locations, which of- fers better explainability. Combined with sparse priors (Olshausen & Field, 1996] [997] Hyvärinen we show that RG-Flow achieves hierarchical disentangled representations.
& Oja 2000
Renormalization group (RG) is a powerful tool to analyze statistical mechanics models and quantum field theories in physics (Kadanofl 1966 [ilson 197I). It progressively extracts more coarse-scale statistical features of the physical system and decimates irrelevant fine-grained statistics at each scale. Typically, the local transformations used RG are designed by human physicists and they are not bijective. On the other hand, the flow-based models use cascaded invertible global transfor- mations to progressively turn a complicated data distribution into Gaussian distribution. Here, we would like to combine the key ideas from RG and flow-based models. The proposed RG-flow en- ables the machine to learn the optimal RG transformation from data, by constructing local invertible transformations and build a hicrarchical generative model for the data distribution. Latent represen- tations are introduced at different scales, which capture the statistical features at the corresponding scales. Together, the latent representations of all scales can be jointly inverted generate the data. This method was recently proposed in the physics community as NeuraIRG (Li & Wang 2018 Hu
al. 2020
Our main contributions are two-fold: First, RG-Flow can separate the signal statistics of different scales in the input distribution naturally, and represent information at each scale in its latent vari-

---

Under review as a conference paper at ICLR 2021
ables& Those hierarchical latent variables live hyperbolic tree. Taking CelebA dataset Liu et al. 2015) as an example, the network will not only find high-level representations, such as the gender factor and the emotion factor for human faces, but also mid-level and low-level representa- tions. To visualize representations of different scales, we adopt the concept of receptive field from convolutional neural networks (CNN) (LeCun) 1988, [eCun al.. 1989) and visualize the hidden structures in RG-flow. In addition, since the statistics are separated into hierarchical fashion, we show that the representations can be mixed at different scales. This achieves an effect similar style mixing. Second, we introduce the sparse prior distribution for latent variables. We find the sparse prior distribution is helpful to further disentangle representations and make them more explainable. The widely adopted Gaussian prior is rotationally symmetric. As a result, each of the latent vari- ables in a flow model usually does not have a clear semantic meaning. By using a sparse prior, we demonstrate the clear semantic meaning in the latent space.
2 RELATED WORK
Some flow-based generative models also possess multi-scale latent space (Dinh et al. 2017 Kingma & Dhariwal p018), and recently hierarchies of features have been utilized in Schirmeisser et al. (2020), where the top-level feature is shown to perform strongly in out-of- tion task. Yet, previous models do not impose hard locality constraint in the multi-scale structure. In Appendix C the differences between globally connected multi-scale flows and RG-Flow are discussed, and we see that semantic, meaningful receptive fields do not show up in the globally connected cases. Recently, other more expressive bijective maps have been developed (Hoogeboom et al. 2019| Karami et al. [0199] Durkan et al. 2019), and those methods can be incorporated into the proposed structure to further improve the expressiveness of RG-Flow.
(OOD) detec-
Some other classes of generative models rely on a separate inference model to obtain the latent representation. Examples include variational autoencoders (Kingma & Welling 2014), adversarial autoencoders (Makhzani et al., 2015). InfoGAN (Chen et al. 2016). and BiGINN (Donahue et al., 2017 Dumoulin et al.) 2017p. Those techniques typically do not use hierarchical latent variables, and the inference of latent variables is approximate. Notably, recent advances suggest that having hierarchical latent variables may be beneficial (Vahdat & Kautz 2020). In addition, the coarse- to-fine fashion of the generation process has also been discussed in other generative models, such
as Laplacian pyramid of adversarial networks (Denton et al. 2015). and multi-scale autoregressive
models (Reed et al. 2017).
Disentangled representations (Tenenbaum Freeman 2000 DiCarlo Cox
2007 Bengio et al.
2013 is another important aspect in understanding how a model generates images
(Higiins et al.
2018 Especially, disentangled high-level representations have been discussed and improved from
information theoretical principles (Cheung et al.] 2015 Chen et al. 2016 201 Sa; Higgins et al.
2017 Kipf et al. 2020; Kim & Mnih] 2018 & Locatello et al. 2019| Ramesh et al.) 2018). Apart from the rerresstttinr the multi scale structure also lis in the distribution of natural images.
If a model can separate information of different scales, then its multi-scale representations can be used perform other tasks, such as style transfer (Gatys al. 2016 Zhu al. (017), face mixing (Karras et al., 2019: Gambardella et al. 20199; Karras et al. (2200). and texture synthesis (Bergmann
al. 2017 Jetchev et al. 2016 Gatys et al. 2013 Johnson al. 2016 Ulyanov al. 2016).
Typically, in flow-based generative models, Gaussian distribution is used as the prior for the latent space. Due to the rotational symmetry of Gaussian prior, an arbitrary rotation of the latent space would lead to the same likelihood. Sparse priors (Olshausen & Field 1996. 1997 Hyvärinen & Oja, 2000) was proposed as an important tool for unsupervised learning and leads to better explain- ability in various domains (Ainsworth et al.] 2018 Arora et al. 2018 Zhang et al., 2019). To break the symmetry of Gaussian prior and further improve the explainability, we introduce sparse prior to flow-based models. Please refer to Figure I2 for quick illustration on the difference between Gaussian prior and the sparse prior, where the sparse prior leads to better disentanglement.
Renormalization group (RG) has a broad impact ranging from particle physics to statistical physics. Apart from the analytical studies in field theories (Wilson] 1971; Fisher 1998] Stanley| 1999). RG has also been useful in numerically simulating quantum states. The multi-scale entanglement renor- malization ansatz (MERA) [Vidal 2008), Evenbly & Vidal 2014) implements the hierarchical struc- ture of RG in tensor networks to represent quantum states. The exact holographic mapping (EHM)
2

---

Under review as a conference paper at ICLR 2021
Q 2013 Lee & Qi 2 You al. 2016) further extends MERA bijective (unitary) flow between latent product states and visible entangled states. Recently, Li & Wang 2018): Hu et al. 2020 incorporates the MERA structure and deep neural networks to design flow-base generative model that allows machine to learn the EHM from statistical physics and quantum field theory ac- tions. In quantum machine learning, recent development of quantum convolutional neural networks also (Cong et al., 2019) utilize the MERA structure. The similarity between RG and deep learn- ing has been discussed in several works (Bény) 2013; Mehta Schwab 2014 Bény Osborne 2013 Oprisa & Toth 2017 Lin et al. 2017] Gan & Shu, 2017). 'hhe information theoretic objective that guides machine- learnin RG transforms are proposed recent works (Koch-Janusz & Ringel,
2018
Hu et al. 2020 Lenggenhager et al., 2020). The meaning of the emergent latent space has
been related to quantum gravity (Swingle 2012| Pastawski et al. 2015). which leads to the exciting
development of machine learning holography (You et al. 2018 Hashimoto et al. 2018 Hashimoto
2019
Akutagawa of al. 2020
Hashimoto et al. 2020).
METHODS
Flow-based generative models. Flow-based generative models are a family of generative models with tractable distributions. which allows efficient sampling and exact evaluation of the probability
density (Dinh et al. 2015 2017 Kingma & Dhariwal 2018 Chen et al. 2019). The key idea is to build bijective map G(z) = x between visible variables æ and latent variables Visible variables a are the data that we want to generate, which may follow a complicated probability distribution. And latent variables usually have simple distribution that can be easily sampled, for example the i..d. Gaussian distribution. this way, the data can efficiently generated by first sampling z and mapping them to through G(z). In addition, data sample x,
get the probability associated with each
OG(z) Oz
log px (x) = log
log
The bijective map G(z) = x is usually composed as series of bijectors, G(z) = G1 G2 ... (z), such that each bijector layer G; has a tractable Jacobian determinant and can be inverted efficiently. The two key ingredients in bow-based models are the design of the bijective map G and the choice of the prior distribution (z)..
Structure of RG-Flow networks. Much the prior research has focused on designing more powerful bijective blocks for the generator G to improve its expressive power and achieve better approximations of complicated probability distributions. Here, we focus on designing the archi- tecture that arranges the bijective blocks in hierarchical structure separate features of different scales in the data and to disentangle latent representations.
(a) Renormalization
(b) Generation
Fine-grained
Coarse- RG scale
Coarse-grained grained
Coarse-grained
Fine-grained
inverse
RG scale
!
(b
(2)
2
decimated features
latent variables
visible variables
Figure 1: (a) The forward RG transformation splits out decimated features at different scales. (b)
The inverse RG transformation generates the fine-grained image from latent variables.
Our design is motivated by the idea of RG in physics, which progressively separates the coarse- grained data statistics from fine-grained statistics by local transformations at different seales. Let be the visible variables, or the input image (level-O), denoted as (0) = æ. step of the RG transformation extracts the coarse-grained information (!)) to send to the next layer (level-1), 1), and
splits the rest of fine-grained information as auxiliary variables 20) The procedure can be
3

---

Under review as a conference paper at ICLR 2021
described by the following recursive equation (at level-h for example),
x (1++1) z (h) = RA (xx!)).
which is illustrated in Fig. a), where dim (z (x(+++))) + dim(2(h) = dim(x!")), and the RG trans- formation Rg can be made invertible. At each level, the transformation RG is local bijective map, which is constructed by stacking trainable bijective blocks. We will specify its details later. The split-out information 2(h) can be viewed as latent variables arranged at different scales. Then the inverse RG transformation G₁, R simply generates the fne-grained image,
x R (xx (hx(h1)),2 (h)) Gn (h)).
(3)
The highest-level image (hh2) Ghz (z(hz)) can be considered generated directly from latent variables ((hz) without referring any higher-level poase-grained image, where h logg I. logg m, for the original image of size I. x with local transformations acting on kernel size m x m.
Therefore, given the latent variables = = {x(h)) at all levels h, the original image can be restored by the following nested maps, as illustrated in Fig. Tb).
x Go(G)(G2(.... ,z
(0)) (0) G(z),
(4)
where = = {=", 3. RG-Flow a flow-based generative model that uses the above compos- ite bijective map G as the generator.
To model the RG transformation, we arrange the bijective blocks in hierarchical network archi- tecture. Fig. _ a) shows the side view of the network, where each green or yellow block is local bijective map. Following the notation of MERA networks, the green blocks are the disentanglers, which reparametrize local variables to reduce their correlations, and the yellow blocks the deci- mators, which separate the decimated features out as latent variables. The blue dots on the bottom the visible variables from the data, and the red crosses are the latent variables z. We omit color channels of the image in the illustration, since we keep the number of color channels unchanged through the transformation.
Fig. b) shows the top-down view of a step of the RG transformation. The green/yellow blocks (disentanglers/deecimtooo" are interwoven on top of each other. The covering area of a disentangler or decimator is defined as the kernel size m. X m of the bijector. For example, in Fig. 2b), the kernel size 4 x 4. After the decimator, three fourth of the degrees of freedom are decimated into latent variables (red crosses in Fig. 2 a)), so the edge length of the image is halved.
As a mathematical description, for the single-step RG transformation Ra. in each block (p, q) la- 1, the mapping from (h)) (x((+))), (x)) is given by
beled by p. q = 0, 1,..., 2 2
(b) (mp+ T +a.mat T +b) (a,
-Redi
(<02 T +a,mat 7 +b) (o.A)En
(1+1) a (mpta.matt) (o,b)E('
(h) 22h (mpta.mmt))) (a,b)EOL -rele (1o5)) (m (a.))
(b)
where D6 {Kka, kb) | b, 0.1..... t} denotes the set of pixels in a x square with stride kt and y is the intermediate result after the disentangler but not the decimator. The no- tation x ((.)) stands for the variable (a vector of all channels) at the pixel (i,j) and at the RG level (similarly for y and 2). The disentanglers Rdi and decimators Rd can be any bijective neural network. Practically, We use the coupling layer proposed in the Real NVP networks (Dinh et al. 2017) to build them, with a detailed description in Appendix A By specifying the RG transforma- tion Rr redee Ddis above, the generator Gr. R is automatically specified as the inverse transformation.
Training objective. After decomposing the statisties into multiple scales, we need make the la- tent features decoupled. So we assume that the latent variables are independent random variables, described by factorized prior distribution
Pz P2(2)= = Il p(z))
(6)
4

---

Under review as a conference paper at ICLR 2021
(b)
Generation Causal Cone
(d)
Inference Classal Cone
Lakeni variabiles (
Latent variables
Latent variables
Vistble variabies
Visible varlables
varlables
Figure 2: Subplot (a) shows the side view of the network. Green/Yellow blocks denote the disentan- glers/decimators, which are bijective maps. Subplot (b) shows the top-down view of the network. The red area in the subplot (c) illustrates the generation causal cone for a latent variable. The blue area in the subplot (d) illustrates the inference causal cone for a visible variable.
where labels every element in z, including the RG level, the pixel position and the channel. This prior gives the network the incentive to minimize the mutual information between latent variables. This minimal bulk mutual information (minBMI) principle was previously proposed to be the infor-
mation theoretic principle that defines the RG transformation (L & Wang 2018 Hu et al. (2200).
Starting from set of independent latent variables x, the generator G should build up correlations locally at different scales, such that the multi-scale correlation structure emerge in the resulting image x to model the correlated probability distribution of the data. To achieve this goal, we should maximize the log likelihood for æ drawn from the data set. The loss function minimize reads
C - (eerpar log logpx(x) = -Be-punn (logp2(((x))) + log 0R(x)
Ox
where R(x)= = (x) = z denotes the RG transformation, which contains trainable parameters.
By optimizing the parameters, the network learns the optimal RG transformation from the data.
Receptive fields of latent variables. Due to the nature of local transformations in our hierarchical network, define the generation causal cone for latent variable be the affected area when that latent variable is changed. This is illustrated as the red cone in Fig. C(c..
To visualize the latent space representation, we define the receptive field for latent variable 2
aG(x)
RF, = (x))(())
(8)
22
where | denotes the 1-norm on the color channel. The receptive field reflects the response of the generated image an infinitesimal change of the latent variable 2,, averaged over pz (z). Therefore, the receptive field of a latent variable always contained in its generation causal cone. Higher-level latent variables have larger receptive fields than those of the lower-level ones. Especially, if the receptive fields of two latent variables do not overlap, which is often the case for lower-level latent variables, they automatically become disentangled in the representation.
Image inpainting and error correction. Another advantage of the network locality can be demonstrated in the inpainting task. Similar the generation causal cone, we can define the in- ference causal cone shown as the blue cone in Fig. d).. If we perturb a pixel at the bottom of the blue cone, all the latent variables within the blue cone will be affected, whereas the latent variables outside the cone cannot be affected. An important property of the hyperbolic tree-like network is that the higher level contains exponentially fewer latent variables. Even though the inference causal cone is expanding well, resulting in a constant number latent variables covered by the inference causal cone on each level. Therefore, if small local region need to be modified, where L is the edge length of the entire image. While for globally connected networks, all (T.2) latent variables have be varied.
go into higher levels, the number of latent variables dilutes exponentially as
image is corrupted, only Oloo L) latent variables
5

---

Under review as a conference paper at ICLR 2021
Sparse prior distribution. We have chosen to hard-code the RG information principle by using factorized prior distribution, i.c. Pz(2) = II p(2)). The common practice is to choose p(æ:) to be the standard Gaussian distribution, which is spherical symmetric. If we apply any rotation to 2, the distribution will remain the same. Therefore, we cannot avoid different features from being mixed under the arbitrary rotation.
To overcome this issue, we use an anisotropic sparse prior distribution for pz (z). In our implemen- tation, we choose the Laplacian distribution p(z)) 26 exp(-Izyl/b), which sparser compared to Gaussian distribution and breaks the spherical symmetry of the latent space. In Appendix [ we show a -ro-dimensional pinwheel example to illustrate this intuition. This heuristic method will en- courage the model to find more semantically meaningful representations by breaking the spherical symmetry.
EXPERIMENTS
Synthetic multi-scale datasets. To illustrate RG-Flow's ability to disentangle representations at different scales and spatially separated representations, we propose two synthetic datasets with multi-scale features, named MSDS1 and MSDS2. Their samples are shown in Appendix [B] In each image, there are 16 ovals with different colors and orientations. In MSDS1, all ovals in an image have almost the same color, while their orientations are randomly distributed. So the color is global feature in MSDSI, and the orientation is a local feature. In MSDS2, the contrary, the orientation is a global feature, and the color a local one.
We implement RG-Flow as shown in Fig. 2 After training, we find that RG-Flow can easily capture the characteristics of those datasets. Namely, the ovals in each image from MSDSI have almost the same color; and from MSDS2, the same orientation. Especially, in Fig. we plot the effect of varying latent variables at different levels, together with their receptive fields. For MSDS1, if we vary a high-level latent variable, the color of the whole image will change, which shows that the network has captured the global feature of the dataset. And if we vary a low-level latent variable, the orientation of only the corresponding one oval will change. As the ovals are spatially separated, the low-level representation of different ovals is disentangled. Similarly, for MSDS2, if we vary hige-level latent variable, the orientations of all ovals will change. And if we vary a low-level latent variable, the color of only the corresponding one oval will change.
For comparison, we also trained Real NVP on our synthetic datasets. We find that Real NVP fails to learn the global and local characteristics of those datasets. Details can be found in Appendix B
High level
MSDS1
MSDS2
Low level
Figure Multi-scale latent representations for MSDS1 and MSDS2.
Human face dataset. Next, we apply RG-Flow to more complicated multi-scale datasets. Most of our experiments use the human face dataset CelebA (Liu et al., 2015). and we crop and scale the images to 32 x 32 pixels. Details of the network and the training procedure can be found in Appendix A Experiments on other datasets, such as CIFAR- (Krizhevsky et a and quantitative evaluations can also be found in Appendix G
After training, the network learns to progressively generate finer-grained images, as shown in Fig. 4 a). The colors in the coarse-grained images are not necessarily the same as those at the same positions in the fine-grained images, because there is no constraint to prevent the RG transformation from mixing color channels.

---

Under review as a conference paper at ICLR 2021
Course-gratned
(a)
(b)
Receptive Fields
(c)
hm3
Generation (Inverse RG)
h-3
h=2
lagicount]
h-1
Fine-grained
Receptive field strength
Figure 4: Subplot (a) shows the progressive generation of images during the inverse RG. Subplot (b) shows some receptive fields of latent variables from low level to high level. The strength of each receptive field is rescaled to one for better visualization. Subplot (c) shows the statistics of the receptive fields' strength.
Receptive fields. To visualize the latent space representation, we calculate the receptive field for each latent variable, and list some of them in Fig. Lb). We can see the receptive size small for low-level variables and large for high-level ones, as indicated from the generation causal cone. In the lowest level (h 0), the receptive fields are merely small dots. In the second lowest level (h = 1), small structures emerge, such as an eyebrow, an eye, part of hair, etc. In the middle level (h 2), we can see eyebrows, eyes, forehead bang structure emerge. In the highest level (h 3), each receptive field grows to the whole image. We will investigate those explainable latent representations the next section. For comparison, we show receptive fields of Real NVP in Appendix C Even though Real NVP has multi-scale structure, since it is not locally constrained, semantic representations at different scales do not emerge.
Learned features on different scales. this section, we show that some of these emergent struc- tures correspond to explainable latent features. Flow-based generative model is the maximal en- coding procedure, because the core of flow-based generative models the bijective maps, and they preserves the dimensionality before and after the encoding. Usually, the images in the dataset live on a low dimensional manifold, and we do not need to use all the dimensions to encode such data. In Fig. 4 c) we show the statistics of the strength of receptive fields. We variables have receptive fields with relatively small strength, meaning that if we change the value of those latent variables, the generated images will not be affected much. We focus on those latent variables with receptive field strength greater than one, which have visible effects on the generated images. We use label the RG level of latent variables, for example, the lowest-level latent vari- ables have h = O, whereas the highest-level latent variables have h = 4. In addition, we will focus on (low level), h (mid level), h 3 (high level) latent variables. There are a few latent variables with h = 0 that have visible effects, but their receptive fields are only small dots with no emergent structures.
most of the latent
For high-level latent representations, we found in total 30 latent variables that have visible effects, and six of them are identified with disentangled and explainable meanings. Those factors are gender, emotion, light angle, azimuth, hair color, and skin color. In Fig. (Ta),, we plot the effect of varying those six high-level variables, together with their receptive fields. For the mid-level latent represen- tations, plot the four leading variables together with their receptive fields in Fig. 5]b,,, and they control eye, eyebrow, upper right bang, and collar respectively. For the low-level representations, some leading variables control eyebrow and an eye as shown in Fig. (]... We them achieve better disentangled representations when their receptive fields do not overlap.
Image mixing in scaling direction. Given two images x and &g, the conventional image mixing takes linear combination between # G (xx)) and æg G (æx) by A + (1 x) with À e [0, 1] and generates the mixed image from x = G(z). In our model, latent variables x is coordinated by the pixel position ( and the RG level h. The direct access of the latent variable
7

---

Under review as a conference paper at ICLR 2021
(a)
High level representations
(b)
Mid level latent representations
Gender
Azimuth
Eye
Emotion
Hair
Eyebrow
Light
Skin
Bang
(c)
level representations
eyehrow
Left
eye
Collar
Figure 5: Semantic factors found on different levels.
(a)
(c)
High level source
Content
Siet
Mid level souree
Contart
010
0x1
0x2
Figure 6: Image mixing in the hyperbolic tree-like latent space.
((,) at each point enables to mix the latent variables different manner, which may be dubbed as "hyperbolic mixing". We consider mixing the large-scale (high-level) features of æ and the small-scale (low-level) features of æ by combining their corresponding latent variables via
(h)
for h O for < θ
(h)
where serves as dividing line of the scales. As shown in Fig. C a), as we change Ô from 0 to 3, more low-level information in the blonde-hair image is mixed with the high-level information of the black-hair image. Especially when = 3, we see the mixed face have similar eyes, nose, eyebrows, and mouth the blonde-hair image, while the high-level information, such face orientation and hair color, is taken from the black-hair image. In addition, this mixing not symmetric under the interchange of A and n,, Fig. (b) for comparison. This hyperbolic mixing achieves the similar effect of StyleGAN (Karras et al. 2019 2020) that we can take mid-level information from an image and mix it with the high-level information of another image. In Fig. (e). show more examples of mixing faces.
Image inpainting and error correction. The existence of the inference causal cone ensures that most Oloog I.) latent variables will be affected, if we have small local corrupted region to be inpainted. In Fig. 7 we show that RG-Flow can faithfully recover the corrupted region (marked as red) only using latent variables locating inside the inference causal cone, which are around one third of all latent variables. For comparison, if randomly pick the same number of latent variables to modify in Real NVP, fails to inpaint as shown in Fig. (Constrinee Real NVP). To achieve the recovery of similar quality in Real NVP, shown in Fig. (Real NVP), all latent variables need to be modified, which are O(L") order. See Appendix for more details about the inpainting task and its quantitative evaluations.
8

---

Under review as a conference paper at ICLR 2021
Ground truth
Corrupted Image
RG-Flow
Constrained Real NVP
Real NVP
Figure 7: Inpainting locally corrupted images.
5 DISCUSSION AND CONCLUSION
In this paper, we combined the ideas of renormalization group and sparse prior distribution to design RG-Flow, a probabilistic flow-based generative model. This versatile architecture can be incorpo- rated with any bijective map to achieve an expressive flow-based generative model. have shown that RG-Flow can separate information at different scales and encode them in latent variables living on a hyperbolic tree. To visualize the latent representations fields for flow-based models in analogy that in CNN. Taking CelebA dataset we have shown that RG-Flow will not only find high-level representations, but also mid-level and low-level ones. The receptive fields serve visual guidance for find explainable representa- tions. In contrast, the semantic representations of mid-level and low-level structures do not emerge in globally connected multi-scale flow models, such Real NVP. We have also shown that the latent representations can be mixed at different scales, which achieves an effect similar to style mixing.
RG-Flow, we defined the receptive
main example,
In our model, if the receptive fields of latent representations do overlap, they are naturally disentangled. For high-level representations, we propose to utilize sparse prior to encourage dis- entanglement. We find that if the dataset only contains a few high-level factors, such as the 3D Chair dataset (Aubry et al., 2014) shown in Appendix G it is hard to find explainable high-level disentangled representations, because of the redundant nature of the encoding flow-based models. Incorporating information theoretic criteria to disentangle high-level representations in the redun- dant encoding procedure will be an interesting future direction.
REFERENCES
Samuel Ainsworth, Nicholas Foti, Adrian KC Lee, and Emily B Fox. oi-vae: Output inter- pretable vaes for nonlinear group factor analysis. In International Conference on Machine Learn- ing. pp. 119-128, 2018
Tetsuya Akutagawa, Koji Hashimoto, and Takayuki Sumimoto, Deep learning and ads/qed. Physical Review D, 102(2), Jul 2020. ISSN 2470-0029, doi: 10. 10.1p00//ysrr 102.026020.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma. and Andrej Risteski. Linear algebraic struc- ture of word senses, with applications to polysemy. Transactions of the Association for Compu- tational Linguistics, 6:483-495 2018.
Mathicu Aubry, Daniel Maturana, Alexci A. Efros, Bryan C. Russell, and Josef Sivic. Seeing 3d chairs: Exemplar part-based 2d-3d alignment using large dataset of CAD models. In 2014 IEEE Conference on Computer Vision and Patern Recognition, CVPR 2014, pp- 3762-3769. IEEE Computer Society, 2014.
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and förn-Henri Jacobsen. Invertible residual networks. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pp. 573-582. PMLR, 2019.

---

Under review as a conference paper at ICLR 2021
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Cédric Bény and Tobias J Osborne. The renormalization group via statistical inference. New Journal of Physics, 17(8):083005, aug 2015, doi: 10. 00.008//136--2630/77888030
Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. Learning texture manifolds with the periodic spatial GAN. In Proceedings of the 34th International Conference on Machine Learning. ICML, 2017, volume 70 Proceedings of Machine Learning Research, pp. 469-477, PMLR, 2017.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estima- tion. CoRR, abs/2003. 13913, 2020.
Cédric Bény. Deep learning and the renormalization group, 2013.
Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems 31; Annual Conference on Neural Information Processing Systems 2018,, pp. 2615-2625, 201 8a.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, pp. 6572-6583, 2018b.
Tian Qi Chen, Jens Behrmann, David Duvenaud, and Jörn- Henrik Jacobsen. Residual flows for in- vertible generative modeling. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, pp. 9913-9923, 2019.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Πla Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Informa- fion Processing Systems 2016, pp. 2172 2180, 2016.
Brian Cheung, Jesse A. Livezey, Arjun K. Bansal, and Bruno A. Olshausen. Discovering hidden fac- tors of variation in deep networks. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Iris Cong, Soonwon Choi, and Mikhail D. Lukin. Quantum convolutional neural networks. Nature Physics, 15(12):1273-1277 Dec 2019. ISSN 1745-2481. doi: 10. 10.10388/41157-019--6666 0648---..
Emily L. Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015. pp. 1486-1494, 2015.
James J DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, 1 113333333-34, 2007.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non non-linear independent components esti-
mation. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, Conference Track Proceed- ings. OpenReview.net, 2017.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In 5th Interna- tional Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martín Arjovsky, Olivier Mastropi- etro, and Aaron C. Courville. Adversarially learned inference. In 5th International Conference on Learning Representations, ICLR 20/7. OpenReview.net, 2017.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Ad- vances Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, pp. 7509-7520, 2019.
10

---

Under review as a conference paper at ICLR 2021
G. Evenbly and G. Vidal. Class of highly entangled many-body states that can be efficiently simu- lated. Phys. Rev. Lett., 112:240502, Jun 2014. doi: 10.1 10.110//PtysReeeLttt11..... 12.244000.....
Michael E. Fisher. Renormalization group theory: Its basis and formulation statistical physics. Rev. Mod. Phys., 70:653-681 681, Apr 1998. doi: 10.1 10110eeeooPPyyys 70.653.
Andrew Gambardella, Atilim Günes Baydin, and Philip H. S. Torr. Transflow learning: Repurposing flow models without retraining. CoRR, 105/1911.112700 2019.
Wen-Cong Gan and Fu- We- Shu. Holography as deep learning. International Journal of Modern
Physics D, 26226112) 26(12))1743000, Oct 2017. ISSN 1793-6594. doi: 10.1 10.11422001122718174300000
L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414-2423, 2016.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems 28: Annual Conference Neural Information Processing Systems 2015, pp. 262-270, 2015.
Koji Hashimoto. AdS/ /CFT correspondence deep boltzmann machine. Phys. Rev. D, 99: 106017, May 2019. doi: 10.1103/PtyyeeeDD99..006
Koji Hashimoto, Sotaro Sugishita, Akinori Tanaka, and Akio Tomiya. Deep learning and holo- graphic qed. Phys. Rev. D, 98: 106014, Nov 2018. doi: 10. 103/Physseev..9.... 10660000.....001111.. 106014.
Koji Hashimoto, Hong-Ye Ye Hu, and Yi-Zhuang You. Neural ode and holographic qed, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, pp- 770-778. IEEE Computer Society, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In Sth International Conference on Learning Representations, ICLR 2017. OpenReview.net, 2017.
Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loic Matthey, Danilo J. Rezende, and Alexander Lerchner. Towards definition of disentangled representations. CoRR, abs/1812.02230, 2018
Emiel Hoogeboom, Rianne van den Berg, and Max Welling. Emerging convolutions for generative normalizing flows. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 Proceedings of Machine Learning Research, pp. 2771-2780 PMLR, 2019.
Hong-Yz Ye Hu, Shuo-Hui Li, Lei Wang, and Yi-Zhuang You. Machine learning holographic mapping by neural network renormalization group. Phys. Rev. Research, 2:023369, Jun 2020.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4):41 1-430, 2000.
Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf. Texture synthesis with spatial generative adversarial networks. CoRR, abs/161 /tt//6611.08200 1082077 2016.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision ECCV 2016 14th European Conference, Proceedings, Part II, volume 9906 of Lecture Notes in Computer Science, pp. 694-711. Springer, Sppiinrr 2016.
Lco Kadanoff. Scaling laws for ising models near Tę. Physics Physique Fizika, 2:263-272, Jun 1966.
Mahdi Karami, Dale Schuurmans, Jascha Sohl-Dickstein, Laurent Dinh, and Daniel Duckworth. Invertible convolutional flow. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, pp. 5636-5646 2019.
11

---

Under review as a conference paper at ICLR 2021
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pp. 4401-4410 Computer Vision Foundation IEEE, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Ana- lyzing and improving the image quality of stylegan. In 2020 TEEE/CVF Conference on Computer Vision and Pultern Recognition, CVPR 2020, pp. 8107-8116 16. IEEE, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the 35th Inter- national Conference on Machine Learning, ICML 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2654 2663. PMLR, 2018.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolu- tions. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, pp. 10236-10245, 2018.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, 2014.
Thomas N. Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world mod- els. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020.
Maciej Koch-Janusz and Zohar Ringel. Mutual information, neural networks and the renormal- ization group. Nature Physics, 14(6):578-582, Jun 2018. ISSN 1745-2481. doi: 10. 10310 541567-018-0081-4
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced search). URL TRLpttt:://wwtt www. cs www.otooooooooooooooooo....o Ccooootto.dda//A//////ieeaa
Y. LeCun. A theoretical framework for back-propagation 1988.
Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. Handwritten digit recogmition with a back-propagaee net- work. In Advances in Neural Information Processing Systems 2, pp. 396-404. Morgan Kaufmann, 1989.
Ching Hua Lee and Xiao-Liang Qi. Exact holographic mapping free fermion systems. Phys. Rev. B, 93:035112, Jan 2016. doi: 10. 1.110///hyyReeB.93..333
Patrick M. Lenggenhager, Doruk Efe Gökmen, Zohar Ringel, Sebastian D. Huber, and Maciej Koch- Janusz. Optimal renormalization group transformation from information theory. Phys. Rev. X, 10: 011037, Feb 2020. doi: 10. N0.1103//hyseev... 10..0eeexx..00010.7.
Shuo-Hui Li and Lei Wang. Neural network renormalization group. Phys. Rev. Lett., 121:260601, Dec 2018
Henry W. Lin, Max Tegmark, and David Rolnick. Why does deep and cheap learning work so well? Journal of Statistical Physics, 16266):1233-1247, Sep 2017. ISSN 1572-9613. doi: 10.1107773009555-01--1366-
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In 2015 IEEE International Conference on Computer Vision, 2015, Santiago, Chile, December 7-13, 2015, 3730-3738 IEEE Computer Society, 2015. doi: 10. 0..100//CCC.20115.442 URI.
https: //do:////o.... .rrgg/rggg.. / 10. 1109/ICV 2015. 425
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pp. 4114-4124. PMLR, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference Learning Representations, ICLR 2019. OpenReview.net, 2019.
12

---

Under review as a conference paper at ICLR 2021
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and lan J. Goodfellow. Adversarial autoen- coders. CoRR, abs/151 2015.
Pankaj Mehta and David J. Schwab. An exact mapping between the variational renormalization group and deep learning, 2014.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607, 1996.
Bruno A Olshausen and David Field. Sparse coding with overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311 131331333,,,,, 1997.
Dan Oprisa and Peter Toth. Criticality & deep learning ii: Momentum renormalisation group, 2017.
Fernando Pastawski, Beni Yoshida, Daniel Harlow, and John Preskill. Holographic quantum erro- correcting codes: toy models for the bulk/boundary correspondence. Journal of High Energy Physics, 2015(6):149, Jun 2015. ISSN 1029-8479) doi: 10.1007/JHEP6((055544..
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pp. 8024-8035, 2019.
Xiao-Liang Qi. Exact holographic mapping and emergent space-time geometry. arXiv: High Energy Physics Theory, 2013.
Aditya Ramesh, Youngduck Choi, and Yann LeCun. spectral regularizer for unsupervised disen- tanglement. CoRR, abs/ 1///1812.0116// 2018.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gomez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density esti- mation. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2912-2922229 2921. PMLR, 2017.
Danilo Jimenez Rezende, George Papamakarios, Sébastien Racanière, Michael Albergo, Gurtej Kanwar, Phiala E. Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. CoRR, 024/20002.02288, 2020.
Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to ac- celerate training deep neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference Neural Information Processing Systems 2016, pp. 901, 2016.
Robin Tibor Schirrmeister, Yuxuan Zhou, Tonio Ball, and Dan Zhang. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. CoRR, abs/2006. 10848, 2020.
H. Eugene Stanley. Scaling, universality, and renormalization: Three pillars of modern critical phenomena. Rev. Mod. Phys., 71:6358-$366, Mar 1999. doi: 10. 10.100//
Brian Swingle. Entanglement renormalization and holography. Phys. Rev. D, 86:065007, Sep 2012. doi: 10. 10.1103/Phyyseee.36..66000.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818-2826 IEEE Computer Society, 2016. doi: 10.1109/VPPR.2006..33 URL https:
7/doi. orgii.org/iie 10. 1109/CVPR 2016 3.80
Joshua B. Tenenbaum and William T. Freeman. Separating style and content with bilinear models. Neural Comput,, 1247-1224, 2000.
13

---

Under review as a conference paper at ICLR 2021
Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor S. Lempitsky. Texture networks: ferwaro synthesis of textures and stylized images. Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, volume 48 of JMLR Workshop and Conference 2016.
Proceedings, pp.
Arash Vahdat and Jan Kautz. abs/2007.03898,, 2020.
NVAE: A deep hierarchical variational autoencoder. CoRR,
G. Vidal. Class of quantum many-body states that can be efficiently simulated. Phys. Rev. Lett., 101: 100001 Sep 2008. doi: 10.1 10.1103/PhysReveet.. 101. 10501.
Kenneth G. Wilson. Renormalization group and critical phenomena. i. renormalization group and the kadanoff scaling picture. Phys. Rev. B, 4:3174-3183, Nov 1971.
Yi-Zhuang You, Xiao-Liang Qi, and Cenke Xu. Entanglement holographic mapping of many-body localized system by spectrum bifurcation renormalization group. Phys. Rev. B, 93:104205, Mar 2016. doi: 10.1 1w.. 104205.
Yi-Zhuang You, Zhao Yang, and Xiao-L.ang Qi. Machine learning spatial geometry from entangle- ment features. Phys. Rev. 97:04515,, Jan 2018. doi: R..103/Pyyseev..97..0 97.055553.
Juexiao Zhang, Yubei Chen, Brian Cheung, and Bruno A. Olshausen. Word embedding visualization via dictionary learning. CoRR, abs/1910.03833, 2019.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE Interna- tional Conference on, 2017.
14


Reviews:
[{"review_id": "yj4q2wBrVw8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper proposes a method, named as RG-flow, which combines the ideas of Renormalization group (RG) and flow-based models. The RG is applied to separate signal statistics of different scales in the input distribution and flow-based idea represents each scale information in its latent variables with sparse prior distribution. Inspired by receptive field from CNNs, the authors visualize the latent space representation, which reveals the progressive semantics at different levels as the instinctive expectation.\n\n## Strengths\n\n+. The proposed method presents a simple method to combine renormalization group and flow-based models.\n\n+. The visualized receptive field of latent space representation shows meaningful semantics at different levels, which is verified on CelebA.\n\n## Weaknesses/Concerns\n\n-. It seems like a combination of existing methods, including Flow-based methods, renormalization group and receptive field. Compared with the two mentioned NeuralRG papers(Li & Wang, 2018; Hu et al., 2020), what are the differences for the renormalization group part? The main contributions of the proposed method are not so clear.\n\n-. In the flow part, the high-dimensional visual variables are modeled by a sparse prior distribution, while the visual results (e.g., Figure 4, 5, 6 in the main paper) are with limited diversity. Does mode collapse happen during modeling the distribution?\n\n-. It could be too strong that enforces the high-dimensional features at different layers to follow a same sparse prior distribution. Though the proposed method achieve acceptable results on CelebA which is an almost aligned face dataset, it fails to model the images in the wild (e.g., CIFAR-10 on Figure 8 in supplementary). Does it mean the proposed method works only on aligned/structured scenes? Specially, compared with the style-based generator[1,2], where the sparse prior distribution is enforced only in the input z latent space, I don’t find superiorities of the proposed method.\n\n[1] A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR 2019.\n\n[2] StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows, arXiv 2020\n\n-. Given the visualization methods can have a great effect on the results, Fig. 9 is a bit weak to show the advantages of breaking the spherical symmetry of the style latent. What are the clusters in latent space, visual space and target?   \n\n-. The experiments are conducted on a low-resolution setting (i.e., 32x32 on CelebA). Does it work on higher resolution?\n\n-. There is no qualitative and quantitative evalution to compare the performance or show the improvement gained by the proposed method.\n\n-. In order to verify renormalization group helps on learning more meaningful semantics at different levels, it's better to show an ablation study to compare the visualization of receptive field between w/ RG and w/o RG. \n\n-. What’s the meaning of y in equation (5)? \n\n\n## Overall Recommendation\n\nBased on the above weaknesses/concerns, I rate the paper \"Marginally below acceptance threshold\". \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZcBHIOU991", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper introduces an RG-Flow model -- a hierarchical flow model based on the idea of the renormalization group. The experiments suggest that this model is expressive and capable of learning the disentangled representations at different scales. Additionally, the authors show how this method can be adapted for image inpainting and theoretically analyze its complexity.\n\nPros:\n- Novel and interesting idea.\n- The proposed method allows for separating the features at different scales (at low, mid, and high levels) due to the hierarchical structure of the model.\n\nConcerns:\n- The paper would benefit from additional experiments and comparison to other methods. For example, you can experiment with the datasets specific for the analysis of disentangled representations, e.g., Shapes 3D, MPI 3D, etc.\n- The paper lacks the quantitative assessment of the degree of disentanglement of learned representations. I suggest to refer to the paper [1] for the list of disentanglement metrics. While they were invented for VAE-based models, you can adapt it to your method.\nSimilarly, you could add the assessment of the inpainting quality apart from the visual evaluation in Figure 6.   \n\nQuestions: \n- Did you measure the quality of generated images, e.g., by computing FID or Inception score? While the samples for the CelebA dataset look good, the samples for CIFAR-10 (Fig.8) are not impressive.\n-  I did not find any details on the experiments on the 3D Chair dataset apart from Table 1 in the Appendix. Was the setting the same as for CelebA and CIFAR-10 datasets? Is it possible to compare the results of RG-Flow to Real NVP for the 3D Chair dataset?\n\nComments:\n- I suggest adding the paper [2] to the related work; this paper introduces Multiscale Entanglement Renormalization Ansatz (MERA), a structure similar to your network.\n- Take a look at the paper [3], bridging the entanglement renormalization and wavelets. \n\nUPD: I am satisfied with the authors' response, and therefore I increase the score. \n\nReferences:\n\n[1] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., & Bachem, O. (2019, May). Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning (pp. 4114-4124).\n\n[2] Vidal, G. (2008). Class of quantum many-body states that can be efficiently simulated. Physical review letters, 101(11), 110501.\n\n[3] Evenbly, G., & White, S. R. (2016). Entanglement renormalization and wavelets. Physical review letters, 116(14), 140403.", "extra_scores": {}, "extra_text": {}}, {"review_id": "N6B0VtZR3oR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe submission suggests a generative model with a normalizing flow based on a renormalization group idea. This gives rise to a multi-scale latent representation by arranging bijective transformations hierarchically with depth scaling logarithmically with the input dimension. Extensive experiments on CelebA are conducted, illustrating the latent feature on different scales and image inpainting experiments.\n\nPositives:\nThe paper includes substantive experiments that (i) visualize the latent features on the different hierarchical levels, (ii) illustrate mixing of latent variables across different levels, (iii) show good performance in error correction. \nThe proposed RG-Flow network architecture is very interesting indeed and is new for the considered tasks as far as I am aware. The complexity of the approach can improve from O(L^2) for previous normalizing flows for image inpainting to O(log L). This is a very significant improvement (but it is not clear to me how well the architecture performs for corruptions that are not just in a single local region).\n\nNegatives:\nSimilar flow architectures have been used before in different contexts (mostly physics) as also mentioned in the paper, but I feel that it is not so clear how the proposed normalizing flow model differs from previous work with a renormalisation group (NeuralRG). The role of symmetric priors for disentangled representations has been studied previously (using information theoretical ideas, for VAEs etc) and I also feel that it should be clarified how the sparse Laplace prior differs from previous work. \n\nRecommendation:\nThis appears a borderline submission for me, but my confidence for assessing it is not that high. The experimental section is solid and I would consider increasing the score if the differences to previous work are made clearer.\n\nComments:\nDoes the choice of the temperature for the prior has a large influence on the learned features?\nHow does the computational complexity and running time of the RG architecture compare with other flow based generative models like Glow/(plain) RNVP?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "67X7r-ufl1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new architecture for flow based generative models. The model imposes a hierarchical structure over information at different scales. The paper shows that the hierarchical structure results in disentangled features at different levels of abstractions. The paper claims that the approach is based on the renormalization group\n\n\nStrengths:\n+ The overall structure of the approach is intuitive, and the writing/figures are understandable \n+ I like how the approach naturally produces disentangled features at different levels of abstraction. \n+ Figure 9 demonstrates how the sparse prior leads to better disentanglement than the Gaussian prior. Due to the rotation invariance of the Gaussian prior, the hidden dimensions of other flow based generative models like Real NVP don't have grounded semantic meaning.\n+ The hierarchical approach shows a small improvement over \n\n\nWeaknesses:\n- (Intro/related work) There should be more discussion how this paper relates to Neural Network Renormalization Group (Li & Wang). Some more background/examples of the renormalization group in physics / meaning of the disentangler/decimator would be helpful\n\n- There is no evaluation of the image inpainting performance. Some other papers (Structured Output Learning with Conditional Generative Flows ,Lu & Huang) use PSNR. \n- Constrained NVP seems to be too weak a baseline for the image inpainting task. Are there more reasonable comparisons? Since the paper claims a log(L) vs L^2 improvement, timing or flop counts should be used to compare to Real NVP. \n- The paper provides limited quantitative results and comparisons. Only qualitative comparisons are provided on the in-painting task. There are no quantitative results provided in the main paper, and only results on CelebA, CIFAR-10, and 3D chair in the appendix.  There is no discussion of the 3D chair experiment in the appendix.\n\nSome minor points:\n* Notation in Eq 5. should be improved for better understanding\n* What is the reason for the checkerboard artifacts shown in the progressive generation (Figure 3)?  \n* While theoretically each pixel only depends on log(L) latent variables\n\nUpdate: The authors response has addressed my major concerns. Particularly, the revised version is more clear about how this paper relates to prior work, more comparisons/metrics are added, and issues I had related to clarity were addressed. The revised version has significantly expanded the scope of experiments, which better demonstrate the advantage of hierarchical features for normalizing flows. I have update my rating to marginal accept", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a hierarchical flow-based generative model to learn disentangled features at different levels of abstractions.  The key technical contribution is a combination of renormalization group and flow-based models. The reviewers do find the idea interesting. However, the merit of the work with respect to StyleGAN and StyleFlow has not been well established. AR3 made the following comment:  “Specially, compared with the style-based generator[1,2], …, I don’t find superiorities of the proposed method.” The authors responded to the comment briefly (but not convincingly) in their rebuttal. There is no mention of it in the revised paper. A proper account of the issue would require major revision to the paper.
Decision: Reject