Deep automodulators
Authors: Ari Heljakka
Yuxin Hou
Juho Kannala
Arno Solin
Venue:  (2020)
Keywords: unsupervised learning
generative models
autoencoders
disentanglement
style transfer

Abstract: We introduce a novel autoencoder model that deviates from traditional autoencoders by using the full latent vector to independently modulate each layer in the decoder. We demonstrate how such an 'automodulator' allows for a principled approach to enforce latent space disentanglement, mixing of latent codes, and a straightforward way to utilize prior information that can be construed as a scale-specific invariance. Unlike GANs, autoencoder models can directly operate on new real input samples. This makes our model directly suitable for applications involving real-world inputs. As the architectural backbone, we extend recent generative autoencoder models that retain input identity and image sharpness at high resolutions better than VAEs. We show that our model achieves state-of-the-art latent space disentanglement and achieves high quality and diversity of output samples, as well as faithfulness of reconstructions.

Under review as a conference paper at ICLR 2020
DEEP AUTOMODULATORS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce novel autoencoder model that deviates from traditional autoen- coders by using the full latent vector to independently modulate each layer in the decoder. We demonstrate how such an 'automodulator' allows for princi- pled approach to enforce latent space disentanglement, mixing of latent codes, and straightforward way to utilize prior information that can be construed as a scale-specific invariance. Unlike the GAN models without encoders, autoencoder models can directly operate on new real input samples. This makes our model directly suitable for applications involving real-world inputs. As the architectural backbone, we extend recent generative autoencoder models that retain input iden- tity and image sharpness at high resolutions better than VAEs. We show that our model achieves state-of-the-ar latent space disentanglement and achieves high quality and diversity of output samples, as well as faithfulness of reconstructions.
INTRODUCTION
This paper introduces a new generative autoencoder for learning representations of image data sets, in way that allows arbitrary combinations of latent codes to generate images (see Fig. I We achieve this with an architecture that uses adaptive instance normalization (AdaIn, Dumoulin et al. 20176] Huang & Belongie 2017), and training methods that let the model learn a highly disentangled latent space by utilizing progressively growing autoencoders (Heljakka et al. 2019). In a typical autoencoder, input images are encoded into latent space, and the information of the latent variables is then passed through successive layers of decoding until a reconstruction of the input image has been formed. In our model, the latent vector independently modulates the statistics of each layer of the decoder-the output of layer n. is no longer solely determined by the input from layer n - 1.
In image generation, the probability mass representing sensible images (such as human faces) lies concentrated on a low-dimensional manifold. Even if impressive results have been shown for image generation (e.g., by GANs. [Goodfellow et al. [2144] Karras et al., (209)). efficient reconstruction and manipulation remain open problems. Deep generative autoencoders provide a principled approach for feature extraction, editing, and image synthesis. We show that modulation of decoder layers with AdaIn further improves these capabilities. It also yields representations that are more disen- tangled, a property here broadly defined as something that allows for fine control of one semantic (image) sample attribute independently of others. Previous works on Adaln are mostly based on GAN models (Karras et al. (219)] Chen et al., 2019) with no encoder for new input images.
decoder-decoder structures, where sample enters from one end and is reconstructed at the other. For robustness, the reconstructed samples could be re- introduced the encoder, repeating the process, and requiring consistency between the passes. In comparison to stochastic models (like VAEs, Kingma & Welling 2014 Rezende et al., 2014). our deterministic model is better suited to take advantage of this. We can mix the latent codes of separate samples and measure the conservation of layer-ppecific information for each. This enforces disentanglement of layer-specific properties, because we can ensure that the latent code introduced from certain layer onwards on the 1st pass should not affect the previous layers on the 2nd pass. Image autoencoders with convolutional architectures are often used for finding attributes that are disentangled from each other in the latent space. Often, they have relatively poor output image quality, and prior information must be fed in either via ctass-conditionin or by more complex loss functions for the latent variables. Fully implicit methods avoid this, but without built-in encoding mechanism often cover only a relatively small fraction of the variation in the training data. For the first time, we show full autoencoder architecture that has state-of-the-ard disentanglement
Autoencoders typically single-pass

---

Under review as a conference paper at ICLR 2020
Inpat
Modulated oulput
Modulated
Real input images
modullatdd
modulated
CCorre" features modullaed
Ioput
Reconstruction
Random sample
Conditional samples given "coame" and "lnee Teatures from
Illustration of some automodulator capabilities. The model can directly encode real (un- seen) imput images (left). Inputs be mixed by modulating with another or with randomly drawn sample, at desired scales (center). Finally, taking random modulations for certain scales produces novel samples conditioned the input image (right).
Figure I:
performance for unsupervised training setup in several data sets, sharp image output quality, good coverage of the variation of the training data, arbitrary mixing of latent representations and principled approach for incorporating scale-specific prior information.
Our contributions are as follows. (i) We introduce the automodulator, a new autoencoder-liks model with powerful properties not found regular autoencoders, including scale-specific style transfer (aatys et al.) (0016). In contrast to architecturally similar 'style" -baed GANs, we can now directly encode manipulate new inputs. (ū) We present methodology that allows fully unsupervised train- ing of such model to learn rich latent representation in high resolutions, and specifically explain a novel multi-pass training principle that enables the model to learn a more disentangled represen- tation of training data than regular autoencoders. (iii) We demonstrate qualitative and quantitative performance and applications on CELEBA-HO and LSUN Bedrooms and Cars data sets, taking advantage of the highly disentangled model structure.
2 RELATED WORK
Our work builds upon several lines of previous work in unsupervised representation learning. The most relevant concepts are variational autoencoders (VAEs, Kingma & Welling 2014) and genera- tive adversarial networks (GANs, Goodfellow et al. 2014). In VAEs, an encoder maps data points to lower dimensional latent space and a decoder maps the latent representations back to the data space. The model is learnt by minimizing the reconstruction error together with regularization term that encourages the distribution latents to match a predefined prior. Latent representations often provide useful features for applications (e.g., image analysis and manipulation) and decod- ing random samples from the prior allows novel data generation. However, in case of images, the generated samples are often blurry and not fully photorealistic, with imperfect reconstructions.
Current state-of-fe saate-of-the-ee the-hatet art in generative image modeling represented by GAN models (Brock et al.) 2019 Karras al. 2019) which achieve higher image quality than VAE based models. Neverthe- less, these GANs lack an encoder for obtaining the latent representation for a given image, limiting their usefulness. In some cases, given image can be semantically mapped to the latent space via generator inversion but this iterative process is prohibitively slow for many applications, and the result may depend on initialization (Abdal et 2019 Creswell & Bharath 2019).
Bidirectional mapping has been targeted by VAE-GAN hybrids such as Makhzani et al. 2016);
Mescheder al. (2017; Makhzani (018). and adversarial models such as Donahue et al. (017);
Dumoulin et al. 7201aa Donahue & Simonyan (2019). These models learn mappings between the data space and latent space using combinations of encoders, generators and discriminators. In- tro (Huang al. 2018 and AGE (Ulyanov et al. 2018) are more compact autoencoders, which both contain only two deep networks, encoder and decoder, learnt using combination of recon- struction loss and adversarial loss, which encourages the distributions of real and generated images to match with the prior in latent space. A progressively grown version of AGE, PIONEER, was pro- posed by Heljakka et al. (2018 2019. There is also recent work vector quantized variational
2

---

Under review as a conference paper at ICLR 2020
autoencoders (VQ-VAE, Razavi et al. 2019) with discrete latent space. However, while suitable for image compression, the discrete latent representation cannot e.g. be interpolated and hence may not be optimal for semantic image manipulation and learning disentangled latent space.
The architecture of our generator and AdaIn utilization was inspired by the recent StyleGAN (Kar- ras et al. 2019), but has discriminator (nor the 'mapping network' f). Instead, our model simply contains the generator (i.e,, decoder) and an encoder, jointly trained applying a similar progressive growing strategy as [Heljakka et al. 2018), but with modified loss formulation. Besides the re- construction loss and adversarial loss from Ulyanov et al. (2018) and Heljakka et al. (2018), we can also recirculate style-mixed reconstructions as "second pass' training samples in order to better encourage the independence and disentanglement of emerging styles and conservation of layer- specific information. The recireulation idea is biologically motivated and conceptually related to many works, such as that by Hinton & McClelland (1988). Utilizing the outputs of the model as inputs for the next iteration is related to, e.g., Zamir et al. (2017), where feedback shown to benefit image classification, and in RNN-based methods (Rezende et al. 2016 Gregor et al. 2015 2016).
e.g..Zamir
3 METHODS
We now describe the model architecture, starting from regular PIONEER and building up to the automodulator (Sec. (.1). We then describe the unsupervised training via our extensions to the regular PIONEER loss function, to support flexible mixing of latents (Sec. (S2c.3.2). Finally, we introduce an additional weakly supervised training method to utilize known invariances (Sec. 3.3
3.1 AUTOMODULATOR ARCHITECTURE
Our interest is in unsupervised training of autoencoder wherein the inputs are images fed through an encoder Φ to form a ciw-dimensional latent space representation z (we use z E R212 normalized unity). This representation can then be decoded back into image & through decoder 0. For stable training, we adopt the progressively growing architecture of Balanced PTO- NEER (Heljakka et al., [2018] 2019). The convolutional layers of the symmetric encoder and decoder are faded in gradually during the training, in tandem with the resolutions of training images and generated images (Fig. 2).
Adaptive Instance Normalization (AdaIn) A traditional decoder architecture would start from a small-resolution image and expand it layer by layer until the full image is formed, feeding the full information of the latent code via the decoder layers. In contrast, our decoder is composed of layer-wise functions (E,-1, 2) that separately take a 'canvas' variable ε denoting the content input from the preceding decoder layer (see Figs. and 3ap, and the actual (shared) latent code z. First, each deconvolutional layer ti computes its feature map output X; from & as in traditional decoders. Second, we take the AdaIn normalization step that uses mean y and variance o² of the output. To do this, we need a map gi 2 + (M4c, o arriving at:
modulate the channel-wise
Xi ((x:) E, (.daIn(xx.g.:2))) = ( (xx) (
+ 14:
We implement gi for layer #i as fully connected linear layer, with output size 2 2 for C; chan- nels. Layer #1 starts from a constant input (0)) E R2x44 Here, we focus on pyramidal decoders with monotonically increasing resolution and decreasing number of channels, but any deep decoder would be applicable. The Adaln-based architecture allows the output of layer #i to be not solely determined by the input from layer #i 1, enabling finer control over the image information, image mixing schemes (Sec. 3.2 and utilization of scale-specific invariances in training data (Sec. 3.3
3.2 STRONG CONSERVATION OF CYCLIC INFORMATION: LAYER-SPECIPE Loss METRICS
In training of regular AGE and PIONEER models, the encoder & and the decoder 0 are trained at separate steps, optimizing either the loss L& Lo, correspondingly:
L. = DKL (z [ æ) [], D) D (gs (t) N(0 D)] + Xx (()()))))
Lo = DKL (s () (O, I)] + >z deas (z, ((0())))
3

---

Under review as a conference paper at ICLR 2020
Easoding
Enceading
Encoding
Encoding
(
(0)
(0)
(0)
Decoder
Decoder.
Decodder
Decoder. 0
Eueodde
Eitooder
Training progresses
The model grows step-wise during training; the resolution doubles on every step. Input is encoded into a latent enocoding z (a dimensionality of 512 used throughout this paper). The decoder acts by modulating an empty canvas ((0) by the latent encoding and produces the output ă.
Figure 2:
where æ: is sampled from the training set, ~ (0 22) 2 ~ N(O, I), dx is L1 or 1.2 distance, and deos is cosine distance. Since the model allows sampling from latent space (similarly to VAEs and GANs), the KL divergence terms during training can be calculated from empirical distributions, despite the model itself being fully deterministic. Yet we retain, in principle, the full information about the image, at every stage of the processing. For an image encoded into latent vector x, decoded back to image space as it, and re-encoded as latent vector z', it is possible and desirable to require that z is as close to z' as possible, yielding the latent reconstruction enror deas (0))))))
Layer-specific Loss We now show how to encourage the latent space to become hierarchically disentangled with respect the levels of image detail, allowing one to separately retrieve 'coarse' and 'fine' aspects of a latent code. This allows for egg. conditional sampling by fixing the latent code for specific layers of the decoder, or mixing the scale-specific features of two or more input images-impossible feats for traditional autoencoder with mutually entangled decoder layers.
The latent reconstruction error as such can trivially generalized to any layer of O. We simply pick the layer of measurement, and from there, pass the sample once through full encoder-decoder cycle. But now, in the automodulator, latent codes can be introduced on a per-layer basis, enabling more powerful reconstruction measurements. (Without loss of generality, here only consider mixtures of codes.) We can present a decoder (Fig. 3b) with N layers, split after the j"h one, as a composition of 0 (0.. (g() 2A), æB). Crucially, we can choose ZA * 2B (extending the method of Karras et al. 2019), such ZA (xxxA and zB Φ(xB) for (image) inputs æ # ZB Because the earlier layers #1:j operate on image content at lower resolutions, the output fusion image mixes the 'coarse' features of 2A with 'fine' features of zg. Now, holds feature information at different levels of detail, some of which are mutually independent. Hence, when re-encoding an image,
should keep the representation of those levels disentangled in z, even if they originate from separate source images. Hence, when we re-encode the fusion image into AAR, and decode once more, the output of (E: ABB) should be unaffected by žB-
This leads to the following conjecture. Assume that the described network reconstructs input sam- ples perfectly, i.e. x = ((((x)). Any zA and zB can be mixed, decoded and re-encoded as
AAB (z x) Q0 (æ) g()), zB) Qol.; (E(3)
(0) =A)-
Now, between {()) of the first and g(i> of the second pass (see Fig. 3c), the mutual information is
Igaa,, (t) (0) AAB); gen., (g()) (0) 2A)]
(4)
With EA, EB ~ N(O, I), for each j. we can now maximize by minimizing
C, d d(0 (0) 2AB), 01. xA))
for some distance function d (here, L2 norm). In other words, the fusion image can be encoded into new latent vector & AB in such a way that, at each layer, the decoder will treat the new code similarly to whichever of the original two separate latent codes was originally used there (see Fig. Bb). For perfect network, C, can be viewed as a layer entanglement error. Randomizing j during the training,
measure L, for any layers of the decoder.
4

---

Under review as a conference paper at ICLR 2020
Latent encading
Camas visiable (0)
(0
(0)
Donolas
Decolas
Decooder
8X8
1
16x 16
20
20
IFAA
MAR
(a) Architecture
(b) 1-pass flow
(c) 2-pass flow
Figure 3: (a) The autoencoder-like usage of the model. (b) Modulations in the decoder can come from different latent vectors. This can be leveraged in feature/style mixing, conditional sampling. and during the model training (first pass). (c) The second pass during training.
Our results improved by using robust im- age reconstruction loss dp of Barron (019) instead of LI/L2. dp generalizes various norms and exposes robustness as an explicit continuous parameter vector a. The complete loss functions are
Final Unsupervised Automodulator Training Loss
L. = max Map: (RI ( 2)) N(0, r)] (2 ()) [(), + Ax dp(x,, (((())))
C. = DKI. 2) [ I)] + Az deos (z, ((0(z))) Lj.
where á 1. ~ ( [ 2) with z ~ N(O, I), and 2 M.M ~ qa(x: (AB). with a set 3:4 ratio of reg- ular and mixed samples of batch size M and Ú1L NN. Margin Mgap 0.5 in low resolutions, then 0.2 for 128² and 0.4 for 256 (see Heljakka et al. 2019). To avoid discontinuities in a, we uti- lize seerressivelyggroiing variation of dp. where we first learn the the lower resolutions (e.g., 4)4). Each a corresponds to one pixel. Then, when switching to the higher resolution stage, we take take each parameter a; that corresponds to pixels Pa, in the lower resolution, to initialize the x4 aj that, in the higher resolution, corresponds to Pz,y, Px+1,y Pa,++ and Pz+1.++1. respectively.
3.3 ENFORCING KNOWN INVARIANCES AT SPECIFIC LAYERS
The architecture and the cyclic training method also allow for novel principled approach to leverage known scale-specific invariances in training data. Assume that images I1 and share certain identical characteristics at some scales, but differ on others, with this information further encoded into 21 and 22, correspondingly. In the automodulator, we could expect to try to represent this information in decoder layers #j:k. Following the reasoning of the previous section, we then must have 0f::( (ii-1 22) = (f (g z2). Let assume that is possible to represent the rest of the information in the images of that dataset in layers #1:(j 1) and #(K + 1):N. This situation occurs, e.g., when two images are known to differ only in high-frequency properties, fully representable in the 'fine' layers. Utilizing the independence of the layers, our goal is to have
(O, (015- 22), 21), 22) Ox 1:N (O, (11j- 22), 22), 22)
11.N (g(0), x2) = 0 (()(x)))
which turns into the optimization target (for some distance function
(0((((2)) 1x+1 (0 (0.- (5 , 21), z2))
(8)
By construction of & and 0, we immediately see this is equivalent to directly minimizing
Cinv = d( d( 1
z1),,
(9)
whose complement term L you can construct by swapping with 22 and Fi with x2. For each known invariant pair æ1 and 22 of the minibatch, you can now add the terms Liv + L to Ls of Eq. (6). Note that in the case of 2 = Σ2, Linv reduces to the regular sample reconstruction loss, revealing formulation generalization thereof.
5

---

Under review as a conference paper at ICLR 2020
Source
Source A
Course foatrres from sntree B.
(b) Generated samples: LSUN Bedrooms
Fine fromn R. Medium feamies from smurae B
2
(a) Facial feature mixing
Generated samples: LSUN Cars
Figure 4: (a) Style-mixing example using the same source images as Karras et al. 2019) (underlining that our model can directly work with real input images). (b-q --c Random samples 256x256.
As we push the invariant information to layers #j.k, and the other information away from those layers, this reduces the number of layers available for the rest of the image information. Thus we may need add extra layers to retain the overall decoder capacity. Note that in pyramidal deconvolutional stack, the layers #j:k can safely span only one or two consecutive levels of detail.
EXPERIMENTS
We run our experiments on images using CELEBA- HQ (Karras et al., 2018). FFHQ (Karras et al.
Table 1: Effect of loss terms, CelebA-HQ.
2019). and LSUN Bedrooms and Cars (Yu et al.
Method 61, steps)
(noema))
100-10kur (mix)
2015.. To quantify the image quality and diversity of random draws from the model at 256x256 reso- lution,
Automodulator architecture
11.54 + 0.32
15.90 0.22
Styll-miisin ving
12.71
0.28
14.88 0.16
Rabust loss (no LI loss)
12:23 = 0.21
14.76 4: 0.14
the Fréchet inception distance (FID,
Heusel et al. 2017). which is comparable across models when sample size is fixed (Binkowski et al. 2018). However, FID is known to remain constant under changing precision-recall characteristics Kynkäänniemi et al. 2019). We use LPIPS (Zhang et al. 2018) as the similarity metric, which has better correspondence to human evaluation than traditional L2 metries. The degree of latent space disentanglement is often considered the most important property of latent variable model. Qualitatively, is the necessary condition for, e.g., style mixing capabilities. Quantitatively, Karras al. (2019) noted that, for constant-length step in the latent space, more entangled model will produce larger overall perceptual change than a less entangled model. The extent of this change, with LPIPS the perceptual space metric, is the basis of measuring disentanglement Perceptual Path Length (PPL).
Baseline Methods For autoencoders, we compare to Balanced PIONEER by Heljakka et al. (2019) vanilla VAE, and more recent Wasserstein Autoencoder (WAE, Tolstikhin al.) (017). For non- autoencoders, we compare to GLOW (Kingma Dhariwal 2018) and two recent GAN models: StyleGAN and Progressively Growing GAN (PGGAN, Karras al. 2018).

---

Under review as a conference paper at ICLR 2020
Table 2: Performance in CELEBA-HQ (CAHQ), FFHQ, and LSUN Bedrooms and Cars. We measure LPIPS, Fréchet Inception Distance (FID), and perceptual path length (PPL). Resolution 256x256, except *128x125 For all numbers, smaller is better.
(a) Encoder-decoder comparison
(b) Generative models comparison
[Using CAHQ")
LPIPS (cropped)
FID
PPL
FID (CAHQ)
FID (FHHQ)
HD (Bedroomss
HID (Cars)
PPL (CAHQ")
PPL
(HFHQ)
R-PIONEER
0.092
21.51
92.84
AAE-Adala
0.165
100.02
62.17
StylGANN
5.17
4.40
2.65
3.27
50.08
195.9
WAE-clasic
236.82
8:04
VAF-Addin
0.267
114.52
83.82
68.83
138.21
VAE-classic
0.291
173.37
71.75
B PIONEER
25.25
21.52
42.81
92.61
Automoduhatte
0.102
36.19
41.45
Automodulatee
51.96
64.07
35.74
35.61
41.45
262.3
Ablation We illustrate the contribution of the style-mixing loss L, and the robust loss dp on 64 x64 CELEBA-HQ (Table TT. Lj increases the FID regular random samples, but reduces it when two random latents are mixed. d, then reduces some of the FID of regular samples.
4.1 ENCODING, DECODING, AND RANDOM SAMPLING
In Table 2a we compare encoder-decoder performance of the autoencoders on 128 128 CELEBA- HQ, with our proposed architecture ('AdaIn') )) and the corresponding regular architecture ('classic'), ). We measure LPIPS, FID and PPL. Our method has the best PPL, while Balanced PIONEER has the best FID (50k batch of generated samples compared training samples).
Table shows comparison of random sampling (examples in Figs, 4b 4c) performance via FID and latent space smoothness via PPL for 256x256 images on CELEBA HQ FFHQ and LSUN Cars and Bedrooms. The performance of the automodulator is comparable to the Balanced PIONEER on most data sets, but the FID is worse in general. Models that only generate random samples have clearly best FID results all data sets (NB: hyper-parametee search with various schemes was used in Karras et al., 2019, to achieve their PPL for FFHQ). We train on the actual 60k training set of FFHQ only (StyleGAN trained on all 70k images). We also evaluate the 4-w image interpolation capabilities in unseen FFHQ test images (Fig. 12 in the appendix) and observe smooth transitions. We emphasize that in GANs, such interpolations are often made between the codes of generated samples. As such, they cannot tell much about the recall characteristics of those models.
4.2 STYLE MIXING
We demonstrate the style-mixing capabilities of our model. For comparison with prior work, we use the source images from the StyleGAN paper (Karras et al. 2019). In Fig. 4a we mix specific input faces so that the 'coarse' (latent resolutions 4 4x4 8x8), 'intermediate" (16x16 - -22x32) or 'fine' (64x64- 256 256) layers of the decoder use one input, and the rest of the layers use the other. Importantly, StyleGAN cannot take real inputs, so it can only mix between random images created by the model itself. For new input images, one must determine the most fitting latent code. For our model, those images appear as completely new test images. Additional style mixing results are included in Figs. 13 14 in the appendix.
separate costly optimization process to
4.3 LEVERAGING INVARIANCES IN A WEAKLY SUPERVISED SETUP
We now consider the cases where there is knowledge of some scale-specific invariances within the training data. Here, we demonstrate a proof-of-concept experiment that uses the simplest image transformation possible: horizontal flipping. For the CELEBA-HQ face data, this provides us with pairs of images that share every other property except the azimuth rotation angle of the face, making the face identity invariant amongst each pair. Since the original rotation of faces in the set varies, the tip-auumented data set contains faces rotated across a wide continuum of angles. For further simplicity, we make artificially strong hypothesis that the 2D projected face shape is the only relevant feature at 4 scale, and does not need to affect scales finer than 8x8. This lets us enforce the Linv loss for layers #1-2. Since we do not want to restrict the scale 8x8 for the shape features alone, we add an extra 8x8 layer after layer #2 of the regular stack, so that layers #2-3 both operate 8x8, layer #4 only at x6x 16116, etc. Now, with 22 that corresponds to the horizontally flipped counterpart of 21, have 0 (g(2( 21) Ox.N (2) 22). Our choices amount j 3, k N.
7

---

Under review as a conference paper at ICLR 2020
Source B
Enforced invariance
Souree A
Regular
Enforced id.
invariance
Sourre
Regular
Figure 5: The effect of training with face identity invariance enforcement under azimuth rotation. We generate images with the 'non-coarse' styles from the source images (left), and the 'coarse' styles from each top image, in turn. With 'Enforced id. invariance', the top row only drives the face pose while conserving identity. In comparison, the 'Regular' training lets the top row also affect other characteristics, including identity. Here, the actual data that defined the invariance was too simple to allow learning fine-grained control of the hair at coarse levels, making it blurry.
allowing us to drop the outermost part of Eq.
Hence, our additional encoder loss terms are
Linv 2( 03.N ( z2), #1)) and
(10)
S d(x 0AHN (g (0) 21), 22))
(11)
Fig. 5ishows the results after training with the new loss (50% of the training samples flipped in each minibatch). With the invariance enforcement, the model forces decoder layers #1-2 to only affect the pose. We generate images by driving those layers with faces at different poses, while modulating the rest of the layers with the face whose identity we seck to conserve. We reccive variations of the original face that only differ in terms of the pose, unlike in regular automodulator training.
DISCUSSION AND CONCLUSION
In this paper, proposed the first generative autoencoder model with hierarchical latent repre- sentation that supports controllable image generation and editing, conditional image sampling by fixing styles of specific layers, and style-mixing of real images. In model, the latent vector in- dependently modulates each decoder layer. The model outperforms other generative autoencoders in terms of latent space disentanglement and matches them faithfulness of reconstructions, with slight reduction of output sample quality. We use the term automodulator to denote any autoencoder that uses the latent code only to modulate the statistical properties of the information that flows through the layers of the decoder. Such decoders could also include, e.g. 3D or graph convolutions.
Potential future applications include introducing completely independent 'plugin' layers or modules in the decoder, trained afterwards on top of the pretrained base automodulator, leveraging the mu- tual independence of the layers. The affine maps themselves could also be re-used across domains, potentially offering mixing of different domains. Such examples highlight that the range of applica- tions of our model far wider than the initial ones shown here, making the family of automodulators viable alternative ffate-of-tee art autoencoders and GANs. Upon acceptance for publication, our source code will be released at httt:://githubb.. wttp:///oo//oooocmc///.a
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2StyleGAN: How to embed images into the StyleGAN latent space? In International Conference on Computer Vision (ICCV), 2019.
Jonathan T. Barron. A general and adaptive robust loss function. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recogmition (CVPR), pp. 4331-4339, 2019.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations (ICLR), 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
Ting Chen, Mario Lucie, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adver- sarial networks. In International Conference on Learning Representations (ICLR), 2019.
Antonia Creswell and Anil Anthony Bharath. Inverting the generator of a generative adversarial network. IEEE Transactions Neural Networks and Learning Systems, 30(7):167-19774, 2019.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Jeff Donahue, Philipp Krähenbühhl, and Trevor Darrell. Adversarial feature learning. In International Conference on Learning Representations (ICLR), 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially learned inference. In International Conference on Learning Representations (ICLR), 2017a.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In International Conference on Learning Representations (ICLR), 2017b.
Leon A. Gatys, Alexander Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings the IEEF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 2414-2423, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems (NIPS), volume 27, pp. 2672-2680. Curran Associates, Inc., 2014.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Rezende, and Daan Wierstra. DRAW: A recurrent neural network for image generation. In Proceedings of the 32nd International Conference on Machine Learning (ICML), volume 37 of PMLR, pp. 1462-1471, 2015.
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. To- wards conceptual compression. In Advances in Neural Information Processing Systems (NIPS), volume 29, pp. 3557- Curran Associates, Inc., 2016.
Ari Heljakka, Arno Solin, and Juho Kannala. Pioneer networks: Progressively growing generative autoencoder. In Asian Conference on Computer Vision (ACCV), pp. 22-38, 2018.
Ari Heljakka, Arno Solin, and Juho Kannala. Towards photographic image manipulation with bal- anced growing of generative autoencoders. arXiv preprint 1904vv1904.0611 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems (NIPS), volume 30, pp. 6626-6637. Curran Associates, Inc., 2017.
Geoffrey E. Hinton and James McClelland. Learning representations by recirculation. In Neural Information Processing Systems (NIPS), pp. 358-366. American Institute of Physics, 1988.

---

Under review as a conference paper at ICLR 2020
Huaibo Huang, Zhihang Li, Ran He, Zhenan Sun, and Tieniu Tan. IntroVAE: Introspective varia- tional autoencoders for photographic image synthesis. In Neural Information Processing Systems (NeurIPS), volume 31, pp. 52 63. Curran Associates, Inc., 2018.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pp- 1501-1510, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations (ICLR), 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEÉE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4401-4410, 2019.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Confer- ence on Learning Representations (ICLR), 2014.
Durk P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, pp. 10236-10245. Curran Associates, Inc., 2018
Tuomas Kynkääänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Alireza Makhzani. Implicit autoencoders. arXiv preprint arXiv:1805.0900 09804, 2018.
Alireza Makhzani. Jonathon Shlens, Navdeep Jaitly, and Ian Goodfellow. Adversarial autoencoders. In International Conference on Learning Representations (ICLR), 2016.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational Bayes: Unifying
variational autoencoders and generative adversarial networks. In Proceedings of the 34th Inter- national Conference on Machine Learning (ICML), volume 70 of PMLR, pp. 2391-2400, 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQE-2 arXiv preprint urXiv:I90e 00446, 2019.
Danilo J. Rezende, Shakir Mohamed, Ivo Danihelka, Karol Gregor, and Daan Wierstra. One-shot generalization in deep generative models. In Proceedings of the 33rd International Conference on International Conference Machine Learning (ICML), volume 48 of PMLR, pp. 1521-1529, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. Proceedings of the 31 International Con- ference Machine Learning (ICML), volume 32 of PMLR, pp.
2014.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. Wasserstein auto- encoders. arXiv preprint arXiv:1711.0115: 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. It takes (only) two: Adversarial generator- encoder networks. In Proceedings the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), pp. 1250-1257, 2018.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 750 0006003665 2015.
10

---

Under review as a conference paper at ICLR 2020
Amir Roshan Zamir, Te-Lin Wu, Lin Sun, William B. Shen, Bertram E. Shi, Jitendra Malik, and Silvio Savarese. Feedback networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1888-18177 2017.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 586-595, 2018.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
In the appendix, we include further details on training and complement the results in the main paper with examples of random samples, reconstruction, latent space interpolations, and style mixing.
A.I TRAINING DETAILS
The training method follows largely those in Heljakka et al. (2019), with progressively growing symmetric encoder and decoder, and decreasing the batch size when moving to higher resolutions. The encoder and decoder consist of 7 blocks each containting two residual blocks with 3x3 filter. In the encoder, these are followed by a spectral normalization operation (Miyato et al., 2018) and binomial filtering. In the decoder, by AdaIn normalization and binomial filtering. leaky ReLU (p 0.2) is used as activation. In the encoder, each block halves the resolution of the convolution map, while in the decoder, each block doubles it. The output of the final encoder layers is flattened into 512-dimensional latent block. As in Karras et al. 2019), the block is mapped by affine mapping layers so that each convolutional layer C' in the decoder block is preceded by its own fully connected layer that maps the latent to two vectors each of length N, when N equals the number of channels in C.
learning rate 0.0005 and thereafter 0.001. Optimization is done with ADAM (31 = 0, 32 = 0.99, c = 10-8). After the first two resolution steps, the KL margin is turned on and fixed 0.5. The length of training phases amounts to 2.4M training samples until 64 x66 resolution phase, which lasts for 10.4M samples. For FFHQ and CelebAHQ, the 128x128 phase uses 13.0M samples while LSUN Bedrooms and Cars use 10.0M samples. The final 256x256 phase uses 7-10M samples for each data set. The training of the final stage was generally cut off when reasonable FID results had been obtained. More training and learning rate optimization would likely improve results. With two Titan V100 GPUs for pre- training stages and four GPUs for the 256x256 stage, the training time for CELEBA-HQ and FFHQ were 18 days each, and for LSUN Bedfrooms 17 days and Cars 19 days.
Each resolution phase until 32x32 for all data sets
For evaluating the model after training, a moving expontential running average of generator weights was used, as in both Karras et al. (2018) and Heljakka et al. (2019). For all data sets, training/test set splits were used as given or defined by data set authors, except for LSUN Cars, where we used 4,968,695 samples for training and 552,061 for testing. Note that in regular GAN training, complete data sets are often used without train/test split, leading them to use larger training sets.
For baselines, we used pre-trained models for StyleGAN, PGGAN, PIONEER, and GLOW with default settings provided by the authors. We trained the VAE and WAE models manually. For all VAE baselines the weight for KLD loss was 0.005. For all WAE baseline, we used the WAE-MMD algorithm. The weights for the MMD loss with automodular architecture (WAE-AdaIn) was 4 and with Balanced PIONEER (WAB-classic) architecure it was 2. For VAES, the learning rate for the encoder was 0.0001, and for the generator 0.0005. For WAEs. the learning rate for both was 0.0002.
For evaluating the encoding and decoding performance, we used 10k unseen test images from the FFHQ data set, cropped the input and reconstruction to 128x 128 as in Karras et al. (2019) and evaluated the LPIPS distance between the inputs and reconstructions. We evaluated 50k random samples in all data sets and compare against the provided training set. The GLOW model has not been shown to work with 256 x 256 resolution on LSUN (the authors show qualitative result only for 128x 128). Training of PIONEER did not converge on FFHQ. however we believe this is an issue with the default
not suitable for FFHQ.
For Perceptual Path Length (PPL), repeatedly take random vector of length 10 in the latent space, generate images at its endpoints, crop them around mid-face to 128x 128 or 64x64, and measure the LPIPS between them (Karras et al. 2019). PPL equals the scaled expectation of this value (for a sample of 100k vectors).
Pre-trained models for PGGAN and GLOW were used with default settings provided by the authors. Note that we train on the actual 60k training images of FFHQ only (unlike StyleGAN that trained on all 70k images).
12

---

Under review as a conference paper at ICLR 2020
A.2 RANDOM SAMPLES
Our model is capable of fully random sampling by specifying N(O, I) to be draws from unit Gaussian. Fig. / show samples from an automodulator trained with the FFHQ/CELEBA- HQ/LSUN data sets up to resolution x56x256. The samples here indicate the full range of samples and face features the model can support.
Figure 6: Random samples from the automodulator trained on FFHQ at a resolution x56x256.
Figure 7: Random samples for automodulator trained CELEBA-HQ resolution 256x256.
(a) LSUN Bedrooms
(b) LSUN Cars
Figure 8: Additional samples from an automodulator trained on LSUN Bedrooms and Cars reso-
lution of at 256.
A.3 RECONSTRUCTIONS
We include examples of the reconstruction capabilities of the automodulator at 256 x 256 in for uncurated test set samples from the FFHQ and CELEBA-HO data sets. These examples are provided in Figs. 9 10
13

---

Under review as a conference paper at ICLR 2020
Figure 9: Uncurated examples of reconstruction quality in 256x256 resolution with images from the FFHQ test set (top row).
Figure 10: Uncurated examples of reconstruction quality 256 resolution with images from the CELEBA-HO -HQ test set (top row).
A.4 CONDITIONAL SAMPLING
The automodulator directly allows for conditional sampling the sense of fixing a latent encoding za. but allowing some of the modulations come from random encoding zB N(0, 1). In Fig. II we show conditional sampling of 128x 128 random face images based on 'coarse' (latent resolutions 4x4-8x8) and 'intermediate" (16 x 16 32 x2)) latent features of the fixed input. The input image controls the coarse features (such as head shape, pose, gender) on the top and more fine features (expressions, accessories, eyebrows) on the bottom.
A.5 STYLE MIXING AND INTERPOLATION
The well disentangled latent space allows for interpolations between encoded images. We show regular latent space interpolations between the reconstructions of new input images (Fig. D2p.
As two more systematic style mixing examples, we include style mixing results based on both FFHQ and LSUN Cars. The source images are unseen real test images, not self-generated images. In Figs. 13 and 14 we show matrix of cross-mixing either 'coarse' (latent resolutions 4x4- 8x8) or "intermediate" (16x 16 - 32 x2) latent features. Mixing coarse features results in large-scale changes, such pose, while the intermediate features drive finer details, such as color.
14

---

Under review as a conference paper at ICLR 2020
Input
Samples with inpul
Input
Samples with intermediate features from input
Conditional sampling of 128 128 random face images based on 'coarse' (latent reso- lutions 4x4 - 8x8) and 'intermediate' (16x 16 x2x32) latent features of the fixed input. The input image controls the coarse features (such as head shape, pose, gender) on the top and more fine features (expressions, accessories, eyebrows) on the bottom.
Figure 11:
Figure 12: Interpolation between random test set CELEBA-HO images in x28x 128 (in the corners) which the model has not seen during training. The model captures most of the salient features in the reconstructions and produces smooth interpolations at all points in the traversed space.
15

---

Under review as a conference paper at ICLR 2020
Source
Source A
(a) Using 'coarse" (latent resolutions 4x4 8x8) latent features and rest from
Source
Source
(b) Using the 'intermediate' (16x 16 32x32) latent features from B and the rest from A.
Figure 13: Style mixing of FFHQ face images. The source images are unseen real test images, not self-generated images.
16

---

Under review as a conference paper at ICLR 2020
Source
Source
(a) Using 'coarse" (latent resolutions 4x4 8x8) latent features and rest from A. Most notably, the cars drive the car pose.
Source
Source
(b) Using the 'intermediate' (16x 16 32x32) latent features from B and the rest from A.
Figure 14: Style mixing of LSUN Cars. The source images are unseen real test images, not self- generated images.
17


Reviews:
[{"review_id": "rkgGbTwr5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Deep Automodulators introduces a generative autoencoder architecture that replaces the canonical encoder decoder autoencoder architecture with one inspired by StyleGAN. The encoder interacts with the decoder by modulating layer statistics via Adaptive Instance Normalization (AdaIN) conditioned on the latent. The paper trains this architecture with the loss framework of the Adversarial Generator–Encoder (AGE) and utilizes the progressive growing trick originally introduced in Progressive GAN which is also adapted by the Pioneer models, recent followups to AGE.\n\nThe use of AdaIN conditioning across multiple layers and multiple scales (like StyleGAN) and the ability to directly compute latent codes via the encoder allows the authors introduce a disentanglement objective L_j and also an invariance objective L_inv to help encourage these properties in the models via consistency objectives \n\nThe paper shows results demonstrating StyleGAN style coarse/fine visual transfer on two high quality face datasets (importantly this is demonstrated on real inputs rather than samples as in StyleGAN) as well as respectable sample quality on LSUN Bedrooms and the LSUN Cars dataset.\n\nMy decision is weak reject. Overall, I think the paper is promising and shows a nice combination of efficient latent inference and controllable generation but the authors do not include ablations to validate some of their core contributions such as the L_j objective. Additionally, the improved controllability of the approach seems to unfortunately result in lower reconstruction quality than direct prior work such as Balanced Pioneer and this potential tradeoff is not investigated/discussed.\n\nTo expand a bit, there are three changes from that prior work that that stood out to me. 1) The StyleGAN inspired architecture 2) the disentangling objective L_j and 3) using the loss function dρ of Barron 2019. Successful ablations to demonstrate the importance of 2) to the presented results as well as better motivating / demonstrating the impact of including 3) would raise my score to an weak acceptance.\n\nMy other concern is that the reconstruction quality seems noticeably lower than that of the proceeding work, Balanced Pioneer. This is reflected in its 10% reduction in LPIPS compared to the Automodulator’s paper. In general there also seems to be noticeable grid artifacts in the samples across all datasets samples/reconstructions, which don’t seem as prominent in Balanced Pioneer. It is not immediately clear why this is the case and additional investigation of this, such as checking whether this is due to the introduction of the disentanglement objective, or the inclusion of the Barron 2019 loss function would be informative.\n\nAdditional Comments:\n\nEach subsection of 3 could be improved by providing a brief introduction to the motivation for and aim of each contribution before launching directly into how it is implemented / achieved. Without that bit of context on the goals of each subsection, it was more difficult to follow along with what was being done and why.\n\nThe presentation of L_j with lots of inlined equations intermixed with text gets a bit difficult to read / follow.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJxBHBNP5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper makes the following contribution:\n- using the AdaIn architecture proposed by Karras et al., 2019 with the autoencoding architecture of AGE/PIONEER;\n- a cyclic loss to enforce disentangling between different layers;\n- a method to enforce invariances at specific layers.\n\nThe adaptation of the AdaIn architecture in an autoencoding fashion (a la AGE/PIONEER) is sensible and well motivated, combining state-of-the-art generator while allowing inference in a compact setting (i.e. not requiring an additional discriminator). \n\nThe other contribution are harder to read and the writing should be improved.\nThe cyclic loss should be better described. The notation of the KL divergence is confusing if you are using the KL divergence defined by AGE/PIONEER and will need to be explained. I will also assume that d_cos is the cosine loss as defined by the PIONEER paper. This should be mentioned as well.\nThe method to enforce invariance is also not clear to me. While the authors introduce F as a \"known invariance\", it is unclear what role it plays in the cost function. Is F an invariant on which we measure this reconstruction loss d? What is d? Explaining that might shed light on the result Figure 5, e.g. why the images become blurry when doing this rotation. \n\nThe experiment demonstrates the sampling quality of the model and the transfer of features at different level (coarse-medium-fine) Figure 4. It is unclear what was the contribution of the layer specific loss metric to allow that feature transfer. It seems from Figure 5 the invariance objective has been roughly satisfied but at the cost of a significant drop in image quality.\n\nThe clearest contribution from this paper is definitely the AGE/PIONEER approach to train the AdaIn architecture. The two other contributions are unclear, both in their explanation and in what they contribute: the layer specific loss not compared to an architecture just trained in an AGE way, and the enforcing of invariance, although filling its objective, might deteriorate other desirable properties of the model (e.g. sample quality). ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkxmXW7AYS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes an autoencoder architecture which combines two recent GAN-based architectural innovations, namely the progressive growing of the decoder architecture (as well as the encoder architecture in this case) and the use of the encoded representation to modulate the decoder via a feature-wise transformation mechanism.\n\nI think the overall idea behind the paper is valuable, but I don’t think the submission meets the acceptance bar from a clarity point of view. I also have concerns with its characterization of the literature.\n\nClarity-related comments:\n\n- “This allows the layers to work independently of each other.” This is an imprecise use of the term “independently”. How can two layers work independently if one’s input is the output of the other?\n- “The reconstructed samples could be re-introduced to the encoder, repeating the process, and requiring consistency between passes.” The rest of the paragraph is built on the premise that this is a desirable property, but I’m not sure I understand why this is a desirable property in the first place.\n- How to define “disentanglement” in the context of representation learning is in itself an unsettled question as far as I’m aware, but the submission uses the term without an intuitive or formal definition. What do the authors mean by “disentangled representation”? What is measured by perceptual path length (PPL), and in which ways does PPL relate to the author’s definition of “disentangled representation”?\n- “[...] a new autoencoder-like model with powerful properties not found in regular autoencoders, including style transfer.” The term “style transfer” is overloaded; what do the authors mean?\n- The submission defines AdaIn as a way to combine “content” and “style”, and defines the style “y” in terms of mean and variance. In the context of the AdaIn paper, this makes sense: the instance normalization shifting and scaling coefficients are heuristically defined as the channel-wise means and standard deviations of a “style” stack of feature maps. However, in the context of this submission I’m not sure this definition makes as much sense: the instance normalization shifting and scaling coefficients are the result of a linear projection of the latent representation and do not involve the channel-wise means and standard deviations of an external stack of feature maps; is this correct?\n- “This setup follows the same logic as that of Karras et al. (2019), but we do not require an ad-hoc disentanglement stack.” Can the authors clarify what they mean by an “ad-hoc disentanglement stack”?\n- Section 3.2 uses some notation for the encoder without introducing it first. I believe the only way to understand that \\phi(x) refers to the encoder network is to look at Figure 2.\n- Section 3.2 as a whole is hard to follow, in part due to the use of imprecise language (“mutually independent”, “representation of those levels disentangled in z”). At some point probability distributions are introduced (up until now the reader is operating under the assumption that the model is an autoencoder with no probabilistic interpretation), and mutual information is mentioned to justify an L2 reconstruction loss in z-space (which I would argue is an instance of mathiness that does not serve the reader’s comprehension). Can the authors explain in plain language how layer-specific losses are defined and how the complete loss is obtained?\n- The submission presents model samples, but as far as I can tell the procedure for sampling is not provided. Unlike VAEs, autoencoders do not explicitly model the empirical distribution -- although reconstruction in denoising autoencoders is related to the score of the empirical distribution (Alain & Bengio, 2014). How are samples obtained from the trained model?\n\nLiterature-related comments:\n\n- The use of normalization layers to implement feature-wise transformation mechanisms is fairly widespread nowadays, but for instance normalization specifically the work of Dumoulin et al. (2017) pre-dates that of Huang & Belongie (2017). Both are cited by Karras et al. (2019) in relation to AdaIn (which is termed “conditional instance normalization” in Dumoulin et al. (2017)).\n- I disagree with the characterization of ALI/BiGAN as “hybrid models that combine the properties of VAEs and GANs”: unlike AAE and AVB, which minimize KL-divergence terms in the VAE loss adversarially, the objective for ALI/BiGAN is purely adversarial. I would also include IAE (Makhzani et al., 2018) and BigBiGAN (Donahue et al., 2019) in the list of GAN variants that incorporate an inference mechanism.\n- The submission repeatedly asserts that GANs lack an inference mechanism: “Unlike GANs, autoencoder models can directly operate on input samples.”; “To work on new input images, GANs either need to be extended with a separate encoder, or inverted [...].”; “[...] GANs show good image quality, but have no built-in encoding mechanism [...]”; “[...] the problem with GANs is that they lack the encoder [...]”. This is false: see for example ALI, BiGAN, and BigBiGAN. The problem in my opinion is elsewhere: the kinds of reconstructions these models yield are not suited to the downstream applications investigated in this submission, because they oftentimes fail to preserve low-level details.\n\nReferences:\n\n- Alain, G., & Bengio, Y. (2014). What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1), 3563-3593.\n- Dumoulin, V., Shlens, J., & Kudlur, M. (2017). A learned representation for artistic style. In Proceedings of the International Conference on Learning Representations.\n- Makhzani, A. (2018). Implicit autoencoders. arXiv:1805.09804.\n- Donahue, J., & Simonyan, K. (2019). Large scale adversarial representation learning. arXiv:1907.02544.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript proposes an autoencoder architecture incorporating two recent architectural innovations from the GAN literature (progressive growing & feature-wise modulation), trained with the adversarial generator-encoder paradigm with a novel cyclic loss meant to encourage disentangling, and procedure for enforcing layerwise invariances. The authors demonstrate coarse/fine visual transfer on generative modeling of face images, as well as generative modeling results on several Large Scale Scene Understanding (LSUN) datasets. 

Reviewers generally found the results somewhat compelling and the ideas valuable and well-motivated, but criticized the presentation clarity, lack of ablation studies, and that the claims made were not sufficiently supported by the empirical evidence. The authors revised, and while it was agreed that clarity was improved, some reviewers were still not satisfied with the level of clarity (the revision appeared at the very end of the discussion period, unfortunately not allowing for any further refinement). Ablation studies were added in the revised manuscript, which were appreciated, but seemed to suggest that the proposed loss function was of mixed utility: while style-mixing quantitatively improved, overall sample quality appeared to suffer.

As the reviewers remain unconvinced as to the significance of the contribution and the clarity of its presentation, I recommend rejection at this time, while encouraging the authors to further refine the presentation of their ideas for a future resubmission.
Decision: Reject