Title: No title found

Abstract: We introduce a novel autoencoder model that deviates from traditional autoencoders by using the full latent vector to independently modulate each layer in the decoder. We demonstrate how such an 'automodulator' allows for a principled approach to enforce latent space disentanglement, mixing of latent codes, and a straightforward way to utilize prior information that can be construed as a scale-specific invariance. Unlike the GAN models without encoders, autoencoder models can directly operate on new real input samples. This makes our model directly suitable for applications involving real-world inputs. As the architectural backbone, we extend recent generative autoencoder models that retain input identity and image sharpness at high resolutions better than VAEs. We show that our model achieves state-of-the-art latent space disentanglement and achieves high quality and diversity of output samples, as well as faithfulness of reconstructions.

1. INTRODUCTION:

This paper introduces a new generative autoencoder for learning representations of image data sets, in a way that allows arbitrary combinations of latent codes to generate images (see Fig. 1 ). We achieve this with an architecture that uses adaptive instance normalization (AdaIn, Dumoulin et al., 2017b; Huang & Belongie, 2017) , and training methods that let the model learn a highly disentangled latent space by utilizing progressively growing autoencoders (Heljakka et al., 2019) . In a typical autoencoder, input images are encoded into latent space, and the information of the latent variables is then passed through successive layers of decoding until a reconstruction of the input image has been formed. In our model, the latent vector independently modulates the statistics of each layer of the decoder-the output of layer n is no longer solely determined by the input from layer n -1.

In image generation, the probability mass representing sensible images (such as human faces) lies concentrated on a low-dimensional manifold. Even if impressive results have been shown for image generation (e.g., by GANs, Goodfellow et al., 2014; Karras et al., 2019) , efficient reconstruction and manipulation remain open problems. Deep generative autoencoders provide a principled approach for feature extraction, editing, and image synthesis. We show that modulation of decoder layers with AdaIn further improves these capabilities. It also yields representations that are more disentangled, a property here broadly defined as something that allows for fine control of one semantic (image) sample attribute independently of others. Previous works on AdaIn are mostly based on GAN models (Karras et al., 2019; Chen et al., 2019) with no encoder for new input images.

Autoencoders are typically single-pass encoder-decoder structures, where a sample enters from one end and is reconstructed at the other. For robustness, the reconstructed samples could be reintroduced to the encoder, repeating the process, and requiring consistency between the passes. In comparison to stochastic models (like VAEs, Kingma & Welling, 2014; Rezende et al., 2014) , our deterministic model is better suited to take advantage of this. We can mix the latent codes of separate samples and measure the conservation of layer-specific information for each. This enforces disentanglement of layer-specific properties, because we can ensure that the latent code introduced from a certain layer onwards on the 1 st pass should not affect the previous layers on the 2 nd pass. Image autoencoders with convolutional architectures are often used for finding attributes that are disentangled from each other in the latent space. Often, they have relatively poor output image quality, and prior information must be fed in either via class-conditioning or by more complex loss functions for the latent variables. Fully implicit methods avoid this, but without a built-in encoding mechanism often cover only a relatively small fraction of the variation in the training data. For the first time, we show a full autoencoder architecture that has a state-of-the-art disentanglement performance even for unsupervised training setup in several data sets, sharp image output quality, good coverage of the variation of the training data, arbitrary mixing of latent representations and a principled approach for incorporating scale-specific prior information.

Our contributions are as follows. (i) We introduce the automodulator, a new autoencoder-like model with powerful properties not found in regular autoencoders, including scale-specific style transfer (Gatys et al., 2016) . In contrast to architecturally similar 'style'-based GANs, we can now directly encode and manipulate new inputs. (ii) We present methodology that allows fully unsupervised training of such a model to learn rich latent representation in high resolutions, and specifically explain a novel multi-pass training principle that enables the model to learn a more disentangled representation of training data than regular autoencoders. (iii) We demonstrate qualitative and quantitative performance and applications on CELEBA-HQ and LSUN Bedrooms and Cars data sets, taking advantage of the highly disentangled model structure.

2. RELATED WORK:

Our work builds upon several lines of previous work in unsupervised representation learning. The most relevant concepts are variational autoencoders (VAEs, Kingma & Welling, 2014) and generative adversarial networks (GANs, Goodfellow et al., 2014) . In VAEs, an encoder maps data points to a lower dimensional latent space and a decoder maps the latent representations back to the data space. The model is learnt by minimizing the reconstruction error together with a regularization term that encourages the distribution of latents to match a predefined prior. Latent representations often provide useful features for applications (e.g., image analysis and manipulation) and decoding random samples from the prior allows novel data generation. However, in case of images, the generated samples are often blurry and not fully photorealistic, with imperfect reconstructions.

Current state-of-the-art in generative image modeling is represented by GAN models (Brock et al., 2019; Karras et al., 2019) which achieve higher image quality than VAE based models. Nevertheless, these GANs lack an encoder for obtaining the latent representation for a given image, limiting their usefulness. In some cases, a given image can be semantically mapped to the latent space via generator inversion but this iterative process is prohibitively slow for many applications, and the result may depend on initialization (Abdal et al., 2019; Creswell & Bharath, 2019) .

Bidirectional mapping has been targeted by VAE-GAN hybrids such as Makhzani et al. (2016) ; Mescheder et al. (2017) ; Makhzani (2018) , and adversarial models such as Donahue et al. (2017); Dumoulin et al. (2017a) ; Donahue & Simonyan (2019) . These models learn mappings between the data space and latent space using combinations of encoders, generators and discriminators. In-troVAE (Huang et al., 2018) and AGE (Ulyanov et al., 2018 ) are more compact autoencoders, which both contain only two deep networks, encoder and decoder, learnt using a combination of reconstruction loss and adversarial loss, which encourages the distributions of real and generated images to match with the prior in latent space. A progressively grown version of AGE, PIONEER, was proposed by Heljakka et al. (2018; 2019) . There is also recent work on vector quantized variational autoencoders (VQ-VAE, Razavi et al., 2019) with a discrete latent space. However, while suitable for image compression, the discrete latent representation cannot e.g. be interpolated and hence may not be optimal for semantic image manipulation and learning a disentangled latent space.

The architecture of our generator and AdaIn utilization was inspired by the recent StyleGAN (Karras et al., 2019) , but has no discriminator (nor the 'mapping network' f ). Instead, our model simply contains the generator (i.e., decoder) and an encoder, jointly trained applying a similar progressive growing strategy as Heljakka et al. (2018) , but with a modified loss formulation. Besides the reconstruction loss and adversarial loss from Ulyanov et al. (2018) and Heljakka et al. (2018) , we can also recirculate style-mixed reconstructions as 'second-pass' training samples in order to better encourage the independence and disentanglement of emerging styles and conservation of layerspecific information. The recirculation idea is biologically motivated and conceptually related to many works, such as that by Hinton & McClelland (1988) . Utilizing the outputs of the model as inputs for the next iteration is related to, e.g., Zamir et al. (2017) , where feedback is shown to benefit image classification, and in RNN-based methods (Rezende et al., 2016; Gregor et al., 2015; 2016) .

3. METHODS:

We now describe the model architecture, starting from regular PIONEER and building up to the automodulator (Sec. 3.1). We then describe the unsupervised training via our extensions to the regular PIONEER loss function, to support flexible mixing of latents (Sec. 3.2). Finally, we introduce an additional weakly supervised training method to utilize known invariances (Sec. 3.3).

3.1. AUTOMODULATOR ARCHITECTURE:

Our interest is in unsupervised training of an autoencoder wherein the inputs x are images fed through an encoder φ to form a low-dimensional latent space representation z (we use z ∈ R 512 , normalized to unity). This representation can then be decoded back into an image x through a decoder θ. For stable training, we adopt the progressively growing architecture of Balanced PIO-NEER (Heljakka et al., 2018; 2019) . The convolutional layers of the symmetric encoder and decoder are faded in gradually during the training, in tandem with the resolutions of training images and generated images (Fig. 2 ).

Adaptive Instance Normalization (AdaIn) A traditional decoder architecture would start from a small-resolution image and expand it layer by layer until the full image is formed, feeding the full information of the latent code via the decoder layers. In contrast, our decoder is composed of layer-wise functions θ i (ξ i-1 , z) that separately take a 'canvas' variable ξ i-1 denoting the content input from the preceding decoder layer (see Figs. 2 and 3a ), and the actual (shared) latent code z. First, each deconvolutional layer #i computes its feature map output χ i from ξ i-1 as in traditional decoders. Second, we take the AdaIn normalization step that uses z to modulate the channel-wise mean µ and variance σ 2 of the output. To do this, we need a map g i : z → (µ i , σ 2 i ), arriving at:

EQUATION

We implement g i for layer #i as a fully connected linear layer, with output size 2 C i for C i channels. Layer #1 starts from a constant input ξ (0) ∈ R 4×4 . Here, we focus on pyramidal decoders with monotonically increasing resolution and decreasing number of channels, but any deep decoder would be applicable. The AdaIn-based architecture allows the output of layer #i to be not solely determined by the input from layer #i -1, enabling finer control over the image information, image mixing schemes (Sec. 3.2), and utilization of scale-specific invariances in training data (Sec. 3.3).

3.2. STRONG CONSERVATION OF CYCLIC INFORMATION: LAYER-SPECIFIC LOSS METRICS:

In training of regular AGE and PIONEER models, the encoder φ and the decoder θ are trained at separate steps, optimizing either the loss L φ or L θ , correspondingly:

EQUATION

z ξ (0) x x 4×4 4×4 Encoder, φ Decoder, θ Encoding z ξ (0) x x 16×16 16×16 Encoder, φ Decoder, θ Encoding z ξ (0) x x 64×64 64×64 Encoder, φ Decoder, θ Encoding z ξ (0) x x 256×256 256×256

Encoder, φ Decoder, θ

Encoding:

Training progresses where x is sampled from the training set, x ∼ q θ (x | z), z ∼ N(0, I), d X is L1 or L2 distance, and d cos is cosine distance. Since the model allows sampling from latent space (similarly to VAEs and GANs), the KL divergence terms during training can be calculated from empirical distributions, despite the model itself being fully deterministic. Yet we retain, in principle, the full information about the image, at every stage of the processing. For an image encoded into a latent vector z, decoded back to image space as x, and re-encoded as latent vector z , it is possible and desirable to require that z is as close to z as possible, yielding the latent reconstruction error d cos (z, φ(θ(z))).

Layer-specific Loss We now show how to encourage the latent space to become hierarchically disentangled with respect to the levels of image detail, allowing one to separately retrieve 'coarse' and 'fine' aspects of a latent code. This allows for e.g. conditional sampling by fixing the latent code for specific layers of the decoder, or mixing the scale-specific features of two or more input images-impossible feats for a traditional autoencoder with mutually entangled decoder layers.

The latent reconstruction error as such can be trivially generalized to any layer of θ. We simply pick the layer of measurement, and from there, pass the sample once through a full encoder-decoder cycle. But now, in the automodulator, latent codes can be introduced on a per-layer basis, enabling more powerful reconstruction measurements. (Without loss of generality, here we only consider mixtures of 2 codes.) We can present a decoder (Fig. 3b ) with N layers, split after the j th one, as a composition of θ j+1:N (θ 1:j (ξ (0) , z A ), z B ). Crucially, we can choose z A = z B (extending the method of Karras et al., 2019) , such as z A = φ(x A ) and z B = φ(x B ) for (image) inputs

x A = x B .

Because the earlier layers #1:j operate on image content at lower resolutions, the output fusion image mixes the 'coarse' features of z A with 'fine' features of z B . Now, z holds feature information at different levels of detail, some of which are mutually independent. Hence, when re-encoding an image, we should keep the representation of those levels disentangled in z, even if they originate from separate source images. Hence, when we re-encode the fusion image into ẑAB , and decode once more, the output of θ 1:j (ξ (0) , ẑAB ) should be unaffected by z B .

This leads to the following conjecture. Assume that the described network reconstructs input samples perfectly, i.e. x = θ(φ(x)). Any z A and z B can be mixed, decoded and re-encoded as

EQUATION

Now, between ξ (j) of the first and ξ (j) of the second pass (see Fig. 3c ), the mutual information is

EQUATION

With z A , z B ∼ N(0, I), for each j, we can now maximize (4) by minimizing

EQUATION

for some distance function d (here, L2 norm). In other words, the fusion image can be encoded into a new latent vector ẑAB in such a way that, at each layer, the decoder will treat the new code similarly to whichever of the original two separate latent codes was originally used there (see Fig. 3b ). For a perfect network, L j can be viewed as a layer entanglement error. Randomizing j during the training, we can measure L j for any layers of the decoder. 

EQUATION

where x1: 3 4 M ∼ q θ (x | z) with z ∼ N(0, I), and x 3 4 M :M ∼ q θ (x | ẑAB ), with a set 3:4 ratio of regular and mixed samples of batch size M and j ∼ U{1, N }. Margin M gap = 0.5 in low resolutions, then 0.2 for 128 2 and 0.4 for 256 2 (see Heljakka et al., 2019) . To avoid discontinuities in α, we utilize a progressively-growing variation of d ρ , where we first learn the α in the lower resolutions (e.g., 4×4). Each α i corresponds to one pixel. Then, when switching to the higher resolution stage, we take take each parameter α i that corresponds to pixels p x,y in the lower resolution, to initialize the α 1×4 j that, in the higher resolution, corresponds to p x,y , p x+1,y , p x,y+1 and p x+1,y+1 , respectively.

3.3. ENFORCING KNOWN INVARIANCES AT SPECIFIC LAYERS:

The architecture and the cyclic training method also allow for a novel principled approach to leverage known scale-specific invariances in training data. Assume that images x 1 and x 2 share certain identical characteristics at some scales, but differ on others, with this information further encoded into z 1 and z 2 , correspondingly. In the automodulator, we could expect to try to represent this information in decoder layers #j:k. Following the reasoning of the previous section, we then must have θ j:k (ξ j-1 , z 1 ) = θ j:k (ξ (j-1) , z 2 ). Let us assume that it is possible to represent the rest of the information in the images of that dataset in layers #1:(j -1) and #(k + 1):N . This situation occurs, e.g., when two images are known to differ only in high-frequency properties, fully representable in the 'fine' layers. Utilizing the independence of the layers, our goal is to have

EQUATION

which turns into the optimization target (for some distance function d)

EQUATION

By construction of φ and θ, we immediately see this is equivalent to directly minimizing

EQUATION

whose complement term L inv you can construct by swapping z 1 with z 2 and x 1 with x 2 . For each known invariant pair x 1 and x 2 of the minibatch, you can now add the terms L inv + L inv to L φ of Eq. ( 6). Note that in the case of z 1 = z 2 , L inv reduces to the regular sample reconstruction loss, revealing our formulation as a generalization thereof. As we push the invariant information to layers #j:k, and the other information away from those layers, this reduces the number of layers available for the rest of the image information. Thus we may need to add extra layers to retain the overall decoder capacity. Note that in a pyramidal deconvolutional stack, the layers #j:k can safely span only one or two consecutive levels of detail. We run our experiments on images using CELEBA-HQ (Karras et al., 2018) , FFHQ (Karras et al., 2019) , and LSUN Bedrooms and Cars (Yu et al., 2015) . To quantify the image quality and diversity of random draws from the model at 256×256 resolution, we use the Fréchet inception distance (FID, Heusel et al., 2017) , which is comparable across models when sample size is fixed (Binkowski et al., 2018) . However, FID is known to remain constant under changing precision-recall characteristics (Kynkäänniemi et al., 2019) . We use LPIPS (Zhang et al., 2018) as the similarity metric, which has better correspondence to human evaluation than traditional L2 metrics. The degree of latent space disentanglement is often considered the most important property of a latent variable model. Qualitatively, it is the necessary condition for, e.g., style mixing capabilities. Quantitatively, Karras et al. (2019) noted that, for a constant-length step in the latent space, a more entangled model will produce a larger overall perceptual change than a less entangled model. The extent of this change, with LPIPS as the perceptual space metric, is the basis of measuring disentanglement as Perceptual Path Length (PPL).

4. EXPERIMENTS:

Baseline Methods For autoencoders, we compare to Balanced PIONEER by Heljakka et al. (2019) , a vanilla VAE, and a more recent Wasserstein Autoencoder (WAE, Tolstikhin et al., 2017) . For nonautoencoders, we compare to GLOW (Kingma & Dhariwal, 2018) and two recent GAN models: StyleGAN and Progressively Growing GAN (PGGAN, Karras et al., 2018) . Ablation We illustrate the contribution of the style-mixing loss L j and the robust loss d ρ on 64×64 CELEBA-HQ (Table 1 ). L j increases the FID of regular random samples, but reduces it when two random latents are mixed. d ρ then reduces some of the FID of regular samples.

4.1. ENCODING, DECODING, AND RANDOM SAMPLING:

In Table 2a , we compare encoder-decoder performance of the autoencoders on 128×128 CELEBA-HQ, with our proposed architecture ('AdaIn') and the corresponding regular architecture ('classic'). We measure LPIPS, FID and PPL. Our method has the best PPL, while Balanced PIONEER has the best FID (50k batch of generated samples compared to training samples).

Table 2b shows comparison of random sampling (examples in Figs. 4b-4c ) performance via FID and latent space smoothness via PPL for 256×256 images on CELEBA-HQ, FFHQ and LSUN Cars and Bedrooms. The performance of the automodulator is comparable to the Balanced PIONEER on most data sets, but the FID is worse in general. Models that only generate random samples have clearly best FID results on all data sets (NB: a hyper-parameter search with various schemes was used in Karras et al., 2019 , to achieve their PPL for FFHQ). We train on the actual 60k training set of FFHQ only (StyleGAN trained on all 70k images). We also evaluate the 4-way image interpolation capabilities in unseen FFHQ test images (Fig. 12 in the appendix) and observe smooth transitions. We emphasize that in GANs, such interpolations are often made between the codes of generated samples. As such, they cannot tell much about the recall characteristics of those models.

4.2. STYLE MIXING:

We demonstrate the style-mixing capabilities of our model. For comparison with prior work, we use the source images from the StyleGAN paper (Karras et al., 2019) . In Fig. 4a , we mix specific input faces so that the 'coarse' (latent resolutions 4×4 -8×8), 'intermediate' (16×16 -32×32) or 'fine' (64×64 -256×256) layers of the decoder use one input, and the rest of the layers use the other. Importantly, StyleGAN cannot take real inputs, so it can only mix between random images created by the model itself. For new input images, one must run a separate costly optimization process to determine the most fitting latent code. For our model, those images appear as completely new test images. Additional style mixing results are included in Figs. 13-14 in the appendix.

4.3. LEVERAGING INVARIANCES IN A WEAKLY SUPERVISED SETUP:

We now consider the cases where there is knowledge of some scale-specific invariances within the training data. Here, we demonstrate a proof-of-concept experiment that uses the simplest image transformation possible: horizontal flipping. For the CELEBA-HQ face data, this provides us with pairs of images that share every other property except the azimuth rotation angle of the face, making the face identity invariant amongst each pair. Since the original rotation of faces in the set varies, the flip-augmented data set contains faces rotated across a wide continuum of angles. For further simplicity, we make an artificially strong hypothesis that the 2D projected face shape is the only relevant feature at 4×4 scale, and does not need to affect scales finer than 8×8. This lets us enforce the L inv loss for layers #1-2. Since we do not want to restrict the scale 8×8 for the shape features alone, we add an extra 8×8 layer after layer #2 of the regular stack, so that layers #2-3 both operate at 8×8, layer #4 only at 16×16, etc. Now, with z 2 that corresponds to the horizontally flipped counterpart of z 1 , we have θ 3:N (ξ (2) , z 1 ) = θ 3:N (ξ (2) , z 2 ). Our choices amount to j = 3, k = N , allowing us to drop the outermost part of Eq. ( 9). Hence, our additional encoder loss terms are L inv = d(x 2 , θ 3:N (θ 1:2 (ξ (0) , z 2 ), z 1 )) and ( 10)

EQUATION

Fig. 5 shows the results after training with the new loss (50% of the training samples flipped in each minibatch). With the invariance enforcement, the model forces decoder layers #1-2 to only affect the pose. We generate images by driving those layers with faces at different poses, while modulating the rest of the layers with the face whose identity we seek to conserve. We receive variations of the original face that only differ in terms of the pose, unlike in regular automodulator training.

5. DISCUSSION AND CONCLUSION:

In this paper, we proposed the first generative autoencoder model with a hierarchical latent representation that supports controllable image generation and editing, conditional image sampling by fixing styles of specific layers, and style-mixing of real images. In our model, the latent vector independently modulates each decoder layer. The model outperforms other generative autoencoders in terms of latent space disentanglement and matches them in faithfulness of reconstructions, with slight reduction of output sample quality. We use the term automodulator to denote any autoencoder that uses the latent code only to modulate the statistical properties of the information that flows through the layers of the decoder. Such decoders could also include, e.g., 3D or graph convolutions.

Potential future applications include introducing completely independent 'plugin' layers or modules in the decoder, trained afterwards on top of the pretrained base automodulator, leveraging the mutual independence of the layers. The affine maps themselves could also be re-used across domains, potentially offering mixing of different domains. Such examples highlight that the range of applications of our model is far wider than the initial ones shown here, making the family of automodulators a viable alternative to state-of-the-art autoencoders and GANs. Upon acceptance for publication, our source code will be released at http://github.com/anonymized.

A APPENDIX:

In the appendix, we include further details on training and complement the results in the main paper with examples of random samples, reconstruction, latent space interpolations, and style mixing.

A.1 TRAINING DETAILS:

The training method follows largely those in Heljakka et al. (2019) , with progressively growing symmetric encoder and decoder, and decreasing the batch size when moving to higher resolutions. The encoder and decoder consist of 7 blocks each containting two residual blocks with a 3×3 filter.

In the encoder, these are followed by a spectral normalization operation (Miyato et al., 2018) and binomial filtering. In the decoder, by AdaIn normalization and binomial filtering. A leaky ReLU (p = 0.2) is used as activation. In the encoder, each block halves the resolution of the convolution map, while in the decoder, each block doubles it. The output of the final encoder layers is flattened into a 512-dimensional latent block. As in Karras et al. (2019) , the block is mapped by affine mapping layers so that each convolutional layer C in the decoder block is preceded by its own fully connected layer that maps the latent to two vectors each of length N , when N equals the number of channels in C.

Each resolution phase until 32×32 for all data sets use a learning rate α = 0.0005 and thereafter 0.001. Optimization is done with ADAM (β 1 = 0, β 2 = 0.99, = 10 -8 ). After the first two resolution steps, the KL margin is turned on and fixed to 0.5. For baselines, we used pre-trained models for StyleGAN, PGGAN, PIONEER, and GLOW with default settings provided by the authors. We trained the VAE and WAE models manually. For all VAE baselines the weight for KLD loss was 0.005. For all WAE baseline, we used the WAE-MMD algorithm. The weights for the MMD loss with automodular architecture (WAE-AdaIn) was 4 and with Balanced PIONEER (WAE-classic) architecure it was 2. For VAEs, the learning rate for the encoder was 0.0001, and for the generator 0.0005. For WAEs, the learning rate for both was 0.0002.

For evaluating the encoding and decoding performance, we used 10k unseen test images from the FFHQ data set, cropped the input and reconstruction to 128×128 as in Karras et al. (2019) and evaluated the LPIPS distance between the inputs and reconstructions. We evaluated 50k random samples in all data sets and compare against the provided training set. The GLOW model has not been shown to work with 256×256 resolution on LSUN (the authors show qualitative result only for 128×128). Training of PIONEER did not converge on FFHQ, however we believe this is an issue with the default hyper-parameters not suitable for FFHQ.

For Perceptual Path Length (PPL), we repeatedly take a random vector of length ε = 10 -4 in the latent space, generate images at its endpoints, crop them around mid-face to 128×128 or 64×64, and measure the LPIPS between them (Karras et al., 2019) . PPL equals the scaled expectation of this value (for a sample of 100k vectors).

Pre-trained models for PGGAN and GLOW were used with default settings provided by the authors. Note that we train on the actual 60k training images of FFHQ only (unlike StyleGAN that trained on all 70k images).

A.2 RANDOM SAMPLES:

Our model is capable of fully random sampling by specifying z ∼ N(0, I) to be draws from a unit Gaussian. 

:



Figures and Tables:

Figure 1: Illustration of some automodulator capabilities. The model can directly encode real (unseen) input images (left). Inputs can be mixed by modulating one with another or with a randomly drawn sample, at desired scales (center). Finally, taking random modulations for certain scales produces novel samples conditioned on the input image (right).

Figure 2: The model grows step-wise during training; the resolution doubles on every step. Input x is encoded into a latent enocoding z (a dimensionality of 512 used throughout this paper). The decoder acts by modulating an empty canvas ξ (0) by the latent encoding and produces the output x.

Figure 3: (a) The autoencoder-like usage of the model. (b) Modulations in the decoder can come from different latent vectors. This can be leveraged in feature/style mixing, conditional sampling, and during the model training (first pass). (c) The second pass during training.

Figure 4: (a) Style-mixing example using the same source images as Karras et al. (2019) (underlining that our model can directly work with real input images). (b-c) Random samples at 256×256.

Figure 5: The effect of training with face identity invariance enforcement under azimuth rotation.We generate images with the 'non-coarse' styles from the source images (left), and the 'coarse' styles from each top row image, in turn. With 'Enforced id. invariance', the top row only drives the face pose while conserving identity. In comparison, the 'Regular' training lets the top row also affect other characteristics, including identity. Here, the actual data that defined the invariance was too simple to allow learning fine-grained control of the hair at coarse levels, making it blurry.

Fig. 6-8 show samples from an automodulator trained with the FFHQ/CELEBA-HQ/LSUN data sets up to resolution 256×256. The samples here indicate the full range of samples and face features the model can support.

Figure 6: Random samples from the automodulator trained on FFHQ at a resolution 256×256.

Figure 7: Random samples for an automodulator trained on CELEBA-HQ at resolution 256×256.

Figure 8: Additional samples from an automodulator trained on LSUN Bedrooms and Cars a resolution of at 256×256.

Figure 12: Interpolation between random test set CELEBA-HQ images in 128×128 (in the corners) which the model has not seen during training. The model captures most of the salient features in the reconstructions and produces smooth interpolations at all points in the traversed space.

Effect of loss terms, CelebA-HQ.

Table content: <table><tr><td colspan="2">Method (64×64, 20M steps) FID-10k (normal)</td><td>FID-10k (mix)</td></tr><tr><td>Automodulator architecture</td><td colspan="2">11.54 ± 0.32 15.90 ± 0.22</td></tr><tr><td>+ Style-mixing loss</td><td>12.71 ± 0.28</td><td>14.88 ± 0.16</td></tr><tr><td>+ Robust loss (no L1 loss)</td><td colspan="2">12.23 ± 0.21 14.75 ± 0.14</td></tr></table>

Performance in CELEBA-HQ (CAHQ), FFHQ, and LSUN Bedrooms and Cars. We measure LPIPS, Fréchet Inception Distance (FID), and perceptual path length (PPL). Resolution is 256×256, except *128×128. For all numbers, smaller is better.

Table content: <table><tr><td colspan="4">(a) Encoder-decoder comparison</td><td/><td colspan="5">(b) Generative models comparison</td><td/></tr><tr><td colspan="2">(Using CAHQ  *  ) LPIPS (cropped) B-PIONEER 0.092</td><td>FID 21.51</td><td>PPL 92.84</td><td/><td colspan="6">FID (CAHQ) (FFHQ) (Bedrooms) (Cars) (CAHQ  *  ) (FFHQ) FID FID FID PPL PPL</td></tr><tr><td>WAE-AdaIn</td><td>0.165</td><td>100.02</td><td>62.17</td><td>StyleGAN</td><td>5.17</td><td>4.40</td><td>2.65</td><td>3.27</td><td>50.08</td><td>195.9</td></tr><tr><td>WAE-classic</td><td>0.162</td><td colspan="2">108.93 236.82</td><td>PGGAN</td><td>7.79</td><td>8.04</td><td>8.34</td><td>8.36</td><td>81.33</td><td>412.0</td></tr><tr><td>VAE-AdaIn</td><td>0.267</td><td>114.52</td><td>83.52</td><td>GLOW</td><td>68.93</td><td>-</td><td>-</td><td>-</td><td>138.21</td><td>-</td></tr><tr><td>VAE-classic</td><td>0.291</td><td>173.37</td><td>71.75</td><td>B-PIONEER</td><td>25.25</td><td>-</td><td>21.52</td><td>42.81</td><td>92.84</td><td>-</td></tr><tr><td>Automodulator</td><td>0.102</td><td>36.19</td><td>41.45</td><td colspan="2">Automodulator 51.96</td><td>64.07</td><td>35.74</td><td>35.61</td><td>41.45</td><td>262.3</td></tr></table>

The length of training phases amounts to 2.4M training samples until 64×64 resolution phase, which lasts for 10.4M samples. For FFHQ and CelebAHQ, the 128×128 phase uses 13.0M samples while LSUN Bedrooms and Cars use 10.0M samples. The final 256×256 phase uses 7-10M samples for each data set. The training of the final stage was generally cut off when reasonable FID results had been obtained. More training and learning rate optimization would likely improve results. With two Titan V100 GPUs for pretraining stages and four GPUs for the 256×256 stage, the training time for CELEBA-HQ and FFHQ were 18 days each, and for LSUN Bedfrooms 17 days and Cars 19 days.For evaluating the model after training, a moving expontential running average of generator weights was used, as in bothKarras et al. (2018) andHeljakka et al. (2019). For all data sets, training/test set splits were used as given or defined by data set authors, except for LSUN Cars, where we used 4,968,695 samples for training and 552,061 for testing. Note that in regular GAN training, complete data sets are often used without train/test split, leading them to use larger training sets.

Table content: <table/>

Peer review (iclr2020):
The submission proposes an autoencoder architecture which combines two recent GAN-based architectural innovations, namely the progressive growing of the decoder architecture (as well as the encoder architecture in this case) and the use of the encoded representation to modulate the decoder via a feature-wise transformation mechanism. I think the overall idea behind the paper is valuable, but I don’t think the submission meets the acceptance bar from a clarity point of view. I also have concerns with its characterization of the literature. Clarity-related comments: - “This allows the layers to work independently of each other.” This is an imprecise use of the term “independently”. How can two layers work independently if one’s input is the output of the other? - “The reconstructed samples could be re-introduced to the encoder, repeating the process, and requiring consistency between passes.” The rest of the paragraph is built on the premise that this is a desirable property, but I’m not sure I understand why this is a desirable property in the first place. - How to define “disentanglement” in the context of representation learning is in itself an unsettled question as far as I’m aware, but the submission uses the term without an intuitive or formal definition. What do the authors mean by “disentangled representation”? What is measured by perceptual path length (PPL), and in which ways does PPL relate to the author’s definition of “disentangled representation”? - “[...] a new autoencoder-like model with powerful properties not found in regular autoencoders, including style transfer.” The term “style transfer” is overloaded; what do the authors mean? - The submission defines AdaIn as a way to combine “content” and “style”, and defines the style “y” in terms of mean and variance. In the context of the AdaIn paper, this makes sense: the instance normalization shifting and scaling coefficients are heuristically defined as the channel-wise means and standard deviations of a “style” stack of feature maps. However, in the context of this submission I’m not sure this definition makes as much sense: the instance normalization shifting and scaling coefficients are the result of a linear projection of the latent representation and do not involve the channel-wise means and standard deviations of an external stack of feature maps; is this correct? - “This setup follows the same logic as that of Karras et al. (2019), but we do not require an ad-hoc disentanglement stack.” Can the authors clarify what they mean by an “ad-hoc disentanglement stack”? - Section 3.2 uses some notation for the encoder without introducing it first. I believe the only way to understand that phi(x) refers to the encoder network is to look at Figure 2. - Section 3.2 as a whole is hard to follow, in part due to the use of imprecise language (“mutually independent”, “representation of those levels disentangled in z”). At some point probability distributions are introduced (up until now the reader is operating under the assumption that the model is an autoencoder with no probabilistic interpretation), and mutual information is mentioned to justify an L2 reconstruction loss in z-space (which I would argue is an instance of mathiness that does not serve the reader’s comprehension). Can the authors explain in plain language how layer-specific losses are defined and how the complete loss is obtained? - The submission presents model samples, but as far as I can tell the procedure for sampling is not provided. Unlike VAEs, autoencoders do not explicitly model the empirical distribution -- although reconstruction in denoising autoencoders is related to the score of the empirical distribution (Alain & Bengio, 2014). How are samples obtained from the trained model? Literature-related comments: - The use of normalization layers to implement feature-wise transformation mechanisms is fairly widespread nowadays, but for instance normalization specifically the work of Dumoulin et al. (2017) pre-dates that of Huang & Belongie (2017). Both are cited by Karras et al. (2019) in relation to AdaIn (which is termed “conditional instance normalization” in Dumoulin et al. (2017)). - I disagree with the characterization of ALI/BiGAN as “hybrid models that combine the properties of VAEs and GANs”: unlike AAE and AVB, which minimize KL-divergence terms in the VAE loss adversarially, the objective for ALI/BiGAN is purely adversarial. I would also include IAE (Makhzani et al., 2018) and BigBiGAN (Donahue et al., 2019) in the list of GAN variants that incorporate an inference mechanism. - The submission repeatedly asserts that GANs lack an inference mechanism: “Unlike GANs, autoencoder models can directly operate on input samples.”; “To work on new input images, GANs either need to be extended with a separate encoder, or inverted [...].”; “[...] GANs show good image quality, but have no built-in encoding mechanism [...]”; “[...] the problem with GANs is that they lack the encoder [...]”. This is false: see for example ALI, BiGAN, and BigBiGAN. The problem in my opinion is elsewhere: the kinds of reconstructions these models yield are not suited to the downstream applications investigated in this submission, because they oftentimes fail to preserve low-level details. References: - Alain, G., & Bengio, Y. (2014). What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1), 3563-3593. - Dumoulin, V., Shlens, J., & Kudlur, M. (2017). A learned representation for artistic style. In Proceedings of the International Conference on Learning Representations. - Makhzani, A. (2018). Implicit autoencoders. arXiv:1805.09804. - Donahue, J., & Simonyan, K. (2019). Large scale adversarial representation learning. arXiv:1907.02544.