Disentangled GANs for Controllable Generation of High-Resolution Images
Authors: Weili Nie
Tero Karras
Animesh Garg
Shoubhik Debhath
Anjul Patney
Ankit B. Patel
Anima Anandkumar
Venue:  (2020)
Keywords: Disentangled GANs
controllable generation
high-resolution image synthesis
semantic manipulation
fine-grained factors

Abstract: Generative adversarial networks (GANs) have achieved great success at generating realistic samples. However, achieving disentangled and controllable generation still remains challenging for GANs, especially in the high-resolution image domain. Motivated by this, we introduce AC-StyleGAN, a combination of AC-GAN and StyleGAN, for demonstrating that the controllable generation of high-resolution images is possible with sufficient supervision. More importantly, only using 5% of the labelled data significantly improves the disentanglement quality. Inspired by the observed separation of fine and coarse styles in StyleGAN, we then extend AC-StyleGAN to a new image-to-image model called FC-StyleGAN for semantic manipulation of fine-grained factors in a high-resolution image. In experiments, we show that FC-StyleGAN performs well in only controlling fine-grained factors, with the use of instance normalization, and also demonstrate its good generalization ability to unseen images. Finally, we create two new datasets -- Falcor3D and Isaac3D with higher resolution, more photorealism, and richer variation, as compared to existing disentanglement datasets.

Under review as a conference paper at ICLR 2020
DISENTANGLED GANS FOR CONTROLLABLE GENER- ATION OF HICH-RESOLUTION IMAGES
Anonymous authors Paper under double-blind review
ABSTRACT
Generative adversarial networks (GANs) have achieved great success at generat- ing realistic samples. However, achieving disentangled and controllable genera- tion still remains challenging for GANs, especially in the high-resolution image domain. Motivated by this, we introduce AC-StyleGAN, combination of AC- GAN and StyleGAN, for demonstrating that very limited supervision is sufficient to benefit the generic disentanglement learning of highluteolutton images. Specif- ically, by only using 1% of the labelled data, the disentanglement quality is very close to the fully supervised case and significantly outperforms the unsupervised alternative. Inspired by the observed separation of fine and coarse styles in Style- GAN, we then extend AC-StyleGAN to a new image-to-image model called FC- StyleGAN for semantic manipulation of fine-graing gree-graied factors image. In experiments, we show that FC-StyleGAN performs well in only control- ling fine- -eniigr factors, with the use of instance normalization, and also demon- strate its good generalization ability to unseen images. Finally, we create two new datasets - Falcor3D and Isaac3D with higher resolution, more photorealism, and richer variation, compared to existing disentanglement datasets.
high-resolution
INTRODUCTION
High-resolution controllable generation is an important component in many applications, such as image editing (Yao et al. 2018). 3D scene understanding (Eslami et al., 2018) and inverse graph- ies (Kulkarni et al.) 2015). Generative adversarial networks (GANs) (Goodfellow et af.) 2014) have achieved great success at generating realistic images, of which two representatives are Style- GAN (Karras et al. 2019) for unconditional generation and BigGAN (Brock et al., 2018) for class conditional generation. However, the controllable generation is still a challenge for state-of-the-art GANs, especially with high-resolution images. For instance, StyleGAN cannot be directly used to synthesize high-fidelity human faces by specifically controlling skin color or eye size without af- fecting other face attributes. Also, BigGAN is not able to control the hair color or length of a dog image without changing other features.
Disentanglement of various factors allows us to independently control the variations across all the factors. But this is not easy to learn in GANs without further modifications such adding regular- ization to encourage better disentanglement. For example, InfoGAN (Chen et al., 2016) proposes maximizing mutual information between latent code and its reconstruction for an unsupervised dis- entanglement. Most recently. Locatello et al. (2019a) has showed that unsupervised disentanglement learning impossible without model inductive bias or supervision. In that light, several recent works attempt to learn disentanglement in GANs by introducing strong model bias (Nguyen- Phuoc et al.) 2019. However, these works are restricted a specific domain (e.g. 3D rotations only) and usually difficult to scale up to higher resolution, mainly due to the intrinsic limitations of the model biases themselves. This motivates us to study the impact of limited supervision a generic disentangle- ment learning with GANs, especially in the more challenging high-resolution image domain.
By combining the advantages of AC-GAN (Odena et al.) 2017) and Style- GAN, we introduce AC-StyleGAN, and demonstrate that, very limited supervision is sufficient: just 1% of the labelled data significantly improves the disentanglement quality, as compared to the unsu- pervised alternatives. However, if we are only interested in controlling a subset factors (effectively treating the as random unobserved nuisance variables), find that the disentanglement qual-
Main contributions:

---

Under review as a conference paper at ICLR 2020
ity of AC-StyleGAN degrades significantly. We believe this is due the latent nuisance factors strongly confounding the observed factors, a common and difficult problem in high- nigh-dimensional partially observed latent variables models (Bishop 1998).
To address this, we propose FC-StyleGAN, a new image-to poseetto-mmace model that adds controllable fine- grained factors along with a super-resolution process. This is inspired by the separation of fine and coarse styles, observed in StyleGAN. FC-StyleGAN enables semantic manipulation of fine styles in high-resolution images, without a commonly-used encoder-decoder structure.
Finally, we create two new high-quality datasets Falcor3D and Isaac3D - that present a new challenge for controllable generation in terms of image resolution, photorealism, and richness of style factors, as compared to existing disentanglement datasets such as 3D Chairs (Aubry et al., 2014). dSprite (Matthey et al. 2017) and MPI3D (Gondal et al. 2019).
Thus, we propose new semi-supervised GAN architectures that enable a generic disentanglement learning of high-resolution images as well as new high- -ualitya disentanglement datasets.
2 RELATED WORK
Disentanglement learning. Learning generic disentangled representations in an unsupervised way has attracted a lot of attention. Two representative models are InfoGAN (Chen al.) 2016) and (--AE (Higgins et al., 2017). The nature of unsupervised disentanglement learning does not guarantee that the learned disentangled factors are semantically meaningful without an additional inductive bias or supervision (Locatello et al., 2019a. Nguyen-Phuoc et al. 2019 Locatello et al.) 2019b). Another line of work aims to learn disentangled representations via supervised learning wherein factors observed variables (Kulkarni et al. 2015 Reed et al.) 2014 Xiao et al. 2018 Locatello et al. 2019b). However, little work has systematically investigated the necessity very limited supervision generic disentanglement learning. The analysis of Locatello et al. (201bb) only applies to disentangled VAEs on low-resolution datasets. Instead, our work tries to answer a more practical question: with very little supervision more complex and higher-resolutee can we still well disentangle the factors of variation while maintaining the high generation quality?
datasets,
Deep image manipulation. Deep neural networks have enabled various image editing tasks, such as style transfer (Gatys et al., 2016).. image-to-image translation (Zhu et al. 2017), automatic col- orization (Zhang et al., 2016) and 3D-aware attribute editing (Yao et al. 2018). None of the above methods except the 3D-SDN Yaa al. 2018) has dealt with the semantic maliponation of multiple attributes for scene images. Different from the 3D-SDN and other previous works on attribute edit- ing that have been mostly focused the 3D geometry manipulation, our proposed FC-StyleGAN is designed to semantically manipulate the fine-grained factors instead, such as lighting conditions and object colors. Furthermore, FC-StyleGAN does not apply an encoder-decoder structure as com- monly used in previous works for semantic manipulation (ao et al.) 20188), instead, it adds control- lable fine-gre ane-grained factors along with a super-resolution process. which turns out to perform well in editing fine styles of high-resolution images with a good generalization ability.
MODELS
3.1 BACKGROUND ON AC-GAN AND STYLEGAN
AC-GAN. AC-GAN (Odena et al. 2017) is variant of class conditional GANs. In the AC-GAN, the generator inputs are a latent 2 (or called random noise) and the class label. The input to the discriminator is the real or fake image, whilst its output is the probability that the image is real and the prediction of the class label. Odena et al. (0177) has shown that this modification to the standard conditional GAN formulation produces more realistic images and appears stabilize training.
StyleGAN StyleGAN (Karra al., 2019) is ofate-of-the-ee art GAN architecture for unsuper- vised image generation, particularly for high-fidelity human faces. Basically, StyleGAN comprises mapping network whose role is to map the latent = to intermediate space, which then controls the styles at each convolutional layer in the synthesis network via adaptive instance normalization (AdaIN) (Ulyanov al. 2016. Huang & Belongie| 2017]. StyleGAN also enables the separation of fine-grained and coarse- grasind features. For example, modifying the styles of low-resolution blocks affects only coarse- coarse-grained features (e.g. pose and eyeglasses), while modifying the styles of high-resolution blocks affects only fine-gra tne-grained features (e.g. color scheme and microstructure).
2

---

Under review as a conference paper at ICLR 2020
BraliFisks
Meta code
dss Chesstast
RestPake
Veta oods
Maspins Netwonks
Imsaee
Block
Block
Fine
Downn
Texts tipe. Black
Drwn
Ióxiá
Sayping Network
Copy I=1
Copy $03
[6<ll Bown llbeck
Fode
3aC33 Lipe. Blick
Dran Tirck
Ups Black
64/64
64-64
Block
64-6d Dawa Block
(i Generator
(it) Discriminator
Generator
(ii) Discriminator
(a) StyleGAN
StySeeGGGAN
Figure 1: An overview of model architectures (a) AC-StyleGAN and (h) FC-StyleGAN (a) The generator con- ditions code for generation and the discriminator predicts its value. (h) We downsample the real image into 32x32 resolution and replace the lower resolution blocks (4x4 32x32) in AC-StyleGAN generator by a new input block. Also, the discriminator predicts the value of fine code from the 32x32 block instead.
3.2 AC-STYLEGAN FOR CONTROLLABLE IMAGE SYNTHESIS
We introduce AC-StyleGAN, combination of AC-GAN and StyleGAN, that enables conditional generation of high-resolution images. As shown in Figure Ta the generator in AC-StyleGAN con- ditions on a meta code by simply concatenating the meta code, vector representing all the factors of variation, with the latent The discriminator in AC-StyleGAN now provides two outputs, the classification real/fake and the prediction of meta code. And finally, the outputs of the mapping network the conditioned styles will modulate each block in the synthesis network via AdaIN. We choose AC-GAN, instead of other conditional GAN methods, such as cGANs (with Projec- tion Discriminator) (Miyato & Koyama 2018 mainly because only AC-GAN has the property of reconstructing meta code in the discriminator, which can easily be extended to a semi-supervised disentanglement learning framework.
More formally, let fr(()) denote the n-th image from the set of N real images and let (()) be the corresponding meta code, randomly sampled from the dataset. Let ef denote a random meta code sampled from the label distribution, and let D(-) and G(.) denote the discriminator and generator neural networks. We also assume 2(²)) and 8 represent the real and fake classification logits the discriminator, respectively, and af*) and ef()) denote the predictions of (n) and of respectively. From Figure la we have
(s)"), a(()) D(a{")) and (s) oe = - ()))
Since we are aiming at achieving both high image quality and good controllability. we define a semi-supervised loss function for the generator and discriminator in AC-StyleGAN follows,
N-1
(G, = (of()) .56 (())
Σ ==
lef of
N
GAN loss
unsupervised disentanglement
(iy(n) 2n=0 17(1))
LP)
(sfo)) (n) sf))))
-
|a(m)
c(n)
7=0
GAN loss
unsupervised disentanglerneni
supervised disentanglement
(2)
which consists of three terms: the GAN loss term, unsupervised disentanglement term and super- vised disentanglement term. For the GAN loss, we apply the non-saturating loss plus gradient penalty in the discriminator, as StyleGAN. For both the unsupervised and supervised disentangle- ment terms, we simply use l norm of the reconstruction losses.
Furthermore, we introduce two coefficients 7 and (n) in the above loss function: (i) the disentan- glement coefficient E (0, x)) balances the trade-off between the generation quality and disen- tanglement performance; (ii) the coefficient (n()) is the label mask denoted by nf(n) 1 {a(a)(a)
3

---

Under review as a conference paper at ICLR 2020
where (n) d.d.d. Uniform(0 [0, 1] and a E [0, 1] is a hyperparameteer Accordingly, the hyperparameter controls the fraction of labelled data that will be used for supervision (fully unsupervised in the disentanglement learning if a = O, fully supervised if a =
Note that we sample all the (n)'s before training so that the label mask sf(")) remains unchanged during training to reflect the real semi-supervised case_ We also apply rescaling operation by mul-
tiplying the supervised disentanglement term in Eq. ( with the rescaling coefficient > o sy6n) to
take an average of effective supervised signals within a minibatch. Finally, note that AC-StyleGAN reduces to an InfoGAN variant in the special case of a = 0 (fully unsupervised), when an appropri- ate reconstruction loss is used.
3.3 FC-STYLEGAN FOR SEMANTIC MANIPULATION OF FINE STYLES
While we apply AC-StyleGAN with supervision over all observed factors for demonstrating con- trollable generation of high-resolution images in GANs, in a potentially more realistic case where we are only interested in controlling a subset of factors (effectively treating the rest as random un- observed nuisance variables), the disentanglement quality of AC-StyelGAN drops significantly, as shown in Section 4.2 Furthermore, AC-StyleGAN cannot be directly applied to manipulate an existing high-resolution image. Therefore, we propose new image-to-image model called FC- StyleAGN (i.e., Fine-grained Controlled StyleGAN) for only controlling fine-grained factors.
Inspired by the observations that the lower resolution blocks in the StyleGAN generator learn the coarse-grained features while its high-resolution blocks accounts for fine styles, the generator in FC-StyleGAN does not contain the lower-resolution blocks. As shown in Figure [b] the generator instead takes the real image as one of its inputs by downscaling it to a lower resolution ó (c.g.. = 32 in Figure Lbp. After that, it generates the high-resolution image by only modulating the fine-grained code into higher resolution blocks. Also, the discriminator in FC-StyleGAN predicts the value of fine-gran fine-grained code from the block with resolution 6, instead of the last output block. The intuition is that similar to the observations in the generator, lower resolution blocks in the discriminator have less relationship with fine-grained styles. So it makes more sense to directly predict the meta code from higher resolution blocks. Note that the value of Φ varies from 4 to image size, determining what factors are identified fine-grained. Finally, the loss function in FC-StyleGAN is the same with Eq. (2) in AC-StyleGAN, indicating a semi-supervised approach.
During training of StyStyl,G,N, the generator takes the downscaled image as is, and just em- bellishes it by increasing the resolution and generating the missing pixel-level detail. As it does not need to learn the coarse-grained features any more, presumably it will have much easier time handling complex high-resolution images. Furthermore, since the generator is still style-based in higher-resolution blocks with fine-grained code as its input, we will retain the control over fine styles. However, there are two caveats in FC-StyleGAN The first caveat is that as an image-to- image model, the original fine-grained factors in the input image may interference the control of fine code over the fine styles in the output image. In such sense, emphasize the role of inductive biases on the model - the use of instance normalization in the generator. Because the output of in- stance normalization preserves the spatial structure of image content by only normalizing fine styles (Ulyanov et al. 2016), we argue that the original fine styles in the input image could be washed away by instance normalization, and thus the fine code will fully control the fine-grained factors.
The second caveat is how to identify fine-grained factors in a dataset before training FC-StyleGAN. To this end, we use the meta code of all factors as the input of FC-StyleGAN with different down- scaled resolutions é. For each Φ, we introduce a new term interpolation variance of each factor c, denoted by (d)). based on which the factor e is defined as -rained if its interpolation variance satisfies that (,(()) > fo. where 3o is a pre- pre-defined threshold. The interpolation variance (() is calculated as follows: Given N real images, we do the latent traversal over each factor ((")) of an image ((x) to get S interpolated (fake) images (8(p) .... 1 Each interpolated image j(70) is then fed into the discriminator to get a predicted factor a(m)) Thus, the interpolation variance is the average variance of a(7) over the interpolation s-dimension, which is
Bi(d)
(n)) Var, C
(3)
4

---

Under review as a conference paper at ICLR 2020
Datasets
# of Images
of Factors
Resolution
3D
dSprites
737,280
64x64
X
Noisy dSprites
737.280
7
64x64
x
Scream dSprites
737.280
64x64
x
SmalINORB
48,600
128x128
Cars3D
17,568
64x64
3dshapes
480,000
7
64x64
MPI3D
640,800
7
64x64
Falcor3D
233,280
1024x1024
Isaac3D
737,280
512x512
Table 1: Summary of the proposed two datasets, compared with currently commonly-used datasets (Gondal al. 2019 We can see that the proposed two datasets - Faclor3D and Isaac3D both have much larger resolutions than previous datasets, together with the maximum number of factors. Furthermore, in terms of photorealism, both datasets are rendered based on complex 3D scene, in particular with texturing in Isaac3D.
Intuitively, low interpolation variance (Ø) means that changing the factor e; does not affect much the generated images, which further implies the factor 6; should not be considered as fineggraieed, since FC-StyleGAN with downscaled resolution & cannot control any more.
4 EXPERIMENTS
In this section, we first introduce two new datasets Isaac3D and Falcor3D. We then evaluate the performance of AC-StyleGAN and FC-StyleGAN both datasets, focusing on three aspects: dis- entanglement quality, semantic correctness and image quality. Quantitatively. we use Frechet In- ception Distance (FID) (Heusel al., 2017) to measure image quality, reconstruction error (i.e., hree |a(r) cr) []) to measure semantic correctness, and MIG (Chen et al. 2018 to measure disentanglement quality. Qualitatively, we apply the latent traversals to visually evaluate all three aspects as well. Due to space limitations, we only show results on Isaac3D in this section; results on Falcor3D are quite similar to those on Isaac3D, and are available in the appendix.
4.1 CREATION OF HIGH-QUALITY DISENTANGLEMENT DATASETS
Current disentanglement datasets, such as 3D Chairs (Aubry et al. 2014), dSprites (Matthey et al., 2017) and MPI3D (Gondal al. 2019), are of low resolution and mostly lack of photorealism. It makes them not suitable as disentanglement benchmarks in the high-resolution image domain. To this end, we propose two new datasets Falcor3D and Isaac3D, which possess much higher resolution, better photorealism and richer factors of variations, as shown in Table [
Falcor3D In the Falcor dataset, there are in total 233,280 images and each has a resolution of 1024x1024. This dataset is based the 3D scene of living room, where
move the camera positions and change the lighting conditions. Each image is paired with ground-truth meta code, consisting of factors of variation: lighting intensity (5), lighting x-dir (6), lighting y-dir (6), light- ing z-dir (6), camera x-pos (6), camera y-pos (6), and camera z-pos (6). Note that the number m behind each factor represents that the factor has m possible values, uniformly sampled in the nor- malized range of variations [0, 1]. To interpret this, for example, "lighting z-dir (6)" represents the lighting direction moving along the z-axis and "camera z-pos (6)" represents the camera position moving along the z-axis. Also, both factors have 6 values uniformly sampled from [0, 1].
Isaac3D In the Isaac3D dataset, there are in total 737,280 images and each has a resolution of 512x512. This dataset based on the 3D scene of kitchen, where we can also move the camera positions and vary the lighting conditions. To further increase the number of variations, we put a robotic arm inside, attached with an object. The robotic arm has two degrees of freedom: x- movement (or horizontal rotation) and g-movement (or vertical rotation). The attached object could change its shape, scale and color. To make the rendered images more photorealistic, each object in the 3D scene has been provided with proper textures. Finally, each image is paired with ground- truth meta code, consisting of factors of variation: lighting intensity (4), lighting y-dir (6), object color (4), wall color (4). object shape (3), object scale (4), camera height (4), robot z-movement (8), and robot g-movement (5). Similarly, the number m behind each factor represents that the factor has m possible values, uniformly sampled in the normalized range of variations [0, 1).
5

---

Under review as a conference paper at ICLR 2020
robot c-mor.
contra
beight
lighting
Figure 2: Latent traversal of AC-StyleGAN with full supervision on Isaac3D. For illustration, we only show three factors: (robot z-movement, camera height, lighting y-dir). Please see the appendix for latent traversal of all the factors. Images in the first column (marked by red box) randomly sampled real images with the ground-truth and the rest images each row are their interpolations, respectively, by uniformly varying the given factor from 0 to Unless otherwise stated, this setting applies to all the latent traversal results below.
115
546
Figure 3: Quantitative metrics MIG, hee and FID vary with the supervision coefficient (0.0, 0.01, 0.1, and the disentanglement coefficient y {1, 10} in AC-StyleGAN Isaac3D. For MIG, the higher is better. while and FID, the lower better.
Please see the appendix for more detailed descriptions the two datasets.
4.2 EVALUATION OF CC-STYLEGAN
Latent traversal of AC-StyleGAN. We first show the latent traversal results of AC-StyleGAN on Isaac3D in Figure 2 where we apply full supervision from meta code of all the factors. For illus- tration, here we only show three factors of variation: (robot z-movement, camera height, lighting y-dir). Please see the appendix for latent traversal of all the factors. Images in the first column (marked by red box) are randomly sampled real images and the rest images in each row are their in- terpolations, respectively, by uniformly varying each factor from 0 to 1. Unless otherwise stated, this setting applies to all the latent traversal results below. As we can see, each factor in the interpolated images changes smoothly without affecting other factors. Taking "lighting y-dir" only the direction of the point light gradually moves up from left to right in the bottom row, which the wall and the
example,
can be evidenced by the fact that the light spot moves up from the refrigerator shadow of the cabinet on the wall moves down simultancously. More importantly, all the interpo- lated images visually look the same with the corresponding real images in the first column except for the interpolated factors. and the image quality does not degrade over interpolation. Therefore, the latent traversal results demonstrate three good properties of AC-StyleGAN with supervision: high disentanglement quality and good semantic correctness and high generation quality.
Semi-supervised learning. How does AC-StyleGAN behave when given fewer labels? To this end, we vary the supervision coefficient in Eq. (2) to show the impact of supervision on AC- StyleGAN and the results with 7 E (1, 10} on Isaac3D are Figure 3 First, we can see with limited supervision (a 2 0.1), the difference between
and 10 is quite small. Also, both values of MIG and Leee almost reach their own optimal ones, along with a lower FID. This quantitatively supports the above latent traversal results in Figure 2 As expected, the disentanglement quality and semantic correctness gradually get worse when less labelled data used. Interestingly, the gap between 7 and 10 first increases and then decreases decreases from 0.5 to 0, with the largest

---

Under review as a conference paper at ICLR 2020
4-20
0.19
ebject wade
332-
abject
color
0.13
cua
cot
btteer
677
(a) Supervision from a subset of factors
(b) Identifying fine-grained factors
Figure 4: (a) Latent traversal of AC-StyleGAN on Isaac3D, where we only control subset of factors: (rohot. movement, robot y-movement, object scale, lighting y-dir, object color) and here two of them are shown. Please see the appendix for all five considered factors. (b) Interpolation variances of each factor in FC-StyleGAN with different downscaled resolutions & on Isaac3D. where the meta code of all factors is used as its input. Empirically, the factor with its interpolation variance below the threshold (marked by the green line) cannot be controlled by FC-StylcGAN with the given db
one at a = 0.01. It means that properly increasing ς is the most beneficial for disentanglement in see, by only using
the case where only a very limited number of labelled data is provided. As
1% of the labelled data (a = 0.01) and setting 7 = 10, the disentanglement quality and semantic correctness are still close to those with full supervision (a = 1), and improve over the unsupervised baseline (a = 0) by a significant margin. Please see the appendix for the latent traversal results of AC-StyleGAN with a 0.01 and 7 10. It means adding a very small amount of labelled data into the training dataset could benefit much the disentanglement learning. Finally, in terms of image quality, we observe that larger tends to result a lower FID score, which means better disentanglement is not always obtained by sacrificing the generation quality.
Comparisons with different baselines. Current start-of-the-ar approaches for generic unsuper- vised disentanglement learning are mostly VAE-based models, such as 8- VAE (Higgins et al. (017), Factor VAE (iim& & Mnih 2018) and 3-TCVAE (Chen et al.] 2018). To this end, we use the unsu- pervised version of AC-StyleGAN (a 0) compare with VAE-based models (i) the dSprites dataset and (ii) the Isaac3D dsataset, respectively. Note that for evaluations on Isaac3D, we first 128x 128 because VABS have difficulties in generating higher-resolution
downscale its resolution images. We can see from Table 2 that the unsupervised AC-StyleGAN consistently outperforms all the dataset, disentangled VAEs have much higher FID scores (FID > 100), indicating their poor image generation quality. Please see Appendix A.5 for their randomly sampled images. Interestingly, the disentangled VA also fail to capture all the variations in the Isaac3D dataset, supporting the claim that our datasets provide larger variation of factors for disentanglement learning.
disentangled VAEs on both datasets. Particularly on the downscaled Isaac3D
Previous work on the semi-supervised learning for classification has shown that supervised base- line with only the labelled data could be very competitive (Oliver et al.] 2018 Zhai et al. 2019). Hence, to demonstrate the effectiveness of actual semi-supervised components for disentanglemert we also compare our semi-supervised method with the supervised baseline, in which we remove the unsupervised disentanglement term in the discriminator loss function of Eq. 2). The results are shown in Table 3 We can see that when only 1% or 5% of lablled data is available, semi-supervised AC-StyleGAN consistently outperforms the supervised baseline in terms of all three metrics: FID, MIG and 4re- Moreover, the advantages of using the unsupervised disentanglement term become larger when we have fewer labelled data.
Latent traversal on 256x256 CelebA. To show the performance of AC-StyleGAN on the real- world data, we also provide the qualitative results on CelebA with resolution 256x256 (Liu et al. 2015). In this experiment, the meta code in AC-StyleGAN has length of 40 to capture all the attributes in the dataset. Hence, this is more challenging than the previous works on CelebA in terms of controlling all the attributes in higher resolution (Chen et al. 2016 Higgins et al. 2017 Nguyen-Phur Phuren et al., 2019). Note that since CelebA only has binary attributes, which commonly used disentanglement metrics including MIG and FactorVAR score not applicable to, we focus
7

---

Under review as a conference paper at ICLR 2020
Methods
FactorVAF Score
MIG
Methods
FID
MIG
B VAE
0.731 = 0.020
0.177 E 0.031
B. 3-VAE
2.2.6
0.231 0.068
VactorVAE B-TCVAE
0.757 = 0.107
0.157 + 0.051
FactorVAE 8-TCVAE
305.8 142.1
0.245 - 0.034
0.783 + 0.057
0.198 + 0.019
155.4 13.6
0,216 - 0.074
Ours (y Ours (7 = 10)
0.760 0.050
0.213 0.090
Ours (7 1)
4.27 1.12
0.372 1 0.126
0.840 + 0.090
0.290 + 0.098
Ours ( = 10)
3,20 + 0.30
0.291 + 0.077
(a) dSprites with resolution 64x64
(b) Isaac3D with resolution 128x128
Comparison of unsupervised AC-StyleGAN and state-of-the-ar VAE-based models: 8-VAE, Fac- for VAE and 3-TCVAF on the dSprites and downscaled Isaac3D datasets. Note that the scores of VAE-hased models are obtained based the implementation in Locatello et al. (2019a), where set , = 6 for 3-VAF, y 30 for FactorVAE and 8 for 3-TCVAE after a grid search over different hyperparameters.
Table 2:
Methods
FID
MIG
Lece
Sup. baseline
2.80 + 0.22
0.963 + 0.007
0,131 + 0.013
5%
Ours
2.25 +: 0.41
0.990 + 0.002
0.035 + 0.010
Sup. baseline
3.58 1 0.20
0.766 1 0.042
0.358 1 0.067
1%
Ours
2.74 1 0.20
0.848 0.085
0.231 1 0.157
Table 3: Comparison of semi-supervised AC-StyleGAN with the supervised baseline, where we use 7 = 10 and consider two cases of using 5% and 1% of labelled data. In the supervised baseline, remove the unsupervised disentanglement term the discriminator loss function Eq.
on the qualitative results of latent traversal. The results are shown in Figure We can see that AC-styleGAN can smoothly interpolate in the latent space of face attributes and also maintain high image quality. More interesting interpolations can be seen in Appendix [A.6] where some attributes are more difficult to change during latent traversal, presumably due to the fact that these attributes have low frequency in the CelebA dataset, such Mustache on female faces.
Only controlling subset of style factors. Instead of trying to control all the factors in a dataset, practically speaking we may only be interested in subset of factors in which case the remaining fac- tors will be considered as random nuisances captured by the latent On a first look, it might seem intuitive that the fewer factors of variation we try to control, the better the disentanglement quality should achieve. However, the latent traversal results of AC-StyleGANC Figure 44) show instead that both disentanglement quality and semantic correctness degrade. For example, the camera height changes when interpolating the object scale, and the wall color also varies when interpolating the object scale. Furthermore, the latent z, which is expected to capture the remaining factors: (object shape, camera height, lighting intensity, wall color), can barely change the camera height and wall color (see the appendix for results of other factors). To explain this, we argue that since factors are highly correlated with each other, learning to control one may help a lot in learning to control the others. Without observations on other nuisance factors, it may be very difficult to disentangle the impact of the observed factor with unobserved covariates, especially in the high-dimensiinal space.
4.3 EVALUATION OF FC-STYLEGAN
Identifying fine-grained factors. As we have discussed, a crucial for FC-StyleGAN to quan- titatively identify fine-grained factors. Here we calculate the interpolation variance of each factor according to Eq. (3) by setting N = 100, S = 10. Figure [4b] shows the interpolation variance of each factor in FC- StyllaAG with downscaled resolution o varying from 4 to 128 on Isaac3D. To decide the threshold So we visually check the latent traversal results and find that the interpolations factor c; almost stay the same if (Ø) 0.055, which means 3o 0.055. Please the appendix for the latent traversal results. We can see that as Φ gets larger, more factors have an inter- polation variance below the threshold 9o, meaning more grasiee factors appear. For instance, there are only two coarse-grained factors at Φ 8: camera height and lighting direction, and two more factors become coarse- coarse-grained factors at á 16: robot n-movement and robot y y-movement, and so on. Interestingly, wall color is always fine-grained even at ó = 128 while object color has already become coarse-grained Φ 64, although both of them are about colors.
8

---

Under review as a conference paper at ICLR 2020
Open
Bangs
Groy Hair
Figure 5: Latent traversal of AC-StyleGAN on CelebA with resolution 256x256, where we control all 40 binary attributes at the same time and show four representative ones: Male, Open Mouse, Bangs, Grey Hair.
object seale
object
calor
Figure 6: Latent traversal of FC-StyleGAN with downscaled resolution e = 16 on Isaac3D, where we only control a subset of fine-grained factors: (object scale, lighting intensity, object color, wall color) and here two of them are shown. Please see the appendix for all the four factors.
Latent traversal of FC-StyleGAN. Based on the above observations, we can obtain the latent traversal results of FC-StyleGAN on fine-grained factors, which are shown in Figure Specifically, on Isaac3D. Since Figure 4bb indicates that tne-grained factors ó 16 are of index (0, 4, 5. 7, 8), use its subset: (object scale, light- ing intensity, object color, wall color) as the input while leaving the object shape together with all the coarse-grained factors random nuisances. We
we train FC-StyleGAN with downscaled resolution á
each considered factor in the interpo- lated images changes smoothly without affecting other factors, which implies good disentanglement quality. Because "C-StyleGAN is only applicable for fine-grained control, the one hand, the boarse-grained factors of the interpolated images always stay the same with the corresponding input image, even though they are random nuisances. On the other hand, the object shape, as the only one fine- tne-grained random nuisance, could be different from the corresponding in- put image with some probability. In Figure 6 for example, the object shape stays the same with the input image when interpolating the object scale, while it becomes different from the input image when interpolating the object color. More importantly, compared with the results of AC-StyleGAN in Figure 4a FC-StyleGAN has much better disentanglement quality, when it comes to fine-grained control with supervision from a subset of factors.
observe that, on
Impact of instance normalization. Now we want to show that without instance normalization in FC-StyleGAN, the input image can interference the control of fine-grained code over the output fine styles. The results of FC-StyleGAN with and without instance normalization are shown in Figure 7a where both are trained with downscaled resolution o 32 on Isaac3D, and only a subset of fine- grained factors (lighting intensity, object color, wall color) is used as supervision. For illustration, Figure Talonly shows the the wall color for comparison. Please see the appendix for all three factors. We can see that without instance normalization, latent traversal in FC-StyleGAN cannot change the wall color any more, which instead stays the same with the input image. This confirms previous hypothesis that the instance normalization plays an important role in alleviating the impact of the input image in controlling fine-grained features over generation. Also, the quantitative results in

---

Under review as a conference paper at ICLR 2020
wall rakor
( ld)
w/ FID+6.10, MIG-D. 992, free 10.07
vail
oderr
caker
wio IN: FiD+8.17, MIG-O. 990, 10.10
(a) Impact of instance normalization
(b) Generalization to unseen images
Figure 7: (a) Comparison of FC-StyleGAN with instance normalization (top row) and without instance nor- malization (bottom row). Both are trained with downscaled resolution ø 32 on Isaac3D, where only disentangle subset of fine factors (lighting intensity, object color, wall color) and here only one are shown (see the appendix for all three considered factors). (b) Generalization of FC-StyleGAN by varying the down- scaled resolution & and interpolating the wall color. In the test image, we shift the position of the robot arm to the right-hand hand side, which is also attached with an unseen object (i.c., octahedron).
Figure 7a further confirm the advantages of using instance normalization. In particular, the fact that the reconstruction error Lee becomes larger without instance normalization implies that the strong interference of the input image also degrades the prediction of fine- tne-grained code in the discriminator.
Generalization to unseen images. As image-to-image model, it is natural to ask how FC- StyleGAN generalizes to unseen images. To test its generalization ability, the novel test images are provided as follows: i) we shift the robot position to the right hand side of the camera (instead of standing right in the middle for the training dataset), and ii) we also attach the robot arm with an unseen object. Figure Tigure/7bsshoss 7b]shows the results of interpolating the wall color of the same test image with different downscaled resolutions & E {16, 32, 64). Please see the appendix for results of other test images and different fine-grained factors. As we can see, the wall color keeps changing during its interpolations in each case without affecting other factors, implying good disentanglement quality. Furthermore, the interpolated images each case maintain the new robot position and particularly maintain new object shape (i.c., octahedron) in the case of Φ = 64. The reason the new object shape only maintained at ø not fine-grained at é 64 any more. Therefore, it demonstrates that the disentanglement learning of FC-StyleGAN can generalize well to unseen novel test images. Finally, Figure (7b shows that the generalization results get better in terms of image quality we increase the downscaled resolution 6. As we know, with the larger value of Φ, we can control fewer fine fine-graing the-graiedd factors. Therefore, there exists trade-off between the generalization and controllability in St-SttleGAN, We leave the investigation into how to improve this trade-off in the future work.
is because that the object shape, together with the robot position, is
CONCLUSIONS
In this work, we developed AC-StyleGAN, by combining both advantages of AC-GAN and Style- GAN, to demonstrate that it is possible perfectly disentangle and control different factors of vari- ation in the high-resolution image domain with sufficient supervision. In particular, with only 1% of labelled data, the disentanglement quality is very close the fully supervised case and outperforms the fully unsupervised case by a significant margin, emphasizing the importance of (weak) supervi- sion. To address the performance degradation of AC-StyleGAN in cases where we observe/control only subset of factors, we proposed FC- Stylety,G,N, a new image-to-imagee model for semantically image. We demonstrated the importance
manipulating fine- finennrained factors of a given of instance normalization in FC-StyleGAN, and also showed its good generalization ability to un- seen images. Finally, we proposed two new Faalcrr3 and Isaac3D with higher resolution, better photorealism and richer factors of variation, compared to current disentanglement datasets. We believe that our proposed models, new datasets, along with the useful insights we have gleaned, can benefit the further development of disentangled GANs for the controllable generation of high- fidelity images. For the future work, we would like to explore how to control coarse -raninee factors only in high-resolution images while leaving fine-grained factors as random nuisances.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Mathieu Aubry, Daniel Maturana, Alexei A Efros, Bryan C Russell, and Josef Sivic. Seeing 3d chairs: exemplar part-based 2d-3d alignment using large dataset of cad models. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3762-3769, 2014.
Christopher M Bishop. Latent variable models. Learning in graphical models, pp. 371-403. Springer, 1998.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 1109, 2018.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David Duvenaud. Isolating sources disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp- 2610-208 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, al. Neural scene representation and rendering. Science, 36016394) 1210, 2018.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 2414 2414-2423, 2016
Muhammad Waleed Gondal, Manuel Wüthrich, Dorde Miladinović, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation the real world: disentanglement dataset. arXiv preprint 19X.v:1906.32220 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems (NIPS), 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2(5):6, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Patlern Recognition, pp. 4401-4410, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pp. 2654-2663, 2018.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in neural information processing systems, pp. 2539-2547, 2015.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) December 2015.
11

---

Under review as a conference paper at ICLR 2020
Francesco Locatello, Stefan Bauer, Mario Lucic, Gumnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference Machine Learning, pp. 4114- 4124, 2019a.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. arXiv preprint arXiv:1905.01258,, 2019b.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http://ww...ooeepeeenddddtttaaaa 2017.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. arXiv preprint arXiv:1802.056337 2018.
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Holo- gan: Unsupervised learning of 3d representations from natural images. arXiv preprint arXiv:1904.013326, 2019.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 2642-2651. JMLR. org, 2017.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realis- tic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pp. 3235-3246, 2018.
Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee. Learning to disentangle factors of variation with manifold interaction. In International Conference on Machine Learning, pp. 1431- 2014.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.0802. 2016.
Taihong Xiao, Jiapeng Hong, and Jinwen Ma. Elegant: Exchanging latent encodings with gan for transferring multiple face attributes. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 172-187, September 2018.
Shunyu Yao, Tzu Ming Hsu, Jun- Yan Zhu, Jiajun Wu, Antonio Torralba, Bill Freeman, and Josh Tenenbaum. 3d-aware scene manipulation via inverse graphics. Advances in Neural Informa- fion Processing Systems, pp. 1887-1898, 2018
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S*L: Self-supervised semi- supervised learning. arXiv preprint Xxt5://90.00670 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
Jun-Ya Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
12

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 MORE DETAILS OF THE PROPOSED TWO DATASETS
A.1.1 EXAMPLES THE ISAAC3D DATASET
object shape
robof x-movement
robot y-movement
camera height
object scale
lighting intensity
lighting y-dir
object color
wall color
Figure 8: Examples in the Isaac3D dataset where we vary each factor of variation individually.
13

---

Under review as a conference paper at ICLR 2020
A.1.2 EXAMPLES IN THE FALCOR3D DATASET
lighting intensity
lighting x-dir
lighting y-dir
it
lighting z-dir
R
camera x-pos
PS
T
y-pos
PISP
PSS
TIST
P
camera Z-pos
Figure 9: Examples in the Falcor31 dataset where we vary each factor of variation individually.
14

---

Under review as a conference paper at ICLR 2020
A.2 IMAGE RECONSTRUCTIONS WITH THE SAME META CODE IN AC-STYLEGAN
real images
fake images
Image reconstruction random samples of Isaac3D
R !
real images
PSSy
MSD
fake images
(b) Image reconstruction on random samples of Falcor3D
Figure 10: Image reconstruction random samples of Isaac3D and Falcor3D in AC-StyleGAN with full supervision, where we can see that the generated fake images match well with real images by using the same meta code as the generator input, confirming the semantic correctness of the model.
15

---

Under review as a conference paper at ICLR 2020
A.3
MORE RESULTS ON LATENT TRAVERSAL OF AC-STYLEGAN
object shape
robot X-move.
robot y-move.
camera height
object scale
lighting intensity
lighting y-dir
object
color
wall color
Figure 11: Latent traversal results of AC-StyleGAN with full supervision on Isaac3D for all the fac- tors. Images in the first column (marked by red box) randomly sampled real images of resolution 512x512 and the rest images in each row are their interpolations, respectively, by uniformly varying the given factor from to 1. We without affecting other factors, and the interpolated images in each row visually look almost the same with their input image except the considered varying factor. Also, the image quality does get worse during the interpolations.
that each factor changes smoothly during its interpolation
16

---

Under review as a conference paper at ICLR 2020
lighting intensity
lighting x-dir
lighting y-dir
PSU
lighting z-dir
PE
PISP
P
camera X-pos
7
PS
PRS
y-pos
PSU
PSU
PSU
USJ
PSU
camera z-pos
Latent traversal results of AC-StyleGAN with full supervision on Falcor3D for all the factors. Images in the first column (marked by red box) are randomly sampled real images resolu- tion 1024x 1024 and the rest images in each row are their interpolations, respectively, by uniformly varying the given factor from 0 1. We can see that each factor changes smoothly during its inter- polation without affecting other factors, and the interpolated images in each row visually look very similar their input image except the considered varying factor. Also, the image quality does worse during the interpolations.
Figure 12:
17

---

Under review as a conference paper at ICLR 2020
A.4
LATENT TRAVERSAL RESULTS AC-STYLEGAN WITH 1% LABELLED DATA
object shape
robot x-move.
robot y-move.
camera height
object scale
lighting intensity
lighting y-dir
object color
wall color
Latent traversal results of AC-StyleGAN with semi-supervision (a = 0.01) on Isaac3D for all the factors. Images in the first column (marked by red box) randomly sampled real images of resolution 512x512 and the rest images in cach row their interpolations, respectively, by uniformly varying the given factor from affecting other factors except the entanglement between the object shape and camera height. Also, except the considered varying factor, the interpolated images in each row visually look similar to their input image with small shifts sometimes. It implics reasonably good disentanglement quality and semantic correetness in the AC-StyleGAN with 1% labelled data. Similarly, image quality does not get worse during the interpolations.
Figure 13:
We can see that in most cases, each factor changes smoothly during its interpolation without
18

---

Under review as a conference paper at ICLR 2020
A.5 RANDOMLY SAMPLED IMAGES OF DISENTANGLED VAEs ON THE DOWNSCALED ISAAC3D DATASET
(a) 3-VAE (FID=120.3)
(b) FactorVAB (FID=358.2)
(c) 3-TCVVE (FID=143.8)
The randomly sampled images disentangled VAEs on the Isaac3D dataset resolution 128x 128, where we use the same network architectures as in Locatello et al. (2019a). We can see that compared with AC-StyleGAN, (i) the generated images tend to be quite blurry and of low quality, (ii) the generated images fail cover all the variations in the dataset. The results of disentangled VAEs also demonstrate that our proposed datascts serve as a new disentanglement challenge, particular regarding the much higher resolution, and larger variation of factors.
Figure 14:
A.6 MORE LATENT TRAVERSAL RESULTS OF AC-STYLEGAN CELEBA
Voung
Straight
Wraring
Beadhy Eysbrows
Figure 15: More Latent traversal results AC-StyleGAN on CelebA with resolution 256x256, where we control all 40 binary attributes at the same time. We can see that AC- StyleGAN is capable of controlling most attributes. Some attributes are more difficult to change over interpolations, such Mustache on the female faces. We argue that this is because these attributes have low frequency in the CelebA dataset.
19

---

Under review as a conference paper at ICLR 2020
A.7
MORE RESULTS ON ONLY CONTROLLING A SUBSET OF FACTORS IN AC-STYLEGAN
robot x-mooe,
robot y-move.
object scale
lighting y-dir
object color
latent
(a) Isaae3D
ISU
lighting
intensity
lighting x-dir
camera x-pos
(b) Falcor3D
Latent traversal results of AC-StyleGAN for only controlling a subset of factors on a) Isaac3D: (robot zz-movement, robot y-movement, object scale, lighting y-dir, object color), b) Falcor3D: (lighting inten- sity, lighting z-dir, camera z-pos). The other factors in each dataset will be considered presumably captured by latent z. can see that the disentanglement quality and semantic correctness are both getting worse instead. For example, interpolating the lighting intensity Falcor3D also changes the lighting directions and camera positions.
Figure 16:
random nuisances.
20

---

Under review as a conference paper at ICLR 2020
A.8
MORE RESULTS ON IDENTIFYING FINE-GRAINED FACTORS
robor 1-01
CAmAra hright
<bject
lobcing inteaaily
lighligg y-dle
object
color
wall color
(a) 6== 8
object
robot S-mase.
robol
beghht
sbjjct
lighting Intcesity
fghhtng
y-dir
object
color
odder
(b) á
Figure 17: Latent traversal results of FC-StyleGAN with full supervision Isaac3D for all the factors where we show different downscaled values ò E {8, 64). The factors that cannot be changed by the interpolations (i.e., not (ne-grained) are highlighted by blue boxes. For example, lighting y- y-dir are NOT fine- tininer while if 6 = 64, only the lighting intensity and wall color are fine- grained.. Note that the results are consistent to those in Figure 4b.
only camera height and
21

---

Under review as a conference paper at ICLR 2020
A.9
MORE RESULTS ON THE LATENT TRAVERSAL OF IC-STYLEGAN
object scale
lighting intensity
object color
color
Figure 18: Latent traversal of FC-StyleGAN with full supervision on Isaac3D for a subset of fine-grained factors: (object scale, lighting intensity, object color, wall color). We can that each factor changes smoothly during its interpolation without affecting other factors, and the interpolated images in each almost the same with their input image except the considered varying factor and another fine enaiinaieed factor: object shape (as a random nuisance). Also, the image quality does not get worse during the interpolations.
visually look
lighting
intensity
lighting x-dir
lighting y-dir
ASJ
lighting z-dir
Figure 19; Latent traversal of FC-StyleGAN with full supervision on Falcor3D for all fine- fine-grained factors: (lighting intensity, lighting dir, lighting y-dir, lighting z-dir). We can see that each factor changes smoothly during its interpolation without affecting other factors, and the interpolated images in each row visually look almost the same with their input image except the considered varying factor. Also, the image quality does not get during the interpolations.
22

---

Under review as a conference paper at ICLR 2020
A.10
MORE RESULTS ON THE IMPACT OF INSTANCE NORMALIZATION IN FC- -C-SSTYEGAA
lighting intensity
object color
FID-6.16, MIG 992, 0000
lighting Intensity
object
wall color
FID=8.17, 17, MIG=0.990, 1rec=0.10
(a) Impact of instance normalization
Comparison of FC-StyleGAN with instance normalization (top row) and without instance nor- malization (bottom row). Both are trained with downscaled resolution ø 32 on Isaac3D, where we only disentangle a subset of fine factors (lighting intensity, object color, wall color). We can see that FC-StyleGAN with instance normalization smoothly change each of the factors of variation over interpolation. However, FC-StyleGAN without instance normalization cannot change the lighting intensity and wall color at all.
Figure 20:
23

---

Under review as a conference paper at ICLR 2020
A.11
MORE RESULTS ON THE GENERALIZATION OF FC-STYLEGAN
wall color
16)
wall color
( 32)
wall color 64)
(a) Interpolating the wall color
lighting intensity
(Φ 16)
lighting intensity 32)
lighting intensity
(Φb = 64)
(b) Interpolating the lighting intensity
Figure 21: Generalization of FC-StyleGAN by varying downscaled resolution and interpolating one of the fine iineggrineed factors. In the test image, we shift the robot position to the right-hand side, which is also attached with an unseen object (i.e., octahedron). We can see that in FC-StyleGAN with different downscaled resolutions, the considered factor keeps changing during its interpolations. Furthermore, the interpolated im- object shape (i.c., octahedron)
ages in each the case & = 64. The reason why the new object shape only maintained at á = 64 is beacuse the ject shape, together with the robot position, is not fine- tieeggrined at é = 64 any more. Therefore, demonstrates that the disentanglement learning of FC-StyleGAN can generalize well to unseen novel test images.
maintain the robot position and particularly maintain
24


Reviews:
[{"review_id": "HJxsmxLE9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Disentangled GANs for controllable generation of High-Resolution Images introduces two new high resolution synthetic scene datasets for studying disentanglement in generative models and benchmarks two Style-GAN based architectures for controllable generation on these datasets. The datasets, though still synthetic, provide a significant quality boost over some of the simpler toy datasets previously studied in the disentanglement literature. A variety of experiments are conducted looking at 3 metrics (FID, MIG, and latent reconstruction) as well as qualitative analysis of samples. The paper studies how the performance of these architectures varies with hyperparameter and design decisions. The authors demonstrate that AC-StyleGAN performs well in fully supervised settings achieving the desired conditional generation, but, when controlling only a subset of factors, does not correctly disentangle. To address this, the author’s other architecture modification, FC-StyleGAN an image to image model is demonstrated to improve performance in this setting. The paper provides the reader with a useful overview of the behavior of the two models on these two datasets.\n\nMy rating is weak reject. While the paper has a variety of contributions (most notably introducing two new datasets and modifications of StyleGAN) and interesting results (such as relatively high performance with only 5% labeled data and the disentanglement issues when controlling a subset of factors), the core contributions of new datasets and architectures are not validated or analyzed rigorously enough.\n\n1) No prior work is used as baselines in order to compare / validate the newly introduced architectures. The authors dismiss prior work in the introduction but do not provide any direct evidence that prior work is unable to handle the datasets introduced in the paper. In order to consider acceptance based on the value of the architectures proposed in the paper, there should be some direct evidence that the proposed Style-GAN modifications AC/FC, outperform prior architectures. One particular choice which is unclear to the reviewer is why AC GAN was chosen over the Projection Discriminator of Miyato and Koyama 2018 which demonstrated significantly better results than AC GAN and was adopted by major followup work such as BigGAN.\n\n2) Given the core contributions of new architectures for disentanglement, the lack of any results on real, non synthetic, datasets in order to validate these (for instance, CelebA) which significant prior work on disentanglement has been evaluated on is an unfortunate omission. It is unclear how the introduced architectures handle complex natural image distributions with the much more diverse sources of natural variation they contain.\n\n3) The authors claim their method is effective in the semi-supervised setting and demonstrate this by showing relatively strong performance in a low data regime. This demonstrates the importance of the third term in the loss function, but as other work for semi-supervised learning has shown (Oliver et al 2018) care must be take to properly attribute the performance of a semi-supervised algorithm to the actual semi-supervised components and purely supervised baselines are often quite competitive. The addition of an ablation / analysis demonstrating the contribution of the second term in settings where labels are also available would strengthen the author's claims and fully demonstrate the model is an effective semi-supervised learner.\n\nAdditional comments:\n\nThe datasets introduced in the paper do seem like potentially valuable contributions to the community - more discussion on the motivations behind their creation, the differences with prior work, the open difficulties / challenges, as well as the recommended evaluation protocols could add to their value.\n\nThe authors could more clearly motivate the work / applications the paper is interested in and how each contribution / experiment fits in with this. Without a clear sense of the authors mission / goals with the work, it feels a bit difficult to interpret the results. \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1ldFEHFdH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper is reasonably clear (though see some of my detailed comments on writing below).\nThe idea seems well-motivated and somewhat new though not revolutionary,\nand the new data sets are nice (though see comments below about how I'm not qualified to evaluate them),\nbut I don't understand why the proposed algorithm wasn't evaluated on existing data sets as well,\nand I don't understand why it wasn't compared against other algorithms that purport to do the same thing.\nBarring certain exceptions (again see detailed comments), it seems like those two things are things\nwe ask of essentially all machine learning paper submissions, and I don't see why this paper is different?\nI am open to being persuaded, but as of now I can't recommend acceptance.\n\nDetailed Comments:\n\n> More importantly, only using 5% of the labelled data significantly improves the disentanglement quality.\nHard to parse.\nDoes this mean using only 5% of data is better for disentangling than using 100%?\n\n\n>  Generative adversarial networks (GANs) (Goodfellow et al., 2014) have achieved great success at generating realistic images, such as StyleGAN\nStrange sentence - styleGAN is not itself a realistic image.\n\n\n>  the controllable generation of high-resolution images is possible\nNit: i would cut the 'the'\n\n\n> which characterizes how significant that the model can change a factor.\nWhat does this mean?\n\n\n> present a state-ofthe-art challenge\nWhat does it really mean for a challenge to be state of the art?\n\n> that enables conditional generation of high-fidelity images\nPretty minor nit, but I don't think it makes sense to refer to samples themselves as high fidelity.\nHigh fidelity to what?\nIt makes a little more sense to refer to a trained generator as having high fidelity to the training data set,\nbut TBH I still don't even like the phrase in that context. \n\nFig 1 is good. \nI felt like I could understand what's going on mostly from looking at the figure, which is nice.\n\nThe section surrounding eqs 1 and 2 is a little hard to read for me. \nLots of single letter variable names and it's hard to keep them all in my head when I read the equation.\nI'd replace e.g. c_r with \\text{code}_r and so forth.\n\n> Note that AC-StyleGAN reduces to an InfoGAN variant in the special case\nThis is helpful.\n\n> AC-StyelGAN\n\n>  FCStyleAGN\n\n> by symmetry\nI don't follow this part.\n\n\n> while its high-resolution blocks accounts for fine styles\nI feel like it's worth making a distinction between high frequency bits of an image and non-essential (or nuisance variables or whatever) bits.\nThey often are the same, but not always (it really just depends how close up your photo is, right?) but this technique is really separating high-freq from low-freq, IIUC.\n\n>  handling complex high-fidelity images\nAgain, I just don't feel like this phrase makes any sense.\n\nRe eq 3:\nI guess you can call this the 'interpolation variance'\nif you want, but it's not really a new thing.\nYou're just getting discretized measurements of the Jacobian of the mapping from code to predicted code, and I don't really think in a way that fits with the\nintuition you describe.\nThere are a few things I can think of that are weird about this measurement, but here's just one:\nI might have a dimension of the code that causes one really peaky change at one point, and then the change quickly reverts.\nIf i measure the variance of the output, I won't see much (in fact, depending on how you discretize, maybe I'll miss it altogether),\neven though the fact that the change is peaky doesn't say anything about the 'importance' of the change.\nI think you want something more like a line integral.\nIt's possible I'm misunderstanding something about the description of this, however.\n\nRe: the experiments:\nI like the new data sets, although I'm not familiar enough w/ the robotics literature to know whether they add something really new.\nWhat I don't understand is why there's not really any attempt to do either of:\na) compare the models from the paper to existing models \nb) evaluate the models from the paper on existing data-sets?\n\nI understand that sometimes one comes up with a thing that does something totally new,\nand then reviewers will (maybe unjustifiably) still insist that the some kind of comparison be made even though it doesn't make sense,\nbut that doesn't seem to be what has happened here?\nThis paper proposes a new algorithm for synthesizing images and controlling various attributes of the images, but this has certainly been done in the past,\nso why can't your technique be compared to those prior techniques?\n\nSimilarly, there are lots of data sets that people are already familiar with that you could have evaluated these models on, right?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1gjLKW0FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== A. Summary ===\n\nThis paper proposes to train a new conditional GAN model that allows for controllable image generation by changing the input factors of variations (e.g. object color).\nThe supervised labels for the controllable attributes are obtained from a 3D renderer.\nThat is, the work combines the recent StyleGAN (that learns to generate images with disentangle latent vectors in an unsupervised manner) with AC-GAN (a clas-conditional GAN but here class information is replaced by the attribute information that we want to control).\nThe resultant AC-StyleGAN has essentially two latent vectors, one trained unsupervised and one trained with supervised labels.\n\nThe proposed GANs were thoroughly tested with different factors of variations (lighting, camera, objects) and on two different datasets self-contructed via 3D renderer.\nThe work is a solid demonstration that GANs can be used to synthesize images with fine-grained and coarse controllability if we have supervision signals!\nThe authors also released the anonymous code (which is a plus!).\n\n\n=== B. Decision ===\n\nWeak Reject.\n\nI voted for Weak Reject because this paper presents a fairly incremental advance over what has been done (e.g. HoloGAN, StyleGAN, AC-GAN).\nThe impact of the unsupervised work (e.g. HoloGAN or StyleGAN) is much higher since they are generally applicable to real data without labels.\nHere, although reasonable, the demonstration was done on a relatively small-scale 3D synthetic data (where there is only one scene and one object being manipulated).\nTherefore, the claim that only 5% of supervised labels is required may not carry over to larger-scaled datasets with larger scene variability.\nPlus, I don't see any baseline whatsoever being compared with the proposed methods here. \n\n\n=== C. Suggestions for Improvement ===\n\nI have no problems with the novelty or idea of the work.\nFor me, the key problem with this work is the low impact or significance.\n\nSome suggestions for showing the impact of this work:\n- Show how your pre-trained GANs can be fine-tuned or transferred to the real data where we don't have labels.\n- You could also plug in your pre-trained GANs to a separate synthetic-to-real image translation model to show that we could indeed learn to control these factors of variations of the real images. Hopefully, would the above setups yield better results than HoloGAN or StyleGAN?\n- Clarify the main focus points of this paper and try to substantiate the result. The author wrote \"Our work extends the above works by scaling up the disentanglement learning to high- resolution images, and emphasizing the importance of supervision in controllable generation.\" <---- but (1) high-res images here are synthetic and limited in scene variability; (2) the second part is expected given previous work.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a model combining AC-GAN and StyleGAN for semi-supervised learning of disentangled generative adversarial networks. It also proposes new datasets of 3d images as benchmarks. The main claim is that the proposed model can achieve strong disentanglement property by using 1-5% of the annotations on the factors of variation. The technical contribution is moderate but the architecture itself is not highly novel. While the proposed method seems to work for controlled/synthetic datasets, overall technical contribution seems incremental and it's unclear whether it can perform well on larger-scale, real datasets. The experimental results on CelebA don't look convincing enough.

Decision: Reject