Year: 2019

Abstract:
Generative adversarial networks (GANs) have achieved great success at generating realistic samples. However, achieving disentangled and controllable generation still remains challenging for GANs, especially in the high-resolution image domain. Motivated by this, we introduce AC-StyleGAN, a combination of ACGAN and StyleGAN, for demonstrating that very limited supervision is sufficient to benefit the generic disentanglement learning of high-resolution images. Specifically, by only using 1% of the labelled data, the disentanglement quality is very close to the fully supervised case and significantly outperforms the unsupervised alternative. Inspired by the observed separation of fine and coarse styles in StyleGAN, we then extend AC-StyleGAN to a new image-to-image model called FCStyleGAN for semantic manipulation of fine-grained factors in a high-resolution image. In experiments, we show that FC-StyleGAN performs well in only controlling fine-grained factors, with the use of instance normalization, and also demonstrate its good generalization ability to unseen images. Finally, we create two new datasets – Falcor3D and Isaac3D with higher resolution, more photorealism, and richer variation, as compared to existing disentanglement datasets.

heading: 1 INTRODUCTION
text: High-resolution controllable generation is an important component in many applications, such as image editing (Yao et al., 2018), 3D scene understanding (Eslami et al., 2018) and inverse graphics (Kulkarni et al., 2015). Generative adversarial networks (GANs) (Goodfellow et al., 2014) have achieved great success at generating realistic images, of which two representatives are StyleGAN (Karras et al., 2019) for unconditional generation and BigGAN (Brock et al., 2018) for class conditional generation. However, the controllable generation is still a challenge for state-of-the-art GANs, especially with high-resolution images. For instance, StyleGAN cannot be directly used to synthesize high-fidelity human faces by specifically controlling skin color or eye size without affecting other face attributes. Also, BigGAN is not able to control the hair color or length of a dog image without changing other features.
Disentanglement of various factors allows us to independently control the variations across all the factors. But this is not easy to learn in GANs without further modifications such as adding regularization to encourage better disentanglement. For example, InfoGAN (Chen et al., 2016) proposes maximizing mutual information between latent code and its reconstruction for an unsupervised disentanglement. Most recently, Locatello et al. (2019a) has showed that unsupervised disentanglement learning is impossible without model inductive bias or supervision. In that light, several recent works attempt to learn disentanglement in GANs by introducing a strong model bias (Nguyen-Phuoc et al., 2019). However, these works are restricted to a specific domain (e.g. 3D rotations only) and usually difficult to scale up to higher resolution, mainly due to the intrinsic limitations of the model biases themselves. This motivates us to study the impact of limited supervision in a generic disentanglement learning with GANs, especially in the more challenging high-resolution image domain.
Main contributions: By combining the advantages of AC-GAN (Odena et al., 2017) and StyleGAN, we introduce AC-StyleGAN, and demonstrate that, very limited supervision is sufficient: just 1% of the labelled data significantly improves the disentanglement quality, as compared to the unsupervised alternatives. However, if we are only interested in controlling a subset of factors (effectively treating the rest as random unobserved nuisance variables), we find that the disentanglement qual-
ity of AC-StyleGAN degrades significantly. We believe this is due to the latent nuisance factors strongly confounding the observed factors, a common and difficult problem in high-dimensional partially observed latent variables models (Bishop, 1998).
To address this, we propose FC-StyleGAN, a new image-to-image model that adds controllable finegrained factors along with a super-resolution process. This is inspired by the separation of fine and coarse styles, observed in StyleGAN. FC-StyleGAN enables semantic manipulation of fine styles in high-resolution images, without a commonly-used encoder-decoder structure.
Finally, we create two new high-quality datasets – Falcor3D and Isaac3D – that present a new challenge for controllable generation in terms of image resolution, photorealism, and richness of style factors, as compared to existing disentanglement datasets such as 3D Chairs (Aubry et al., 2014), dSprite (Matthey et al., 2017) and MPI3D (Gondal et al., 2019).
Thus, we propose new semi-supervised GAN architectures that enable a generic disentanglement learning of high-resolution images as well as new high-quality disentanglement datasets.

heading: 2 RELATED WORK
text: Disentanglement learning. Learning generic disentangled representations in an unsupervised way has attracted a lot of attention. Two representative models are InfoGAN (Chen et al., 2016) and β-VAE (Higgins et al., 2017). The nature of unsupervised disentanglement learning does not guarantee that the learned disentangled factors are semantically meaningful without an additional inductive bias or supervision (Locatello et al., 2019a; Nguyen-Phuoc et al., 2019; Locatello et al., 2019b). Another line of work aims to learn disentangled representations via supervised learning wherein factors are observed variables (Kulkarni et al., 2015; Reed et al., 2014; Xiao et al., 2018; Locatello et al., 2019b). However, little work has systematically investigated the necessity of very limited supervision to generic disentanglement learning. The analysis of Locatello et al. (2019b) only applies to disentangled VAEs on low-resolution datasets. Instead, our work tries to answer a more practical question: with very little supervision on more complex and higher-resolution datasets, can we still well disentangle the factors of variation while maintaining the high generation quality?
Deep image manipulation. Deep neural networks have enabled various image editing tasks, such as style transfer (Gatys et al., 2016), image-to-image translation (Zhu et al., 2017), automatic colorization (Zhang et al., 2016) and 3D-aware attribute editing (Yao et al., 2018). None of the above methods except the 3D-SDN (Yao et al., 2018) has dealt with the semantic manipulation of multiple attributes for scene images. Different from the 3D-SDN and other previous works on attribute editing that have been mostly focused on the 3D geometry manipulation, our proposed FC-StyleGAN is designed to semantically manipulate the fine-grained factors instead, such as lighting conditions and object colors. Furthermore, FC-StyleGAN does not apply an encoder-decoder structure as commonly used in previous works for semantic manipulation (Yao et al., 2018), instead, it adds controllable fine-grained factors along with a super-resolution process, which turns out to perform well in editing fine styles of high-resolution images with a good generalization ability.

heading: 3 MODELS

heading: 3.1 BACKGROUND ON AC-GAN AND STYLEGAN
text: AC-GAN. AC-GAN (Odena et al., 2017) is a variant of class conditional GANs. In the AC-GAN, the generator inputs are a latent z (or called random noise) and the class label. The input to the discriminator is the real or fake image, whilst its output is the probability that the image is real and the prediction of the class label. Odena et al. (2017) has shown that this modification to the standard conditional GAN formulation produces more realistic images and appears to stabilize training.
StyleGAN StyleGAN (Karras et al., 2019) is a state-of-the-art GAN architecture for unsupervised image generation, particularly for high-fidelity human faces. Basically, StyleGAN comprises a mapping network whose role is to map the latent z to an intermediate space, which then controls the styles at each convolutional layer in the synthesis network via adaptive instance normalization (AdaIN) (Ulyanov et al., 2016; Huang & Belongie, 2017). StyleGAN also enables the separation of fine-grained and coarse-grained features. For example, modifying the styles of low-resolution blocks affects only coarse-grained features (e.g. pose and eyeglasses), while modifying the styles of high-resolution blocks affects only fine-grained features (e.g. color scheme and microstructure).

heading: 3.2 AC-STYLEGAN FOR CONTROLLABLE IMAGE SYNTHESIS
text: We introduce AC-StyleGAN, a combination of AC-GAN and StyleGAN, that enables conditional generation of high-resolution images. As shown in Figure 1a, the generator in AC-StyleGAN conditions on a meta code by simply concatenating the meta code, a vector representing all the factors of variation, with the latent z. The discriminator in AC-StyleGAN now provides two outputs, the classification of real/fake and the prediction of meta code. And finally, the outputs of the mapping network – the conditioned styles – will modulate each block in the synthesis network via AdaIN. We choose AC-GAN, instead of other conditional GAN methods, such as cGANs (with Projection Discriminator) (Miyato & Koyama, 2018), mainly because only AC-GAN has the property of reconstructing meta code in the discriminator, which can easily be extended to a semi-supervised disentanglement learning framework.
More formally, let x(n)r denote the n-th image from the set of N real images and let c (n) r be the corresponding meta code, randomly sampled from the dataset. Let c(n)f denote a random meta code sampled from the label distribution, and let D(·) and G(·) denote the discriminator and generator neural networks. We also assume s(n)r and s (n) f represent the real and fake classification logits in the discriminator, respectively, and ĉ(n)r and ĉ (n) f denote the predictions of c (n) r and c (n) f , respectively. From Figure 1a, we have
(s(n)r , ĉ (n) r ) = D(x (n) r ) and (s (n) f , ĉ (n) f ) = D(G(c (n) f , z (n))) (1)
Since we are aiming at achieving both high image quality and good controllability, we define a semi-supervised loss function for the generator and discriminator in AC-StyleGAN as follows,
L(G)(G,D) = 1
N N−1∑ n=0 lGAN(s (n) r , s (n) f )︸ ︷︷ ︸
GAN loss
+γ ‖ĉ(n)f − c (n) f ‖︸ ︷︷ ︸
unsupervised disentanglement
L(D)(G,D) = − 1 N N−1∑ n=0 lGAN(s (n) r , s (n) f )︸ ︷︷ ︸
GAN loss
+γ ‖ĉ(n)f − c (n) f ‖︸ ︷︷ ︸
unsupervised disentanglement
+ Nη(n)γ∑N−1 n=0 η (n) ‖ĉ(n)r − c(n)r ‖︸ ︷︷ ︸
supervised disentanglement
(2)
which consists of three terms: the GAN loss term, unsupervised disentanglement term and supervised disentanglement term. For the GAN loss, we apply the non-saturating loss plus gradient penalty in the discriminator, as in StyleGAN. For both the unsupervised and supervised disentanglement terms, we simply use l2 norm of the reconstruction losses.
Furthermore, we introduce two coefficients γ and η(n) in the above loss function: (i) the disentanglement coefficient γ ∈ (0,∞) balances the trade-off between the generation quality and disentanglement performance; (ii) the coefficient η(n) is the label mask denoted by η(n) = 1{α(n)<α}
where α(n) i.i.d.∼ Uniform[0, 1] and α ∈ [0, 1] is a hyperparameter. Accordingly, the hyperparameter α controls the fraction of labelled data that will be used for supervision (fully unsupervised in the disentanglement learning if α = 0, and fully supervised if α = 1).
Note that we sample all the α(n)’s before training so that the label mask η(n) remains unchanged during training to reflect the real semi-supervised case. We also apply a rescaling operation by multiplying the supervised disentanglement term in Eq. (2) with the rescaling coefficient N∑N−1
n=0 η (n) to take an average of effective supervised signals within a minibatch. Finally, note that AC-StyleGAN reduces to an InfoGAN variant in the special case of α = 0 (fully unsupervised), when an appropriate reconstruction loss is used.

heading: 3.3 FC-STYLEGAN FOR SEMANTIC MANIPULATION OF FINE STYLES
text: While we apply AC-StyleGAN with supervision over all observed factors for demonstrating a controllable generation of high-resolution images in GANs, in a potentially more realistic case where we are only interested in controlling a subset of factors (effectively treating the rest as random unobserved nuisance variables), the disentanglement quality of AC-StyelGAN drops significantly, as shown in Section 4.2. Furthermore, AC-StyleGAN cannot be directly applied to manipulate an existing high-resolution image. Therefore, we propose a new image-to-image model called FCStyleAGN (i.e., Fine-grained Controlled StyleGAN) for only controlling fine-grained factors.
Inspired by the observations that the lower resolution blocks in the StyleGAN generator learn the coarse-grained features while its high-resolution blocks accounts for fine styles, the generator in FC-StyleGAN does not contain the lower-resolution blocks. As shown in Figure 1b, the generator instead takes the real image as one of its inputs by downscaling it to a lower resolution φ (e.g., φ = 32 in Figure 1b). After that, it generates the high-resolution image by only modulating the fine-grained code into higher resolution blocks. Also, the discriminator in FC-StyleGAN predicts the value of fine-grained code from the block with resolution φ, instead of the last output block. The intuition is that similar to the observations in the generator, lower resolution blocks in the discriminator have less relationship with fine-grained styles. So it makes more sense to directly predict the meta code from higher resolution blocks. Note that the value of φ varies from 4 to image size, determining what factors are identified as fine-grained. Finally, the loss function in FC-StyleGAN is the same with Eq. (2) in AC-StyleGAN, indicating a semi-supervised approach.
During training of FC-StyleGAN, the generator takes the downscaled image as is, and just embellishes it by increasing the resolution and generating the missing pixel-level detail. As it does not need to learn the coarse-grained features any more, presumably it will have much easier time handling complex high-resolution images. Furthermore, since the generator is still style-based in higher-resolution blocks with fine-grained code as its input, we will retain the control over fine styles. However, there are two caveats in FC-StyleGAN. The first caveat is that as an image-toimage model, the original fine-grained factors in the input image may interference the control of fine code over the fine styles in the output image. In such sense, we emphasize the role of inductive biases on the model – the use of instance normalization in the generator. Because the output of instance normalization preserves the spatial structure of image content by only normalizing fine styles (Ulyanov et al., 2016), we argue that the original fine styles in the input image could be washed away by instance normalization, and thus the fine code will fully control the fine-grained factors.
The second caveat is how to identify fine-grained factors in a dataset before training FC-StyleGAN. To this end, we use the meta code of all factors as the input of FC-StyleGAN with different downscaled resolutions φ. For each φ, we introduce a new term – interpolation variance of each factor ci, denoted by βi(φ), based on which the factor ci is defined as fine-grained at φ if its interpolation variance satisfies that βi(φ) > β0, where β0 is a pre-defined threshold. The interpolation variance βi(φ) is calculated as follows: Given N real images, we do the latent traversal over each factor c (n) i of an image I(n) to get S interpolated (fake) images {Î(n)i,0 , · · · , Î (N) i,S−1}. Each interpolated image Î (n) i,s is then fed into the discriminator to get a predicted factor ĉ (n) i,s . Thus, the interpolation variance is the average variance of ĉ(n)i,s over the interpolation s-dimension, which is
βi(φ) = 1
N N−1∑ n=0 Vars [ ĉ (n) i,s ] (3)
Intuitively, low interpolation variance βi(φ) means that changing the factor ci does not affect much the generated images, which further implies the factor ci should not be considered as fine-grained, since FC-StyleGAN with downscaled resolution φ cannot control it any more.

heading: 4 EXPERIMENTS
text: In this section, we first introduce two new datasets – Isaac3D and Falcor3D. We then evaluate the performance of AC-StyleGAN and FC-StyleGAN on both datasets, focusing on three aspects: disentanglement quality, semantic correctness and image quality. Quantitatively, we use Frechet Inception Distance (FID) (Heusel et al., 2017) to measure image quality, reconstruction error (i.e., lrec = ‖ĉ(r) − c(r)‖) to measure semantic correctness, and MIG (Chen et al., 2018) to measure disentanglement quality. Qualitatively, we apply the latent traversals to visually evaluate all three aspects as well. Due to space limitations, we only show results on Isaac3D in this section; results on Falcor3D are quite similar to those on Isaac3D, and are available in the appendix.

heading: 4.1 CREATION OF HIGH-QUALITY DISENTANGLEMENT DATASETS
text: Current disentanglement datasets, such as 3D Chairs (Aubry et al., 2014), dSprites (Matthey et al., 2017) and MPI3D (Gondal et al., 2019), are of low resolution and mostly lack of photorealism. It makes them not suitable as disentanglement benchmarks in the high-resolution image domain. To this end, we propose two new datasets – Falcor3D and Isaac3D, which possess much higher resolution, better photorealism and richer factors of variations, as shown in Table 1.
Falcor3D In the Falcor dataset, there are in total 233,280 images and each has a resolution of 1024x1024. This dataset is based on the 3D scene of a living room, where we can move the camera positions and change the lighting conditions. Each image is paired with a ground-truth meta code, consisting of 7 factors of variation: lighting intensity (5), lighting x-dir (6), lighting y-dir (6), lighting z-dir (6), camera x-pos (6), camera y-pos (6), and camera z-pos (6). Note that the number m behind each factor represents that the factor has m possible values, uniformly sampled in the normalized range of variations [0, 1]. To interpret this, for example, “lighting x-dir (6)” represents the lighting direction moving along the x-axis and “camera z-pos (6)” represents the camera position moving along the z-axis. Also, both factors have 6 values uniformly sampled from [0, 1].
Isaac3D In the Isaac3D dataset, there are in total 737,280 images and each has a resolution of 512x512. This dataset is based on the 3D scene of a kitchen, where we can also move the camera positions and vary the lighting conditions. To further increase the number of variations, we put a robotic arm inside, attached with an object. The robotic arm has two degrees of freedom: xmovement (or horizontal rotation) and y-movement (or vertical rotation). The attached object could change its shape, scale and color. To make the rendered images more photorealistic, each object in the 3D scene has been provided with proper textures. Finally, each image is paired with a groundtruth meta code, consisting of 9 factors of variation: lighting intensity (4), lighting y-dir (6), object color (4), wall color (4), object shape (3), object scale (4), camera height (4), robot x-movement (8), and robot y-movement (5). Similarly, the number m behind each factor represents that the factor has m possible values, uniformly sampled in the normalized range of variations [0, 1].
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 1
Please see the appendix for more detailed descriptions of the two datasets.

heading: 4.2 EVALUATION OF AC-STYLEGAN
text: Latent traversal of AC-StyleGAN. We first show the latent traversal results of AC-StyleGAN on Isaac3D in Figure 2, where we apply full supervision from meta code of all the factors. For illustration, here we only show three factors of variation: (robot x-movement, camera height, lighting y-dir). Please see the appendix for latent traversal of all the factors. Images in the first column (marked by red box) are randomly sampled real images and the rest images in each row are their interpolations, respectively, by uniformly varying each factor from 0 to 1. Unless otherwise stated, this setting applies to all the latent traversal results below. As we can see, each factor in the interpolated images changes smoothly without affecting other factors. Taking “lighting y-dir” as an example, only the direction of the point light gradually moves up from left to right in the bottom row, which can be evidenced by the fact that the light spot moves up from the refrigerator to the wall and the shadow of the cabinet on the wall moves down simultaneously. More importantly, all the interpolated images visually look the same with the corresponding real images in the first column except for the interpolated factors, and the image quality does not degrade over interpolation. Therefore, the latent traversal results demonstrate three good properties of AC-StyleGAN with supervision: high disentanglement quality and good semantic correctness and high generation quality.
Semi-supervised learning. How does AC-StyleGAN behave when given fewer labels? To this end, we vary the supervision coefficient α in Eq. (2) to show the impact of supervision on ACStyleGAN and the results with γ ∈ {1, 10} on Isaac3D are in Figure 3. First, we can see with limited supervision (α ≥ 0.1), the difference between γ = 1 and 10 is quite small. Also, both values of MIG and lrec almost reach their own optimal ones, along with a lower FID. This quantitatively supports the above latent traversal results in Figure 2. As expected, the disentanglement quality and semantic correctness gradually get worse when less labelled data is used. Interestingly, the gap between γ = 1 and 10 first increases and then decreases as α decreases from 0.5 to 0, with the largest
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 1
one at α = 0.01. It means that properly increasing γ is the most beneficial for disentanglement in the case where only a very limited number of labelled data is provided. As we can see, by only using 1% of the labelled data (α = 0.01) and setting γ = 10, the disentanglement quality and semantic correctness are still close to those with full supervision (α = 1), and improve over the unsupervised baseline (α = 0) by a significant margin. Please see the appendix for the latent traversal results of AC-StyleGAN with α = 0.01 and γ = 10. It means adding a very small amount of labelled data into the training dataset could benefit much the disentanglement learning. Finally, in terms of image quality, we observe that larger α tends to result in a lower FID score, which means better disentanglement is not always obtained by sacrificing the generation quality.
Comparisons with different baselines. Current start-of-the-art approaches for generic unsupervised disentanglement learning are mostly VAE-based models, such as β-VAE (Higgins et al., 2017), FactorVAE (Kim & Mnih, 2018) and β-TCVAE (Chen et al., 2018). To this end, we use the unsupervised version of AC-StyleGAN (α = 0) to compare with VAE-based models on (i) the dSprites dataset and (ii) the Isaac3D dsataset, respectively. Note that for evaluations on Isaac3D, we first downscale its resolution to 128x128 because VAEs have difficulties in generating higher-resolution images. We can see from Table 2 that the unsupervised AC-StyleGAN consistently outperforms all the start-of-the-art disentangled VAEs on both datasets. Particularly on the downscaled Isaac3D dataset, disentangled VAEs have much higher FID scores (FID > 100), indicating their poor image generation quality. Please see Appendix A.5 for their randomly sampled images. Interestingly, the disentangled VAEs also fail to capture all the variations in the Isaac3D dataset, supporting the claim that our datasets provide larger variation of factors for disentanglement learning.
Previous work on the semi-supervised learning for classification has shown that a supervised baseline with only the labelled data could be very competitive (Oliver et al., 2018; Zhai et al., 2019). Hence, to demonstrate the effectiveness of actual semi-supervised components for disentanglement, we also compare our semi-supervised method with the supervised baseline, in which we remove the unsupervised disentanglement term in the discriminator loss function of Eq. (2). The results are shown in Table 3. We can see that when only 1% or 5% of lablled data is available, semi-supervised AC-StyleGAN consistently outperforms the supervised baseline in terms of all three metrics: FID, MIG and lrec. Moreover, the advantages of using the unsupervised disentanglement term become larger when we have fewer labelled data.
Latent traversal on 256x256 CelebA. To show the performance of AC-StyleGAN on the realworld data, we also provide the qualitative results on CelebA with resolution 256x256 (Liu et al., 2015). In this experiment, the meta code in AC-StyleGAN has a length of 40 to capture all the attributes in the dataset. Hence, this is more challenging than the previous works on CelebA in terms of controlling all the attributes in a higher resolution (Chen et al., 2016; Higgins et al., 2017; Nguyen-Phuoc et al., 2019). Note that since CelebA only has binary attributes, which commonlyused disentanglement metrics including MIG and FactorVAE score are not applicable to, we focus
on the qualitative results of latent traversal. The results are shown in Figure 5. We can see that AC-styleGAN can smoothly interpolate in the latent space of face attributes and also maintain high image quality. More interesting interpolations can be seen in Appendix A.6, where some attributes are more difficult to change during latent traversal, presumably due to the fact that these attributes have low frequency in the CelebA dataset, such as Mustache on female faces.
Only controlling a subset of style factors. Instead of trying to control all the factors in a dataset, practically speaking we may only be interested in a subset of factors in which case the remaining factors will be considered as random nuisances captured by the latent z. On a first look, it might seem intuitive that the fewer factors of variation we try to control, the better the disentanglement quality we should achieve. However, the latent traversal results of AC-StyleGAN( Figure 4a) show instead that both disentanglement quality and semantic correctness degrade. For example, the camera height changes when interpolating the object scale, and the wall color also varies when interpolating the object scale. Furthermore, the latent z, which is expected to capture the remaining factors: (object shape, camera height, lighting intensity, wall color), can barely change the camera height and wall color (see the appendix for results of other factors). To explain this, we argue that since factors are highly correlated with each other, learning to control one may help a lot in learning to control the others. Without observations on other nuisance factors, it may be very difficult to disentangle the impact of the observed factor with unobserved covariates, especially in the high-dimensional space.

heading: 4.3 EVALUATION OF FC-STYLEGAN
text: Identifying fine-grained factors. As we have discussed, it is a crucial for FC-StyleGAN to quantitatively identify fine-grained factors. Here we calculate the interpolation variance of each factor according to Eq. (3) by setting N = 100, S = 10. Figure 4b shows the interpolation variance of each factor in FC-StyleGAN with downscaled resolution φ varying from 4 to 128 on Isaac3D. To decide the threshold β0, we visually check the latent traversal results and find that the interpolations over a factor ci almost stay the same if βi(φ) < 0.055, which means β0 = 0.055. Please see the appendix for the latent traversal results. We can see that as φ gets larger, more factors have an interpolation variance below the threshold β0, meaning more coarse-grained factors appear. For instance, there are only two coarse-grained factors at φ = 8: camera height and lighting direction, and two more factors become coarse-grained factors at φ = 16: robot x-movement and robot y-movement, and so on. Interestingly, wall color is always fine-grained even at φ = 128 while object color has already become coarse-grained at φ = 64, although both of them are about colors.
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
Latent traversal of FC-StyleGAN. Based on the above observations, we can obtain the latent traversal results of FC-StyleGAN on fine-grained factors, which are shown in Figure 6. Specifically, we train FC-StyleGAN with downscaled resolution φ = 16 on Isaac3D. Since Figure 4b indicates that fine-grained factors at φ = 16 are of index (0, 4, 5, 7, 8), we use its subset: (object scale, lighting intensity, object color, wall color) as the input while leaving the object shape together with all the coarse-grained factors as random nuisances. We can see each considered factor in the interpolated images changes smoothly without affecting other factors, which implies good disentanglement quality. Because FC-StyleGAN is only applicable for fine-grained control, we can observe that, on the one hand, the coarse-grained factors of the interpolated images always stay the same with the corresponding input image, even though they are random nuisances. On the other hand, the object shape, as the only one fine-grained random nuisance, could be different from the corresponding input image with some probability. In Figure 6, for example, the object shape stays the same with the input image when interpolating the object scale, while it becomes different from the input image when interpolating the object color. More importantly, compared with the results of AC-StyleGAN in Figure 4a, FC-StyleGAN has much better disentanglement quality, when it comes to fine-grained control with supervision from a subset of factors.
Impact of instance normalization. Now we want to show that without instance normalization in FC-StyleGAN, the input image can interference the control of fine-grained code over the output fine styles. The results of FC-StyleGAN with and without instance normalization are shown in Figure 7a, where both are trained with downscaled resolution φ = 32 on Isaac3D, and only a subset of finegrained factors (lighting intensity, object color, wall color) is used as supervision. For illustration, Figure 7a only shows the the wall color for comparison. Please see the appendix for all three factors. We can see that without instance normalization, latent traversal in FC-StyleGAN cannot change the wall color any more, which instead stays the same with the input image. This confirms our previous hypothesis that the instance normalization plays an important role in alleviating the impact of the input image in controlling fine-grained features over generation. Also, the quantitative results in
Figure 7a further confirm the advantages of using instance normalization. In particular, the fact that the reconstruction error lrec becomes larger without instance normalization implies that the strong interference of the input image also degrades the prediction of fine-grained code in the discriminator.
Generalization to unseen images. As an image-to-image model, it is natural to ask how FCStyleGAN generalizes to unseen images. To test its generalization ability, the novel test images are provided as follows: i) we shift the robot position to the right hand side of the camera (instead of standing right in the middle for the training dataset), and ii) we also attach the robot arm with an unseen object. Figure 7b shows the results of interpolating the wall color of the same test image with different downscaled resolutions φ ∈ {16, 32, 64}. Please see the appendix for results of other test images and different fine-grained factors. As we can see, the wall color keeps changing during its interpolations in each case without affecting other factors, implying good disentanglement quality. Furthermore, the interpolated images in each case maintain the new robot position and particularly maintain new object shape (i.e., octahedron) in the case of φ = 64. The reason the new object shape is only maintained at φ = 64 is because that the object shape, together with the robot position, is not fine-grained at φ = 64 any more. Therefore, it demonstrates that the disentanglement learning of FC-StyleGAN can generalize well to unseen novel test images. Finally, Figure 7b shows that the generalization results get better in terms of image quality as we increase the downscaled resolution φ. As we know, with the larger value of φ, we can control fewer fine-grained factors. Therefore, there exists a trade-off between the generalization and controllability in FC-StyleGAN. We leave the investigation into how to improve this trade-off in the future work.

heading: 5 CONCLUSIONS
text: In this work, we developed AC-StyleGAN, by combining both advantages of AC-GAN and StyleGAN, to demonstrate that it is possible to perfectly disentangle and control different factors of variation in the high-resolution image domain with sufficient supervision. In particular, with only 1% of labelled data, the disentanglement quality is very close to the fully supervised case and outperforms the fully unsupervised case by a significant margin, emphasizing the importance of (weak) supervision. To address the performance degradation of AC-StyleGAN in cases where we observe/control only a subset of factors, we proposed FC-StyleGAN, a new image-to-image model for semantically manipulating fine-grained factors of a given high-fidelity image. We demonstrated the importance of instance normalization in FC-StyleGAN, and also showed its good generalization ability to unseen images. Finally, we proposed two new datasets – Falcor3D and Isaac3D with higher resolution, better photorealism and richer factors of variation, compared to current disentanglement datasets. We believe that our proposed models, new datasets, along with the useful insights we have gleaned, can benefit the further development of disentangled GANs for the controllable generation of highfidelity images. For the future work, we would like to explore how to control coarse-grained factors only in high-resolution images while leaving fine-grained factors as random nuisances.

heading: A.1.1 EXAMPLES IN THE ISAAC3D DATASET

heading: A.1 MORE DETAILS OF THE PROPOSED TWO DATASETS

heading: A APPENDIX
text: A.1.2 EXAMPLES IN THE FALCOR3D DATASET
A.2 IMAGE RECONSTRUCTIONS WITH THE SAME META CODE IN AC-STYLEGAN

heading: A.3 MORE RESULTS ON LATENT TRAVERSAL OF AC-STYLEGAN
text: NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 1
lighting intensity
object shape
robot x-move.
robot y-move.
camera height
object scale
lighting y-dir
object color
wall color
Figure 11: Latent traversal results of AC-StyleGAN with full supervision on Isaac3D for all the factors. Images in the first column (marked by red box) are randomly sampled real images of resolution 512x512 and the rest images in each row are their interpolations, respectively, by uniformly varying the given factor from 0 to 1. We can see that each factor changes smoothly during its interpolation without affecting other factors, and the interpolated images in each row visually look almost the same with their input image except the considered varying factor. Also, the image quality does not get worse during the interpolations.
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 1
lighting intensity
lighting x-dir
lighting y-dir
lighting z-dir
camera x-pos
camera y-pos
camera z-pos
Figure 12: Latent traversal results of AC-StyleGAN with full supervision on Falcor3D for all the factors. Images in the first column (marked by red box) are randomly sampled real images of resolution 1024x1024 and the rest images in each row are their interpolations, respectively, by uniformly varying the given factor from 0 to 1. We can see that each factor changes smoothly during its interpolation without affecting other factors, and the interpolated images in each row visually look very similar to their input image except the considered varying factor. Also, the image quality does not get worse during the interpolations.

heading: A.4 LATENT TRAVERSAL RESULTS AC-STYLEGAN WITH 1% LABELLED DATA
text: NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 1
lighting intensity
object shape
robot x-move.
robot y-move.
camera height
object scale
lighting y-dir
object color
wall color
Figure 13: Latent traversal results of AC-StyleGAN with semi-supervision (α = 0.01) on Isaac3D for all the factors. Images in the first column (marked by red box) are randomly sampled real images of resolution 512x512 and the rest images in each row are their interpolations, respectively, by uniformly varying the given factor from 0 to 1. We can see that in most cases, each factor changes smoothly during its interpolation without affecting other factors except the entanglement between the object shape and camera height. Also, except the considered varying factor, the interpolated images in each row visually look similar to their input image with small shifts sometimes. It implies a reasonably good disentanglement quality and semantic correctness in the AC-StyleGAN with 1% labelled data. Similarly, the image quality does not get worse during the interpolations.

heading: A.6 MORE LATENT TRAVERSAL RESULTS OF AC-STYLEGAN ON CELEBA

heading: A.5 RANDOMLY SAMPLED IMAGES OF DISENTANGLED VAES ON THE DOWNSCALED ISAAC3D DATASET

heading: A.7 MORE RESULTS ON ONLY CONTROLLING A SUBSET OF FACTORS IN AC-STYLEGAN

heading: A.8 MORE RESULTS ON IDENTIFYING FINE-GRAINED FACTORS
text: A.9 MORE RESULTS ON THE LATENT TRAVERSAL OF FC-STYLEGAN
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 1
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 1
A.10 MORE RESULTS ON THE IMPACT OF INSTANCE NORMALIZATION IN FC-STYLEGAN
A.11 MORE RESULTS ON THE GENERALIZATION OF FC-STYLEGAN

Review summaries:
Disentangled GANs for controllable generation of High-Resolution Images introduces two new high resolution synthetic scene datasets for studying disentanglement in generative models and benchmarks two Style-GAN based architectures for controllable generation on these datasets. The datasets, though still synthetic, provide a significant quality boost over some of the simpler toy datasets previously studied in the disentanglement literature. A variety of experiments are conducted looking at 3 metrics (FID, MIG, and latent reconstruction) as well as qualitative analysis of samples. The paper studies how the performance of these architectures varies with hyperparameter and design decisions. The authors demonstrate that AC-StyleGAN performs well in fully supervised settings achieving the desired conditional generation, but, when controlling only a subset of factors, does not correctly disentangle. To address this, the author’s other architecture modification, FC-StyleGAN an image to image model is demonstrated to improve performance in this setting. The paper provides the reader with a useful overview of the behavior of the two models on these two datasets.