Disentangled GANs for Controllable Generation of High-Resolution Images

1 INTRODUCTION . High-resolution controllable generation is an important component in many applications , such as image editing ( Yao et al. , 2018 ) , 3D scene understanding ( Eslami et al. , 2018 ) and inverse graphics ( Kulkarni et al. , 2015 ) . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have achieved great success at generating realistic images , of which two representatives are StyleGAN ( Karras et al. , 2019 ) for unconditional generation and BigGAN ( Brock et al. , 2018 ) for class conditional generation . However , the controllable generation is still a challenge for state-of-the-art GANs , especially with high-resolution images . For instance , StyleGAN can not be directly used to synthesize high-fidelity human faces by specifically controlling skin color or eye size without affecting other face attributes . Also , BigGAN is not able to control the hair color or length of a dog image without changing other features . Disentanglement of various factors allows us to independently control the variations across all the factors . But this is not easy to learn in GANs without further modifications such as adding regularization to encourage better disentanglement . For example , InfoGAN ( Chen et al. , 2016 ) proposes maximizing mutual information between latent code and its reconstruction for an unsupervised disentanglement . Most recently , Locatello et al . ( 2019a ) has showed that unsupervised disentanglement learning is impossible without model inductive bias or supervision . In that light , several recent works attempt to learn disentanglement in GANs by introducing a strong model bias ( Nguyen-Phuoc et al. , 2019 ) . However , these works are restricted to a specific domain ( e.g . 3D rotations only ) and usually difficult to scale up to higher resolution , mainly due to the intrinsic limitations of the model biases themselves . This motivates us to study the impact of limited supervision in a generic disentanglement learning with GANs , especially in the more challenging high-resolution image domain . Main contributions : By combining the advantages of AC-GAN ( Odena et al. , 2017 ) and StyleGAN , we introduce AC-StyleGAN , and demonstrate that , very limited supervision is sufficient : just 1 % of the labelled data significantly improves the disentanglement quality , as compared to the unsupervised alternatives . However , if we are only interested in controlling a subset of factors ( effectively treating the rest as random unobserved nuisance variables ) , we find that the disentanglement qual- ity of AC-StyleGAN degrades significantly . We believe this is due to the latent nuisance factors strongly confounding the observed factors , a common and difficult problem in high-dimensional partially observed latent variables models ( Bishop , 1998 ) . To address this , we propose FC-StyleGAN , a new image-to-image model that adds controllable finegrained factors along with a super-resolution process . This is inspired by the separation of fine and coarse styles , observed in StyleGAN . FC-StyleGAN enables semantic manipulation of fine styles in high-resolution images , without a commonly-used encoder-decoder structure . Finally , we create two new high-quality datasets – Falcor3D and Isaac3D – that present a new challenge for controllable generation in terms of image resolution , photorealism , and richness of style factors , as compared to existing disentanglement datasets such as 3D Chairs ( Aubry et al. , 2014 ) , dSprite ( Matthey et al. , 2017 ) and MPI3D ( Gondal et al. , 2019 ) . Thus , we propose new semi-supervised GAN architectures that enable a generic disentanglement learning of high-resolution images as well as new high-quality disentanglement datasets . 2 RELATED WORK . Disentanglement learning . Learning generic disentangled representations in an unsupervised way has attracted a lot of attention . Two representative models are InfoGAN ( Chen et al. , 2016 ) and β-VAE ( Higgins et al. , 2017 ) . The nature of unsupervised disentanglement learning does not guarantee that the learned disentangled factors are semantically meaningful without an additional inductive bias or supervision ( Locatello et al. , 2019a ; Nguyen-Phuoc et al. , 2019 ; Locatello et al. , 2019b ) . Another line of work aims to learn disentangled representations via supervised learning wherein factors are observed variables ( Kulkarni et al. , 2015 ; Reed et al. , 2014 ; Xiao et al. , 2018 ; Locatello et al. , 2019b ) . However , little work has systematically investigated the necessity of very limited supervision to generic disentanglement learning . The analysis of Locatello et al . ( 2019b ) only applies to disentangled VAEs on low-resolution datasets . Instead , our work tries to answer a more practical question : with very little supervision on more complex and higher-resolution datasets , can we still well disentangle the factors of variation while maintaining the high generation quality ? Deep image manipulation . Deep neural networks have enabled various image editing tasks , such as style transfer ( Gatys et al. , 2016 ) , image-to-image translation ( Zhu et al. , 2017 ) , automatic colorization ( Zhang et al. , 2016 ) and 3D-aware attribute editing ( Yao et al. , 2018 ) . None of the above methods except the 3D-SDN ( Yao et al. , 2018 ) has dealt with the semantic manipulation of multiple attributes for scene images . Different from the 3D-SDN and other previous works on attribute editing that have been mostly focused on the 3D geometry manipulation , our proposed FC-StyleGAN is designed to semantically manipulate the fine-grained factors instead , such as lighting conditions and object colors . Furthermore , FC-StyleGAN does not apply an encoder-decoder structure as commonly used in previous works for semantic manipulation ( Yao et al. , 2018 ) , instead , it adds controllable fine-grained factors along with a super-resolution process , which turns out to perform well in editing fine styles of high-resolution images with a good generalization ability . 3 MODELS . 3.1 BACKGROUND ON AC-GAN AND STYLEGAN . AC-GAN . AC-GAN ( Odena et al. , 2017 ) is a variant of class conditional GANs . In the AC-GAN , the generator inputs are a latent z ( or called random noise ) and the class label . The input to the discriminator is the real or fake image , whilst its output is the probability that the image is real and the prediction of the class label . Odena et al . ( 2017 ) has shown that this modification to the standard conditional GAN formulation produces more realistic images and appears to stabilize training . StyleGAN StyleGAN ( Karras et al. , 2019 ) is a state-of-the-art GAN architecture for unsupervised image generation , particularly for high-fidelity human faces . Basically , StyleGAN comprises a mapping network whose role is to map the latent z to an intermediate space , which then controls the styles at each convolutional layer in the synthesis network via adaptive instance normalization ( AdaIN ) ( Ulyanov et al. , 2016 ; Huang & Belongie , 2017 ) . StyleGAN also enables the separation of fine-grained and coarse-grained features . For example , modifying the styles of low-resolution blocks affects only coarse-grained features ( e.g . pose and eyeglasses ) , while modifying the styles of high-resolution blocks affects only fine-grained features ( e.g . color scheme and microstructure ) . 3.2 AC-STYLEGAN FOR CONTROLLABLE IMAGE SYNTHESIS . We introduce AC-StyleGAN , a combination of AC-GAN and StyleGAN , that enables conditional generation of high-resolution images . As shown in Figure 1a , the generator in AC-StyleGAN conditions on a meta code by simply concatenating the meta code , a vector representing all the factors of variation , with the latent z . The discriminator in AC-StyleGAN now provides two outputs , the classification of real/fake and the prediction of meta code . And finally , the outputs of the mapping network – the conditioned styles – will modulate each block in the synthesis network via AdaIN . We choose AC-GAN , instead of other conditional GAN methods , such as cGANs ( with Projection Discriminator ) ( Miyato & Koyama , 2018 ) , mainly because only AC-GAN has the property of reconstructing meta code in the discriminator , which can easily be extended to a semi-supervised disentanglement learning framework . More formally , let x ( n ) r denote the n-th image from the set of N real images and let c ( n ) r be the corresponding meta code , randomly sampled from the dataset . Let c ( n ) f denote a random meta code sampled from the label distribution , and let D ( · ) and G ( · ) denote the discriminator and generator neural networks . We also assume s ( n ) r and s ( n ) f represent the real and fake classification logits in the discriminator , respectively , and ĉ ( n ) r and ĉ ( n ) f denote the predictions of c ( n ) r and c ( n ) f , respectively . From Figure 1a , we have ( s ( n ) r , ĉ ( n ) r ) = D ( x ( n ) r ) and ( s ( n ) f , ĉ ( n ) f ) = D ( G ( c ( n ) f , z ( n ) ) ) ( 1 ) Since we are aiming at achieving both high image quality and good controllability , we define a semi-supervised loss function for the generator and discriminator in AC-StyleGAN as follows , L ( G ) ( G , D ) = 1 N N−1∑ n=0 lGAN ( s ( n ) r , s ( n ) f ) ︸ ︷︷ ︸ GAN loss +γ ‖ĉ ( n ) f − c ( n ) f ‖︸ ︷︷ ︸ unsupervised disentanglement L ( D ) ( G , D ) = − 1 N N−1∑ n=0 lGAN ( s ( n ) r , s ( n ) f ) ︸ ︷︷ ︸ GAN loss +γ ‖ĉ ( n ) f − c ( n ) f ‖︸ ︷︷ ︸ unsupervised disentanglement + Nη ( n ) γ∑N−1 n=0 η ( n ) ‖ĉ ( n ) r − c ( n ) r ‖︸ ︷︷ ︸ supervised disentanglement ( 2 ) which consists of three terms : the GAN loss term , unsupervised disentanglement term and supervised disentanglement term . For the GAN loss , we apply the non-saturating loss plus gradient penalty in the discriminator , as in StyleGAN . For both the unsupervised and supervised disentanglement terms , we simply use l2 norm of the reconstruction losses . Furthermore , we introduce two coefficients γ and η ( n ) in the above loss function : ( i ) the disentanglement coefficient γ ∈ ( 0 , ∞ ) balances the trade-off between the generation quality and disentanglement performance ; ( ii ) the coefficient η ( n ) is the label mask denoted by η ( n ) = 1 { α ( n ) < α } where α ( n ) i.i.d.∼ Uniform [ 0 , 1 ] and α ∈ [ 0 , 1 ] is a hyperparameter . Accordingly , the hyperparameter α controls the fraction of labelled data that will be used for supervision ( fully unsupervised in the disentanglement learning if α = 0 , and fully supervised if α = 1 ) . Note that we sample all the α ( n ) ’ s before training so that the label mask η ( n ) remains unchanged during training to reflect the real semi-supervised case . We also apply a rescaling operation by multiplying the supervised disentanglement term in Eq . ( 2 ) with the rescaling coefficient N∑N−1 n=0 η ( n ) to take an average of effective supervised signals within a minibatch . Finally , note that AC-StyleGAN reduces to an InfoGAN variant in the special case of α = 0 ( fully unsupervised ) , when an appropriate reconstruction loss is used .

Summary: Disentangled GANs for controllable generation of High-Resolution Images introduces two new high resolution synthetic scene datasets for studying disentanglement in generative models and benchmarks two Style-GAN based architectures for controllable generation on these datasets. The datasets, though still synthetic, provide a significant quality boost over some of the simpler toy datasets previously studied in the disentanglement literature. A variety of experiments are conducted looking at 3 metrics (FID, MIG, and latent reconstruction) as well as qualitative analysis of samples. The paper studies how the performance of these architectures varies with hyperparameter and design decisions. The authors demonstrate that AC-StyleGAN performs well in fully supervised settings achieving the desired conditional generation, but, when controlling only a subset of factors, does not correctly disentangle. To address this, the author’s other architecture modification, FC-StyleGAN an image to image model is demonstrated to improve performance in this setting. The paper provides the reader with a useful overview of the behavior of the two models on these two datasets.