Unlabeled Disentangling of GANs with Guided Siamese Networks

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) are generative model estimators , where two neural networks ( generator and discriminator ) are trained in an adversarial setting , so that likelihood-based probabilistic modeling is not necessary . This works particularly well for sampling from a complex probability distribution , such as images . Although GANs yield realistic looking images ( Radford et al. , 2015 ) , the original formulation in ( Goodfellow et al. , 2014 ) only allows for randomly sampling from the data distribution without disentangled structural or semantic control over the generated data points . One way to disentangle the generation process is to use conditional GANs ( Mirza & Osindero , 2014 ; Odena et al. , 2017 ) . These models modify the generator by conditioning it with supervised labels . Then , they either take the same labels as input in the discriminator ( Mirza & Osindero , 2014 ) and measure the image-label compatibility , or classify the correct label at the output , given the generated image ( Odena et al. , 2017 ) . Conditional GANs rely on a dataset with labels , which might not always be available or might be time-consuming to collect . In this paper , we propose two GAN-based methods that learns disentangled representations without using labeled data . Our first method , Unlabeled Disentangling GAN ( UD-GAN ) , generates image pairs , then embeds them with Siamese Networks ( Chopra et al. , 2005 ) , and finally learns a distance metric on a disentangled representation space . Whereas our second method , UD-GAN-G , uses guidance functions to restrict the input to our siamese networks , so that they capture desired semantic variations . 2 RELATED WORK . There have been many studies on learning disentangled representations in generative models , which can be grouped into the level of supervision/labeled data they require . Disentangled representations ( supervised ) . In ( Zhu et al. , 2014 ; Yang et al. , 2015 ) , the identity and the viewpoint of an object are disentangled via reconstructing the same object from a different viewpoint and minimizing a reconstruction loss . Whereas in ( Kingma et al. , 2014 ; Makhzani et al. , 2016 ) , the style and category of an object is separated via autoencoders , where an encoder embeds the style of an input image to a latent representation , and a decoder takes the category and style input to reconstruct the input image . In ( Tran et al. , 2017 ; Yin et al. , 2017 ) , autoencoders and GANs are combined to decompose identity and attribute of an object , where the disentangled representation is obtained at the encoder outputs , and image labels are used at the output of the discriminator . Disentangled representations ( semi-supervised ) . In ( Reed et al. , 2014 ) , they clamp the hidden units for a pair of images with the same identity but with different pose or expression to have the same identity representation . Whereas in ( Kulkarni et al. , 2015 ) , synthesized images are used to disentangle pose , light , and shape of an object by passing a batch of images where only one attribute varies and the rest of the representation is clamped to be the same . These techniques only require a batch of samples with one attribute different at a time . Disentangled representations ( unsupervised ) . InfoGAN ( Chen et al. , 2016 ) is an unsupervised technique that discovers categorical and continuous factors by maximizing the mutual information between a GAN ’ s noise variables and the generated image . β-VAE ( Higgins et al. , 2017 ) and DIPVAE ( Kumar et al. , 2018 ) are unsupervised autoencoder-based techniques that disentangle different factors in the latent representation of an encoded image . In β-VAE , the KL-divergence between the latent and a prior distribution is weighted with a factor β > 1 to encourage disentanglement in the posterior latent distributions . Wheres in DIP-VAE , the covariance matrix of the latent distribution is encouraged to be an identity matrix , thus leading to uncorrelated latent representations . For all of the unsupervised methods , after a model is trained , a human needs to investigate which factors map to which semantic property . In addition , as the methods are unsupervised , not all desirable factors might be represented . In contrast , our method builds on existing approaches with two important modifications : ( i ) We operate on pairs of similar/dissimilar image pairs . ( ii ) We compute the image embeddings using separate networks , which allows us to guide the disentangling process with information restriction . 3 UNLABELED DISENTANGLING GAN . 3.1 BACKGROUND : GENERATIVE ADVERSARIAL NETWORKS . In GANs , the generator , G ( . ) , maps a latent variable z , which has an easy-to-sample distribution , into a more complex and unknown distribution , such as images . On the other hand , the discriminator D ( . ) tries to distinguish real images from the ones that are generated by G. In ( Goodfellow et al. , 2014 ) , the training is performed as a minimax game as follows : min G max D V ( G , D ) = E x∼PR [ logD ( x ) ] + E z∼PZ [ log ( 1−D ( G ( z ) ) ) ] , ( 1 ) where PR and PZ are the probability distributions of real images and the latent variable z , respectively . We train our GAN by using the loss in equation 1 . In order to increase stability , we modify the generator loss by maximzing log ( D ( G ( z ) ) ) , instead of minimizing the second term in equation 1 . 3.2 A NOVEL GAN ARCHITECTURE : UD-GAN . In a standard GAN setting , all of the variation in the distribution of real images is captured by the latent variable z . However , a single dimension or a slice of z does not necessarily have a semantic meaning . In this paper , our target is to slice the latent variable into multiple vectors , where each vector controls a different semantic variation . Our network architecture is visualized in Figure 1 . In our method , the latent vector z = [ q1 , q2 , ... , qNA ] is the concatenation of NA knobs , { qi } NA i=1 , which represent different attributes we aim to disentangle . One can add a final variable that captures the variation ( and the noise ) that is not picked up by the knobs . In our experiments , this additional variable did not have a notable effect . In our notation , qi refers to all of the knobs , except qi . In order to train our model , first , for each qi , we sample two different vectors , q ( 1 ) i and q ( 2 ) i from Unif ( −1 , 1 ) . If we would attempt to form a batch by combinatorially concatenating all knob samples , we get a batch size of 2NA , which grows exponentially with the number of attributes . To avoid this computational burden , we train our model through stochastic sampling of one attribute at a time . For example , if the ith attribute is chosen , we generate four images as shown in Figure 1 . Sampled Latents via Concatenation D iscrim inator G enerator Real Images Real or Fake ? Generated Images Contrastive Loss Pull Push Pull Guidance Figure 1 : The flowchart of our architecture . Sampled latents from different attributes are combined into latent vectors . Generated images are grouped with respect to different attributes ( here , represented by shape ) by Siamese Networks ( denoted as φi ) . The image pairs that are generated with the same qi vectors , { x11 , x12 } or { x21 , x22 } , should have the same ith attribute , regardless of the values of qi . We can ensure this via embedding the generated image pairs into a representation space with Siamese Networks ( Chopra et al. , 2005 ) , which are denoted as φi ( . ) , and then learning a distance metric on the embedding vectors by employing Contrastive Loss ( Hadsell et al. , 2006 ) . An optional guidance function is used to restrict the information that goes into a siamese network , thus letting us approximate a desired representation space . The guidance is disabled for our unsupervised UD-GAN approach . Whereas for UD-GAN-G , the guidance is a simple , user-defined function , which is discussed in Section 3.3 . We use a Contrastive Loss function to pull similar image pairs together , and push dissimilar pairs apart as follows : Lφi = 1 2 2∑ ni=1 ρi ( xni1 , xni2 ) 2 + 1 4 2∑ ni=1 2∑ mi=1 max ( 0 , γ ( 1,2 ) i −ρi ( x1ni , x2mi ) ) 2 , ( 2 ) where , Lφi is the Contrastive Loss for the i th Siamese Network φi ( . ) , the function ρi ( xni1 , xni2 ) =∣∣∣∣φi ( xni1 ) − φi ( xni2 ) ∣∣∣∣2 is a shorthand for embedding distance between xni1 and xni2 , and γ ( 1,2 ) i is an adaptive margin of the form γ ( 1,2 ) i = ∣∣∣∣q ( 1 ) i −q ( 2 ) i ∣∣∣∣2 . Using an adaptive margin makes the distance between two latent samples semantically meaningful and we empirically found that it improves the training stability . The discriminator network D is not modified and is trained to separate real and generated image distributions . Donahue et al . ( 2018 ) use a similar latent variable slicing for capturing illumination and pose variations of a face with a fixed identity . Their discriminator needs image pairs , which must be labeled for real images , to judge the quality and identity of the faces . Our method does not require any labels for the real images . Instead , we create similar and dissimilar image pairs via concatenating latent variables and generating image batches . Our final loss function is : Lφ = λφi Lφi , i ∼ Cat ( NA ) min G max D V ( G , D ) = LGAN +Lφ , ( 3 ) where , LWGAN is the GAN loss described in equation 1 , λφi is the weight of the embedding loss , and the sampling of the latent variables depends on i and is performed as described above . 3.3 DISENTANGLING WITH GUIDANCE FUNCTIONS : UD-GAN-G. A guidance function reduces the information content that flows into a siamese network and causes the corresponding embedding space to capture only the variations in the restricted input . For example , consider we want to capture the hair-related attributes in the CelebA dataset ( Liu et al. , 2015 ) , which contains aligned images of human faces . By cropping every region but the top part of a generated image , we are able to guide φtop ( . ) to learn only the variations in the “ Hair Color ” as shown in the first row of Figure 2 . Note that , the knob qtop ( that corresponds to φtop ( . ) ) changes the hair color not only at the cropped part of the image but as a whole . This is due to the interplay between the adversarial part of our loss ( see equation 3 ) , which enforces global realism in images , and the contrastive loss , which administers disentangled representations . As shown in Figure 2 , different guidance functions leads to capturing different variations in the CelebA dataset .

Summary: The paper proposes a framework for learning interpretable latent representations for GANs. The key idea is to use siamese networks with contrastive loss. Specifically, it decomposes the latent code to a set of knobs (sub part of the latent code). Each time it renders different images with different configurations of the knobs. For example, 1) as changing one knob while keeping the others, it expects it would only result in change of one attribute in the image, and 2) as keeping one knob while changing all the others, it expects it would result in large change of image appearances. The relative magnitude of change for 1) and 2) justifies the use of a Siamese network in addition to the image discriminator in the standard GAN framework. The paper further talks about how to use inductive bias to design the Siamese network so that it can control the semantic meaning of a particular knob.