Information-theoretic stochastic contrastive conditional GAN: InfoSCC-GAN

1 INTRODUCTION . Conditional image generation is the task of generating images , based on some attributes . The idea of conditional GAN ( cGAN ) was proposed by Mirza & Osindero ( 2014 ) . The authors modified the classic GAN architecture by adding the attribute as a parameter to the input of the generator to generate the corresponding image . They also added attributes to the discriminator input to better distinguish real data . Since then , a lot of other methods have been developed . ACGAN ( Odena et al . ( 2017 ) ) has an auxiliary classifier to guide the generator to synthesize well-classifiable images . ProjGAN ( Miyato & Koyama ( 2018 ) ) improves the approach proposed in ACGAN by utilizing the inner product of an embedded image and the corresponding attribute embeddings . ContraGAN ( Kang & Park ( 2020 ) ) utilizes contrastive 2C loss with multiple positive and negative pairs to update the generator . While these methods have shown impressive results in conditional image generation , they are known to be difficult to train , to lack the image diversity within the same input attribute , and not to always have meaningful data exploration in the latent space . We propose a new stochastic contrastive conditional generative adversarial network ( InfoSCC-GAN ) with an explorable latent space . As a baseline generator , we use EigenGAN ( He et al . ( 2021 ) ) generator with interpretable and controllable input dimensions , yet trained in an unsupervised way . EigenGAN ensures that different layers of a generative CNN , controlled by noise vectors , hold different semantics of the synthesized images . EigenGAN is able to mine interpretable and controllable dimensions in an unsupervised way from different generator layers by embedding one linear subspace with an orthogonal basis into each generator layer . These layer-wise subspaces automatically discover a set of ” eigen-dimensions ” at each layer corresponding to a set of semantic attributes or interpretable variations , via generative adversarial training to learn a target distribution . By traversing the coefficient of a specific ” eigen-dimension ” , the generator can generate samples with continuous changes corresponding to a specific semantic attribute . The ” core ” latent space of EigenGAN is generated from the Gaussian distribution . In contrast to that , we use the latent space of the contrastive encoder as the ” core ” latent space for the generator . By using the contrastive encoder , we have the ability to discover the ” inner ” attributes from the dataset by clustering the latent space . The ” inner ” attributes are useful for datasets without external annotations , unbalanced datasets , and datasets with subclasses . By using the encoder , we have an opportunity to compare latent spaces of the real images and the generated images . At the same time , the classifier ensures the correspondence between the attributes of training data and conditionally generated ones . Also , by training the encoder and the classifier independently from the generator , we reduce the general training complexity of the system . It allows avoiding training the encoder and classifier on unrealistic synthetic data , when training it jointly with the generator and discriminator . The information-theoretical interpretation of the proposed model is provided in Section 2 . The experiments and ablation studies are provided in Section 4 . We summarize our contributions as follows : • We proposed a novel Stochastic Contrastive Conditional Generative Adversarial Network ( InfoSCC-GAN ) for stochastic conditional image generation with controllable and interpretable latent space . It is based on an EigenGAN , an independent contrastive encoder , and an independent attribute classifier . • We introduce a novel classification regularization technique , which is based on updating the model with classification loss each n-th iteration and updating the generator using the adversarial and classification loss separately . • We propose a novel method for the attribute selection , based on clustering the embeddings , computed using the pre-trained contrastive encoder . • We provide an information-theoretic interpretation of the proposed system . • We perform an ablation study to determine the contribution of each part of the model to overall performance . 2 INFORMATION-THEORETICAL FORMULATION . 2.1 THE TRAINING OF THE ENCODER ( STAGE 1 ) . The encoder training is schematically shown in Figure 1 , stage 1 . The encoder training is based on the maximization problem : φ̂ε = argmax φε Iφε ( X ; E ) , ( 1 ) where Iφε ( X ; E ) = Ep ( x , ε ) [ log qφε ( ε|x ) qφε ( ε ) ] , where qφε ( ε|x ) denotes the encoder and qφε ( ε ) - the marginal latent space distribution . In the framework of contrastive learning , ( 1 ) is maximized based on the infoNCE framework ( van den Oord et al . ( 2018 ) ) . In the practical implementation , one can use approaches similar to SimCLR ( Chen et al . ( 2020 ) ) where the inner product between the positive pairs created from the augmented views originating from the same image is maximized and the inner product between the negative pairs originating from different images is minimized . Alternatively , one can use other approaches to learn the representation ε such as BYOL ( Grill et al . ( 2020 ) ) , Barlow Twins ( Zbontar et al . ( 2021 ) ) , etc . without loose of the generality of the proposed approach . It should be pointed out that the encoder is trained independently from the decoder in the scope of the considered setup . 2.2 THE TRAINING OF THE CLASS ATTRIBUTE CLASSIFIER ( STAGE 2 ) . The class attribute classifier training is schematically shown in Figure 1 , stage 2 . The training of the class attribute classifier is based on the maximization problem : θ̂y = argmax θy Iφ∗ε , θy ( Y ; E ) , ( 2 ) where Iφ∗ε , θy ( Y ; E ) = H ( Y ) −Hφ∗ε , θy ( Y|E ) and H ( Y ) = −Epy ( y ) log py ( y ) and the conditional entropy is defined as Hφ∗ε , θy ( Y|E ) = −Epx ( x ) [ Eqφ∗ε ( ε|x ) [ log pθy ( y|ε ) ] ] . Since H ( Y ) is independent of the parameters of the encoder and classifier , ( 2 ) reduces to the lower bound minimization : θ̂y = argmin θy Hφ∗ε , θy ( Y|E ) , ( 3 ) that under the categorical conditional distribution pθy ( y|ε ) can be expressed as the categorical crossentropy Ly ( y , ŷ ) . 2.3 THE TRAINING OF THE DECODER , I.E. , THE MAPPER AND GENERATOR ( STAGE 3 ) . The training of decoder is shown in Figure 1 , stage 3 . The decoder is trained first to maximize the mutual information between the class attributes ỹ predicted from the generated images and true class attributes y : ( θ̂x , ψ̂ ) = argmax θx , ψ Iψ , θx , φ∗ε , θ∗y ( Y ; E ) , ( 4 ) where Iψ , θx , φ∗ε , θ∗y ( Y ; E ) = H ( Y ) − Hψ , θx , φ∗ε , θ∗y ( Y|E ) and H ( Y ) = −Epy ( y ) log py ( y ) and the conditional entropy is defined as Hψ , θx , φ∗ε , θ∗y ( Y|E ) = −Epy ( y ) [ Epz ( z ) [ Erψ ( ε|y , z ) [ Epθx ( x|ε ) [ Eqφ∗ε ( ε|x ) [ log pθ∗y ( y|ε ) ] ] ] ] ] , where p∗θy ( y | ε ) corresponds to the classifier and q∗φε ( ε|x ) denotes the pre-trained encoder . Since H ( Y ) is independent of the parameters of the encoder and classifier , ( 4 ) reduces to the lower bound minimization : ( θ̂x , ψ̂ ) = argmin θx , ψ Hψ , θx , φ∗ε , θ∗y ( Y|E ) , ( 5 ) that under the categorical conditional distribution pθy ( y|ε ) can be expressed as the categorical crossentropy Ly ( y , ỹ ) . Finally , the decoder should produce samples that follow the distribution of training data px ( x ) that corresponds to the maximization of mutual information : ( θ̂x , ψ̂ ) = argmax θx , ψ Iψ , θx ( X ; E ) , ( 6 ) where Iψ , θx ( X ; E ) = Epx ( x ) [ Epy ( y ) [ Epz ( z ) [ Erψ ( ε|y , z ) [ Epθx ( x|ε ) [ log pθx ( x|ε ) px ( x ) ] ] ] ] ] = Epy ( y ) [ Epz ( z ) [ Erψ ( ε|y , z ) [ DKL ( pθx ( x|E = ε ) ||pθx ( x ) ) ] ] ] − DKL ( px ( x ) ||pθx ( x ) ) , where pθx ( x ) denotes the distribution of generated samples x̃ . Since DKL ( pθx ( x|E = ε ) ||pθx ( x ) ) ≥ 0 , the maximization of the above mutual information reduces to the minimization problem : ( θ̂x , ψ̂ ) = argmin θx , ψ DKL ( px ( x ) ||pθx ( x ) ) . ( 7 ) The above discriminator is denoted asDxx̃ ( x ) . At the same time , one can also envision the discriminator conditioned on the attribute class y Dxx̃ ( x | y ) that is implemented as a set of discriminators for each subset of generated and original samples defined by y . 3 IMPLEMENTATION DETAILS . Dataset We test the proposed method on AFHQ ( Choi et al . ( 2020 ) ) and CelebA ( Liu et al . ( 2015 ) ) datasets . AFHQ dataset contains 16130 images belonging to 3 classes : cats , dogs , and wilds animals , CelebA dataset contains 202599 face images with 40 binary attributes . We use AFHQ and CelebA dataset for visual result inspection and AFHQ dataset for the ablations studies . 3.1 ENCODER . The proposed encoder is designed to produce the interpretable latent space and it can be used for : ( i ) internal latent exploration , ( ii ) feature metric like the VGG-loss ( Ledig et al . ( 2017 ) ) , ( iii ) feature extraction for the classification of the generated samples with ” external ” and ” internal ” labels . We have selected the SimCLR unsupervised encoder since it has shown state-of-the-art performance in unsupervised learning on diverse datasets . By training the encoder in an unsupervised way , it is trained to learn the inner data distribution , which is then used to compare real and generated data . For both AFHQ and CelebA datasets we use Resnet50 ( He et al . ( 2016 ) ) as a base model . We pretrain the SimCLR model for each dataset using contrastive NT-Xent loss ( Sohn ( 2016 ) ) . We apply the same augmentations as in the original SimCLR paper . The 2D t-SNE of the extracted features for the AFHQ dataset is shown in Figure 2 . The 2D t-SNEs of the extracted features for CelebA dataset for selected attributes are shown in the Appendix A . 3.2 CLASSIFIER . The initial idea of using a pre-trained classifier to regularize the generative model is based on the need to generate images that belong to the specific class . Training the classifier jointly with the generator and discriminator requires more time , and is inefficient since in the early iterations the generator network produces poorly generated images that are not similar to the real ones . While it is possible to use L2 or other distance-based metrics to regularize the generator by comparing embeddings between real and generated images computed using the encoder , it requires having predefined pairs , and since our goal is to develop a generative model , we use the pre-trained classifier to regularize the generator . We use a one-layer linear classifier for classification . As an input , we use features extracted using the pre-trained encoder . When training on the AFHQ dataset we use the cross-entropy loss , since each image has one attribute per image , when training on CelebA dataset , we use the binary cross-entropy loss , since each image has multiple attributes per image .

Summary: This paper proposes a GAN for conditional generation. The authors combine an unsupervised contrastive encoder, stochastic EigenGAN generator, and a classifier. InfoSCC-GAN can perform image generation conditioned on external attributes by maximizing mutual information between input data and class attributes. In addition, clustering on the output space of the pre-trained encoder allows us to generate images conditioned on inner attributes.