Learning Diverse Generations using Determinantal Point Processes

1 INTRODUCTION . Deep generative models have gained enormous research interest in recent years as a powerful framework to learn high dimensional data in an unsupervised fashion . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) and Variational AutoEncoders ( VAEs ) are among the most dominant generative approaches . They consist of training two networks : a generator ( decoder ) and a discriminator ( encoder ) , where the generator attempts to map random noise to fake data points that simulate the probability distribution of real data. . GANs are typically associated with higher quality images compared to VAEs . Nevertheless , in the process of learning multi-modal complex distributions , both models may converge to a trivial solution where the generator learns to produce few modes exclusively , as referred to by mode collapse problem . To address this , we propose utilizing Determinantal Point Processes ( DPP ) to model the diversity within data samples . DPP is a probabilistic model that has been mainly adopted for solving subset selection problems with diversity constraints ( Kulesza & Taskar , 2011 ) , such as video and document summarization . However , Sampling from a DPP requires quantifying the diversity of 2N subsets , where N is the size of the ground set . This renders DPP sampling from true data to be computationally inefficient in the generation domain . The key idea of our work is to model the diversity within real and fake data throughout the training process , which does adds an insignificant computational cost . Then , We encourage producing samples of similar diversity distribution to the true-data by back-propagating the DPP metric through the generator . This way , generator explicitly learns to cover more modes of real distribution , and accordingly alleviates mode collapse . Recent approaches tackled mode-collapse in one of two different ways : ( 1 ) improving the learning of the system to reach a better convergence point ( e.g . Metz et al . ( 2017 ) ; Arjovsky & Bottou ( 2017 ) ) ; or ( 2 ) explicitly enforcing the models to capture diverse modes or map back to the true-data distribution ( e.g . Srivastava et al . ( 2017 ) ; Che et al . ( 2017 ) ) . Here we focus on a relaxed version of the former , where we use the same learning paradigm of the standard GANs and only change the objective function . The advantage of such an approach is to avoid adding any extra trainable parameters to the trained system while maintaining the same back-propagation steps as the standard GANs . Thus , our model converges faster to a fair equilibrium point where the generator captures the diversity of the true-data distribution while preserving the quality of generations . Contribution . We introduce a new loss function , that we denote Generative Determinantal Point Processes ( GDPP ) loss . Our loss only assumes an access to a generator G , a feature extraction function φ ( · ) , and sampler from true data distribution pd . The loss encourages the generator to diversify generated samples that match the diversity of real data . This criterion can be considered as a complement to the original adversarial loss which attempts to learn an indistinguishable distribution from the true-data distribution without being specific to diverse modes . We assess the performance of GDPP on three different synthetic data environments , while also verifying the superiority on three real-world images datasets . We compared our approach with state-of-the-art approaches of more complex architectures and learning paradigms . Experiments show that our method outperforms all competing methods in terms of alleviating modecollapse and generations quality . 2 RELATED WORK . Among the tremendous amount of work that tackles the training challenges of Generative Adversarial Networks ( GANs ) , a few methods stood out as significant contributions towards addressing the problem of mode collapse . Methods that map the data back to noise . ( Donahue et al. , 2017 ; Dumoulin et al. , 2017 ) are one of the earliest methods that proposed learning a reconstruction network besides learning the deep generative network . Adding this extra network to the system aims at reversing the action of the generator by mapping from data to noise . Likelihood-free variational inference ( LFVI ) ( Tran et al. , 2017 ) , merge this concept with learning implicit densities using hierarchical Bayesian modeling . Ultimately , VEEGAN ( Srivastava et al. , 2017 ) used the same concept , but the authors did not base their reconstruction loss on the discriminator . This has the advantage of isolating the generation process from the discriminator ’ s sensitivity to any of the modes . Che et al . ( 2017 ) proposed several ways of regularizing the objective of adversarial learning including geometric metric regularizer , mode regularizer , and manifold-diffusion training . Mode regularization specifically has shown a potential into addressing the mode collapse problem and stabilizing the GANs training in general . Methods that provide a surrogate objective function . Chen et al . ( 2016 ) on the other hand propose with InfoGAN an information-theoretic extension of GANs that obtains disentangled representation of data by latent-code reconstitution through a penalty term in its objective function . InfoGAN includes an autoencoder over the latent codes ; however , it was shown to have stability problems similar to the standard GAN and requires stabilization tricks . Ghosh et al . ( 2018 ) base the ModeGAN method on the assumption of the availability of sufficient samples of every mode on the training data . In particular , if a sample from the true data distribution belongs to a particular mode , then the generated fake sample is likely to belong to the same mode . The Unrolled-GAN of Metz et al . ( 2017 ) propose a novel objective to update the generator with respect to the unrolled optimization of the discriminator . This allows training to be adjusted between using the optimal discriminator in the generator ’ s objective . It has been shown to improve the generator training process which in turn helps to reduce the mode collapse problem . Generalized LS-GAN of Edraki & Qi ( 2018 ) define a pullback operator to map generated samples to the data manifold . BourGAN Xiao et al . ( 2018 ) , with a similar philosophy , additionally draws samples from a mixture of Gaussians instead of a single Gaussian . There is , however , no specific enforcement to diversify samples . Spectral normalization strategies have been recently proposed in the works of Miyato et al . ( 2018 ) and SAGAN ( Zhang et al. , 2018 ) to further stabilize the training . We note that these strategies are orthogonal to our contribution and could be implemented in conjunction with ours to further improve the training stability of generator models . Finally , improving the Wasserstein GANs of Arjovsky et al . ( 2017 ) , WGAN-GP ( Gulrajani et al. , 2017 ) introduce a gradient penalization employed in state-of-the-art systems ( Karras et al. , 2018 ) . Methods use multiple generators and discriminators . One of the popular methods to reduce mode collapse is using multiple generator networks to provide a better coverage of the true data distribution . Liu & Tuzel ( 2016 ) propose using two generators with shared parameters to learn the joint distribution of the data . The two generators are trained independently on two domains to ensure a diverse generation . However , sharing the parameters guide both the generators to a similar subspace . Also , Durugkar et al . ( 2017 ) propose a similar idea of multiple discriminators that are being an ensemble , which was shown to produce better quality samples . Recently , Ghosh et al . ( 2018 ) proposed MAD-GAN which is a multi-agent GAN architecture incorporating multiple generators and one discriminator . Along with distinguishing real from fake samples , the discriminator also learns to identify the generator that generated the fake sample . The learning of such a system implies forcing different generators to learn unique modes , which helps in a better coverage of data modes . DualGAN of Nguyen et al . ( 2017 ) improves the diversity within GANs at the additional requirement of training two discriminators . In contrast to these approaches , our DPP-GAN does not require the training of extra networks which provides an easier and faster training as well as being less susceptible to overfitting . Finally , we also refer to PacGAN Lin et al . ( 2018 ) as another approach addressing mode collapse . They do that by modifying the discriminator input with concatenated samples to better sample the diversity within real data . Nevertheless , such approach is subject to memory and computational constraints as a result of the significant increase in batch size . 3 DETERMINANTAL POINT PROCESS ( DPP ) . DPP is a probabilistic measure that was introduced in quantum physics ( Macchi , 1975 ) and has been studied extensively in random matrix theory ( Hough et al. , 2006 ) . It provides a tractable and efficient means to capture negative correlation with respect to a similarity measure , that in turn can be used to quantify the diversity within a subset . A key characteristic in DPP that the model is agnostic about the order of items as pointed out by Gong et al . ( 2014 ) , and therefore can be used to model data that is randomly sampled from a certain distribution . A point process P on a ground set V is a probability measure on the power set of V ( i.e. , 2N ) , where N = |V| is the size of the ground set . A point process P is called determinantal if , given a random subset Y drawn according to P , we have for every S ⊆ Y , P ( S ⊆ Y ) ∝ det ( LS ) ( 1 ) for some symmetric similarity kernel L ∈ RN×N , where LS is the similarity kernel of subset S. L must be real , positive semidefinite matrix L I ( all the eigenvalues of L is between 0 and 1 ) ; since it represents a probabilistic measure and all of its principal minors must be non-negative . L is often referred to as the marginal kernel because it contains all the information needed to compute the probability of any subset S being selected in V . LS denotes the sub-matrix of L indexed by S , specifically , LS ≡ [ Lij ] ; i , j ∈ S. Hence , the marginal probability of including one element ei is p ( ei ∈ Y ) = Lii , and two elements ei and ej is LiiLjj − 2L2ij = p ( ei ∈ Y ) p ( ej ∈ Y ) − L2ij . A large value of Lij reduces the likelihood of both elements to appear together . Kulesza & Taskar ( 2010 ) proposed decomposing the kernel LS as a Gram matrix : P ( S ⊆ Y ) ∝ det ( φ ( S ) > φ ( S ) ) ∏ ei∈S q2 ( ei ) , ( 2 ) Here q ( ei ) ≥ 0 may be seen as a quality score of an item ei in the ground set V , while φi ∈ RD ; D ≤ N and ||φi||2 = 1 is used as an ` 2 normalized feature vector of an item . In this manner , φ > i φj ∈ [ −1 , 1 ] is evaluated as a ” normalized similarity ” between items ei and ej of V , and the kernel LS is guaranteed to be real positive semidefinite matrix . Geometric interpretation : det ( φ ( S ) > φ ( S ) ) = ∏ i λi , where λi is the i th eigen value of the ma- trix φ ( S ) > φ ( S ) . Hence , we may visualize that DPP models diverse representations of data because the determinant of φ ( S ) > φ ( S ) corresponds to the volume in n-D represented by the multiplication of the variances of the data ( i.e. , the eigen values ) . DPP in literature : DPP has proven to be a tremendously valuable tool when addressing the problem of diversity enforcing such as document summarization ( e.g. , Kulesza & Taskar ( 2011 ) ; Hong & Nenkova ( 2014 ) ) , pose estimation ( e.g. , Gupta ( 2015 ) ) and video summarization ( e.g. , Gong et al . ( 2014 ) ; Mahasseni et al . ( 2017 ) ) . For instance , Zhang et al . ( 2016 ) proposed to learn the two parameters q , φ in eq . 2 to quantify the diversity of the kernel LS using MLPs based on spatio-temporal features of the video to perform summarization . Recently , Hsiao & Grauman ( 2018 ) proposed to use DPP to automatically create capsule wardrobes , i.e . assemble a minimal set of items that provide maximal mix-and-match outfits given an inventory of candidate garments .

Summary: This paper proposes generative adversarial networks regularized by Determinantal Point Process (DPP) to learn diverse data space. DPP is a probabilistic model that encourages the diversity between the dataset. Authors observe that previous generative models have a mode-collapse problem, and they add generative DPP (GDPP) loss (eq (5)) as a diversity regularizer. Experiments show the GDPP loss is practically helpful to learn under synthetic multi-modal data and real-world image generation.