Learning Diverse Generations using Determinantal Point Processes

1 INTRODUCTION . Deep generative models have gained enormous research interest in recent years as a powerful framework to learn high dimensional data in an unsupervised fashion . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) and Variational AutoEncoders ( VAEs ) are among the most dominant generative approaches . They consist of training two networks : a generator ( decoder ) and a discriminator ( encoder ) , where the generator attempts to map random noise to fake data points that simulate the probability distribution of real data. . GANs are typically associated with higher quality images compared to VAEs . Nevertheless , in the process of learning multi-modal complex distributions , both models may converge to a trivial solution where the generator learns to produce few modes exclusively , as referred to by mode collapse problem . To address this , we propose utilizing Determinantal Point Processes ( DPP ) to model the diversity within data samples . DPP is a probabilistic model that has been mainly adopted for solving subset selection problems with diversity constraints ( Kulesza & Taskar , 2011 ) , such as video and document summarization . However , Sampling from a DPP requires quantifying the diversity of 2N subsets , where N is the size of the ground set . This renders DPP sampling from true data to be computationally inefficient in the generation domain . The key idea of our work is to model the diversity within real and fake data throughout the training process , which does adds an insignificant computational cost . Then , We encourage producing samples of similar diversity distribution to the true-data by back-propagating the DPP metric through the generator . This way , generator explicitly learns to cover more modes of real distribution , and accordingly alleviates mode collapse . Recent approaches tackled mode-collapse in one of two different ways : ( 1 ) improving the learning of the system to reach a better convergence point ( e.g . Metz et al . ( 2017 ) ; Arjovsky & Bottou ( 2017 ) ) ; or ( 2 ) explicitly enforcing the models to capture diverse modes or map back to the true-data distribution ( e.g . Srivastava et al . ( 2017 ) ; Che et al . ( 2017 ) ) . Here we focus on a relaxed version of the former , where we use the same learning paradigm of the standard GANs and only change the objective function . The advantage of such an approach is to avoid adding any extra trainable parameters to the trained system while maintaining the same back-propagation steps as the standard GANs . Thus , our model converges faster to a fair equilibrium point where the generator captures the diversity of the true-data distribution while preserving the quality of generations . Contribution . We introduce a new loss function , that we denote Generative Determinantal Point Processes ( GDPP ) loss . Our loss only assumes an access to a generator G , a feature extraction function φ ( · ) , and sampler from true data distribution pd . The loss encourages the generator to diversify generated samples that match the diversity of real data . This criterion can be considered as a complement to the original adversarial loss which attempts to learn an indistinguishable distribution from the true-data distribution without being specific to diverse modes . We assess the performance of GDPP on three different synthetic data environments , while also verifying the superiority on three real-world images datasets . We compared our approach with state-of-the-art approaches of more complex architectures and learning paradigms . Experiments show that our method outperforms all competing methods in terms of alleviating modecollapse and generations quality . 2 RELATED WORK . Among the tremendous amount of work that tackles the training challenges of Generative Adversarial Networks ( GANs ) , a few methods stood out as significant contributions towards addressing the problem of mode collapse . Methods that map the data back to noise . ( Donahue et al. , 2017 ; Dumoulin et al. , 2017 ) are one of the earliest methods that proposed learning a reconstruction network besides learning the deep generative network . Adding this extra network to the system aims at reversing the action of the generator by mapping from data to noise . Likelihood-free variational inference ( LFVI ) ( Tran et al. , 2017 ) , merge this concept with learning implicit densities using hierarchical Bayesian modeling . Ultimately , VEEGAN ( Srivastava et al. , 2017 ) used the same concept , but the authors did not base their reconstruction loss on the discriminator . This has the advantage of isolating the generation process from the discriminator ’ s sensitivity to any of the modes . Che et al . ( 2017 ) proposed several ways of regularizing the objective of adversarial learning including geometric metric regularizer , mode regularizer , and manifold-diffusion training . Mode regularization specifically has shown a potential into addressing the mode collapse problem and stabilizing the GANs training in general . Methods that provide a surrogate objective function . Chen et al . ( 2016 ) on the other hand propose with InfoGAN an information-theoretic extension of GANs that obtains disentangled representation of data by latent-code reconstitution through a penalty term in its objective function . InfoGAN includes an autoencoder over the latent codes ; however , it was shown to have stability problems similar to the standard GAN and requires stabilization tricks . Ghosh et al . ( 2018 ) base the ModeGAN method on the assumption of the availability of sufficient samples of every mode on the training data . In particular , if a sample from the true data distribution belongs to a particular mode , then the generated fake sample is likely to belong to the same mode . The Unrolled-GAN of Metz et al . ( 2017 ) propose a novel objective to update the generator with respect to the unrolled optimization of the discriminator . This allows training to be adjusted between using the optimal discriminator in the generator ’ s objective . It has been shown to improve the generator training process which in turn helps to reduce the mode collapse problem . Generalized LS-GAN of Edraki & Qi ( 2018 ) define a pullback operator to map generated samples to the data manifold . BourGAN Xiao et al . ( 2018 ) , with a similar philosophy , additionally draws samples from a mixture of Gaussians instead of a single Gaussian . There is , however , no specific enforcement to diversify samples . Spectral normalization strategies have been recently proposed in the works of Miyato et al . ( 2018 ) and SAGAN ( Zhang et al. , 2018 ) to further stabilize the training . We note that these strategies are orthogonal to our contribution and could be implemented in conjunction with ours to further improve the training stability of generator models . Finally , improving the Wasserstein GANs of Arjovsky et al . ( 2017 ) , WGAN-GP ( Gulrajani et al. , 2017 ) introduce a gradient penalization employed in state-of-the-art systems ( Karras et al. , 2018 ) . Methods use multiple generators and discriminators . One of the popular methods to reduce mode collapse is using multiple generator networks to provide a better coverage of the true data distribution . Liu & Tuzel ( 2016 ) propose using two generators with shared parameters to learn the joint distribution of the data . The two generators are trained independently on two domains to ensure a diverse generation . However , sharing the parameters guide both the generators to a similar subspace . Also , Durugkar et al . ( 2017 ) propose a similar idea of multiple discriminators that are being an ensemble , which was shown to produce better quality samples . Recently , Ghosh et al . ( 2018 ) proposed MAD-GAN which is a multi-agent GAN architecture incorporating multiple generators and one discriminator . Along with distinguishing real from fake samples , the discriminator also learns to identify the generator that generated the fake sample . The learning of such a system implies forcing different generators to learn unique modes , which helps in a better coverage of data modes . DualGAN of Nguyen et al . ( 2017 ) improves the diversity within GANs at the additional requirement of training two discriminators . In contrast to these approaches , our DPP-GAN does not require the training of extra networks which provides an easier and faster training as well as being less susceptible to overfitting . Finally , we also refer to PacGAN Lin et al . ( 2018 ) as another approach addressing mode collapse . They do that by modifying the discriminator input with concatenated samples to better sample the diversity within real data . Nevertheless , such approach is subject to memory and computational constraints as a result of the significant increase in batch size . 3 DETERMINANTAL POINT PROCESS ( DPP ) . DPP is a probabilistic measure that was introduced in quantum physics ( Macchi , 1975 ) and has been studied extensively in random matrix theory ( Hough et al. , 2006 ) . It provides a tractable and efficient means to capture negative correlation with respect to a similarity measure , that in turn can be used to quantify the diversity within a subset . A key characteristic in DPP that the model is agnostic about the order of items as pointed out by Gong et al . ( 2014 ) , and therefore can be used to model data that is randomly sampled from a certain distribution . A point process P on a ground set V is a probability measure on the power set of V ( i.e. , 2N ) , where N = |V| is the size of the ground set . A point process P is called determinantal if , given a random subset Y drawn according to P , we have for every S ⊆ Y , P ( S ⊆ Y ) ∝ det ( LS ) ( 1 ) for some symmetric similarity kernel L ∈ RN×N , where LS is the similarity kernel of subset S. L must be real , positive semidefinite matrix L I ( all the eigenvalues of L is between 0 and 1 ) ; since it represents a probabilistic measure and all of its principal minors must be non-negative . L is often referred to as the marginal kernel because it contains all the information needed to compute the probability of any subset S being selected in V . LS denotes the sub-matrix of L indexed by S , specifically , LS ≡ [ Lij ] ; i , j ∈ S. Hence , the marginal probability of including one element ei is p ( ei ∈ Y ) = Lii , and two elements ei and ej is LiiLjj − 2L2ij = p ( ei ∈ Y ) p ( ej ∈ Y ) − L2ij . A large value of Lij reduces the likelihood of both elements to appear together . Kulesza & Taskar ( 2010 ) proposed decomposing the kernel LS as a Gram matrix : P ( S ⊆ Y ) ∝ det ( φ ( S ) > φ ( S ) ) ∏ ei∈S q2 ( ei ) , ( 2 ) Here q ( ei ) ≥ 0 may be seen as a quality score of an item ei in the ground set V , while φi ∈ RD ; D ≤ N and ||φi||2 = 1 is used as an ` 2 normalized feature vector of an item . In this manner , φ > i φj ∈ [ −1 , 1 ] is evaluated as a ” normalized similarity ” between items ei and ej of V , and the kernel LS is guaranteed to be real positive semidefinite matrix . Geometric interpretation : det ( φ ( S ) > φ ( S ) ) = ∏ i λi , where λi is the i th eigen value of the ma- trix φ ( S ) > φ ( S ) . Hence , we may visualize that DPP models diverse representations of data because the determinant of φ ( S ) > φ ( S ) corresponds to the volume in n-D represented by the multiplication of the variances of the data ( i.e. , the eigen values ) . DPP in literature : DPP has proven to be a tremendously valuable tool when addressing the problem of diversity enforcing such as document summarization ( e.g. , Kulesza & Taskar ( 2011 ) ; Hong & Nenkova ( 2014 ) ) , pose estimation ( e.g. , Gupta ( 2015 ) ) and video summarization ( e.g. , Gong et al . ( 2014 ) ; Mahasseni et al . ( 2017 ) ) . For instance , Zhang et al . ( 2016 ) proposed to learn the two parameters q , φ in eq . 2 to quantify the diversity of the kernel LS using MLPs based on spatio-temporal features of the video to perform summarization . Recently , Hsiao & Grauman ( 2018 ) proposed to use DPP to automatically create capsule wardrobes , i.e . assemble a minimal set of items that provide maximal mix-and-match outfits given an inventory of candidate garments .

Summary: The paper proposes to introduce DPP into the vanilla GAN loss and uses it as a way to regularize the generator to produce more diverse outputs, in order to combat the mode-collapse problem. Since the proposed method is added as a simple loss regularizer, the approach does not introduce additional parameters, therefore, less training difficulties. The results on synthetic data seems promising, but there is insufficient evaluation being performed on real and larger dataset where the mode collapse problems are more likely to happen.