DppNet: Approximating Determinantal Point Processes with Deep Networks

1 INTRODUCTION . Selecting a representative sample of data from a large pool of available candidates is an essential step of a large class of machine learning problems : noteworthy examples include automatic summarization , matrix approximation , and minibatch selection . Such problems require sampling schemes that calibrate the tradeoff between the point-wise quality – e.g . the relevance of a sentence to a document summary – of selected elements and the set-wise diversity1 of the sampled set as a whole . Determinantal Point Processes ( DPPs ) are probabilistic models over subsets of a ground set that elegantly model the tradeoff between these often competing notions of quality and diversity . Given a ground set of size N , DPPs allow for O ( N3 ) sampling over all 2N possible subsets of elements , assigning to any subset S of a ground set Y of elements the probability2 PL ( S ) = detLS/det ( I +L ) ( 1 ) where L ∈ RN×N is the DPP kernel and LS = [ Lij ] i , j∈S denotes the principal submatrix of L indexed by items in S. Intuitively , DPPs measure the volume spanned by the feature embedding of the items in feature space ( Figure 1 ) . First introduced by Macchi ( 1975 ) to model the distribution of possible states of fermions obeying the Pauli exclusion principle , the properties of DPPs have since then been studied in depth ( Hough et al. , 2006 ; Borodin , 2009 , see e.g. ) . As DPPs capture repulsive forces between similar elements , they arise in many natural processes , such as the distribution of non-intersecting random walks ( Johansson , 2004 ) , spectra of random matrix ensembles ( Mehta & Gaudin , 1960 ; Ginibre , 1965 ) , and zerocrossings of polynomials with Gaussian coefficients ( Hough et al. , 2009 ) . More recently , DPPs have become a prominent tool in machine learning due to their elegance and tractability : recent applications include video recommendation ( Chen et al. , 2018 ) , minibatch selection ( Zhang et al. , 2017 ) , and kernel approximation ( Li et al. , 2016b ; Mariet et al. , 2018 ) . However , the O ( N3 ) sampling cost makes the practical application of DPPs intractable for large datasets , requiring additional work such as subsampling from Y , structured kernels ( Gartrell et al. , 1Here , we use diversity to mean useful coverage across dissimilar examples in a meaningful feature space , rather than other definitions diversity that may appear in ML fairness literature . 2We are adopting here the L-Ensemble construction ( Borodin & Rains , 2005 ) of a DPP . 2017 ; Mariet & Sra , 2016b ) , or approximate sampling methods ( Anari et al. , 2016a ; Li et al. , 2016a ; Affandi et al. , 2013 ) . Nonetheless , even such methods require significant pre-processing time , and scale poorly with the size of the dataset . Furthermore , when dealing with ground sets with variable components , pre-processing costs can not be amortized , significantly impeding the application of DPPs in practice . These setbacks motivate us to investigate the use of more scalable models to generate high-quality , diverse samples from datasets to obtain highly-scalable methods with flexibility to adapt to constantly changing datasets . Specifically , we use generative deep models to approximate the DPP distribution over a ground set of items with both fixed and variable feature representations . We show that a simple , carefully constructed neural network , DPPNET , can generate DPP-like samples with very little overhead , while maintaining fundamental theoretical properties of DPP measures . Furthermore , we show that DPPNETs can be trivially employed to sample from a conditional DPP ( i.e . sampling S such that A ⊆ S is predefined ) and for greedy mode approximation . Contributions . • We introduce DPPNET , a deep network trained to generate DPP-like samples based on static and variable ground sets of possible items . • We derive theoretical conditions under which the DPPNETs inherit the DPP ’ s log-submodularity . • We show empirically that DPPNETs provide an accurate approximation to DPPs and drastically speed up DPP sampling . 2 RELATED WORK . DPPs belong to the class of Strongly Rayleigh ( SR ) measures ; these measures benefit from the strongest characterization of negative association between similar items ; as such , SR measures have benefited from significant interest in the mathematics community ( Pemantle , 2000 ; Borcea et al. , 2009 ; Borcea & Brändén , 2008 ; Marcus et al. , 2015 ) and more recently in machine learning ( Anari et al. , 2016b ; Li et al. , 2016a ; Mariet et al. , 2018 ) . This , combined with their tractability , makes DPPs a particularly attractive tool for subset selection in machine learning , and is one of the key motivations for our work . The application of DPPs to machine learning problems spans fields from document and video summarization ( Gong et al. , 2014 ; Chao et al. , 2015 ) , recommender systems ( Zhou et al. , 2010 ; Chen et al. , 2018 ) and object retrieval ( Affandi et al. , 2014 ) to kernel approximation ( Li et al. , 2016b ) , neural network pruning ( Mariet & Sra , 2016a ) , and minibatch selection ( Zhang et al. , 2017 ) . Zou & Adams ( 2012 ) developed DPP priors for encouraging diversity in generative models and Snoek et al . ( 2013 ) showed that DPPs accurately model inhibition in neural spiking data . In the general case , sampling exactly from a DPP requires an initial eigendecomposition of the kernel matrix L , incurring a O ( N3 ) cost . In order to avoid this time-consuming step , several approximate sampling methods have been derived ; Affandi et al . ( 2013 ) approximate the DPP kernel during sampling ; more recently , results by Anari et al . ( 2016a ) followed by Li et al . ( 2016a ) showed that DPPs are amenable to efficient MCMC-based sampling methods . Algorithm 1 Sampling and greedy mode estimation from a DPPNET 1 : function SAMPLE ( k , A ) 2 : S ← A 3 : while |S| < k do 4 : v ← DPPNET ( S ) 5 : if sampling then 6 : i ∼Multinomial ( v/‖v‖ ) 7 : else if greedy mode then 8 : i← argmax v 9 : S ← S ∪ { i } 10 : return S Exact methods that significantly speed up sampling by leveraging specific structure in the DPP kernel have also been developed ( Mariet & Sra , 2016b ; Gartrell et al. , 2017 ) . Of particular interest is the dual sampling method introduced in Kulesza & Taskar ( 2012 ) : if the DPP kernel can be composed as an inner product over a finite basis , i.e . there exists a feature matrix Φ ∈ RN×D such that the DPP kernel is given by L = ΦΦ > , exact sampling can be done in O ( ND2 +NDk2 +D2k3 ) . However , MCMC sampling requires variable amounts of sampling rounds , which is unfavorable for parallelization ; dual DPP sampling requires an explicit feature matrix Φ . Motivated by recent work on modeling set functions with neural networks ( Zaheer et al. , 2017 ; Cotter et al. , 2018 ) , we propose instead to generate approximate samples via a generative network ; this allows for simple parallelization while simultaneously benefiting from recent improvements in specialized architectures for neural network models ( e.g . parallelized matrix multiplications ) . We furthermore show that , extending the abilities of dual DPP sampling , neural networks may take as input variable feature matrices Φ and sample from non-linear kernels L . 3 GENERATING DPP SAMPLES WITH DEEP MODELS . In this section , we build up a framework that allows the O ( N3 ) computational cost associated with DPP sampling to be addressed via approximate sampling with a neural network . Given a positive semi-definite matrix L ∈ RN×N , we take PL to represent the distribution modeled by a DPP with kernel L over the power set of [ N ] : = { 1 , . . . , N } . Given A ⊆ [ N ] , we write Ā = [ N ] \A and LA = [ Lij ] i , j∈A the |A| × |A| submatrix of L indexed by items in A . 3.1 MOTIVATING THE MODEL CHOICE . Although the elegant quality/diversity tradeoff modeled by DPPs is a key reason for their recent success in many different applications , they benefit from other properties that make them particularly well-suited to machine learning problems . We now focus on how these properties can be maintained with a deep generative model with the right architecture . Closure under conditioning . DPPs are closed under conditioning : given A ⊆ Y , the conditional distribution over Y \A given by PL ( S = A ∪B | A ⊆ S ) ( for B ∩A = ∅ ) is also DPP with kernel LA = ( [ ( L+ IĀ ) −1 ] Ā ) −1− I ( Borodin & Rains , 2005 ) . Although conditioning comes at the cost of an expensive matrix inversion , this property make DPPs well-suited to applications requiring diversity in conditioned sets , such as basket completion for recommender systems . Standard deep generative models such as ( Variational ) Auto-Encoders ( Kingma & Welling , 2014 ) ( VAEs ) and Generative Adversarial Networks ( Goodfellow et al. , 2014 ) ( GANs ) would not enable simple conditioning operations during sampling . Instead , we develop a model that given an input set S , returns a prediction vector v ∈ RN such that vi = Pr ( S ∪ { i } ⊆ Y | S ⊆ Y ) where Y ∼ PL : in other words , vi is the marginal probability of item i being included in the final set , given that S is a subset of the final set . Mathematically , we can compute vi as vi = 1− [ ( L+ IS̄ ) −1 ] ii ( 2 ) for i 6∈ S ( Kulesza & Taskar , 2012 ) ; for i ∈ S , we simply set vi = 0 . With this architecture , we sample a set via Algorithm 1 , which allows for trivial basket-completion type conditioning operations . Furthermore , Algorithm 1 can be modified to implement a greedy sampling algorithm without any additional cost . Log-submodularity . As mentioned above , DPPs are included in the larger class of Strongly Rayleigh ( SR ) measures over subsets . Although being SR is a delicate property , which is maintained by only few operations ( Borcea et al. , 2009 ) , log-submodularity3 ( which is implied by SR-ness ) is more robust , as well as a fundamental property in discrete optimization ( Zagoruyko & Komodakis , 2016 ; Buchbinder et al. , 2012 ; Grötschel et al. , 1981 ) and machine learning ( Iyer & Bilmes , 2015 ; Wei et al. , 2015 ) . Crucially , we show in the following that ( log ) -submodularity can be inherited by a generative model trained on a log-submodular distribution : THEOREM 1 . Let P be a strictly submodular function over subsets of Y , and Q be a function over the same space such that DTV ( P , Q ) ≤ min S , T 6=∅ , Y 1 4 [ P ( S ) + P ( T ) − P ( S ∪ T ) − P ( S ∩ T ) ) ] ( 3 ) where DTV indicates the total variational distance . Then Q is also submodular . COROLLARY 1.1 . Let PL be a strictly log-submodular DPP over Y and DPPNET be a network trained on the DPP probabilities p ( S ) , with a loss function of the form ‖p− q‖ where ‖ · ‖ is a norm and p ∈ R2N ( resp . q ) is the probability vector assigned by the DPP ( resp . the DPPNET ) to each subset of Y . Let α = max‖x‖∞=1 1‖x‖ . If DPPNET converges to a loss smaller than min S , T 6=∅ , Y 1 4α [ P ( S ) + P ( T ) − P ( S ∪ T ) − P ( S ∩ T ) ) ] , its generative distribution is log-submodular . The result follows directly from Thm . 3 and the equivalence of norms in finite dimensional spaces . REMARK 1 . Cor . 1.1 is generalizable to the KL divergence loss DKL ( P‖Q ) via Pinsker ’ s inequality . For this reason , we train our models by minimizing the distance between the predicted and target probabilities , rather than optimizing the log-likelihood of generative samples under the true DPP . Leveraging the sampling path . When drawing samples from a DPP , the standard DPP sampling algorithm ( Kulesza & Taskar , 2012 , Alg . 1 ) generates the sample as a sequence , adding items one after the other until reaching a pre-determined size4 , similarly to Alg . 1 . We take advantage of this by recording all intermediary subsets generated by the DPP when sampling training data : in practice , instead of training on n subsets of size k , we train on kn subsets of size 0 , . . . , k − 1 . Thus , our model is very much like an unrolled recurrent neural network .

Summary: Determinantal Point Processes provide an efficient and elegant way to sample a subset of diverse items from a ground set. This has found applications in summarization, matrix approximation, minibatch selection. However, the naive algorithm for DPP takes time O(N^3), where N is the size of the ground set. The authors provide an alternative model DPPNet for sampling diverse items that preserves the elegant mathematical properties (closure under conditioning, log-submodularity) of DPPs while having faster sampling algorithms.