Determinantal point processes based on orthogonal polynomials for sampling minibatches in SGD

∗Alphabetical order †Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . Consider minimizing an empirical loss min θ∈Θ 1 N · N∑ i=1 L ( zi , θ ) + λ ( θ ) , ( 1 ) with some penalty λ : Θ 7→ R+ . Many learning tasks , such as regression and classification , are usually framed that way [ 1 ] . When N 1 , computing the gradient of the objective in ( 1 ) becomes a bottleneck , even if individual gradients ∇θL ( zi , θ ) are cheap to evaluate . For a fixed computational budget , it is thus tempting to replace vanilla gradient descent by more iterations but using an approximate gradient , obtained using only a few data points . Stochastic gradient descent ( SGD ; [ 2 ] ) follows this template . In its simplest form , SGD builds an unbiased estimator at each iteration of gradient descent , independently from past iterations , using a minibatch of random samples from the data set . Theory [ 3 ] and practice suggest that the variance of the gradient estimators in SGD should be kept as small as possible . It is thus natural that variance reduction for SGD has been a rich area of research ; see for instance the detailed references in [ 4 , Section 2 ] . In a related vein , determinantal point processes ( DPPs ) are probability distributions over subsets of a ( typically large or infinite ) ground set that are known to yield samples made of collectively diverse items , while being tractable both in terms of sampling and inference . Originally introduced in electronic optics [ 5 ] , they have been turned into generic statistical models for repulsion in spatial statistics [ 6 ] and machine learning [ 7 , 8 ] . In ML , DPPs have also been shown to be efficient sampling tools ; see Section 2 . Importantly for us , there is experimental evidence that minibatches in ( 1 ) drawn from DPPs and other repulsive point processes can yield gradient estimators with low variance for advanced learning tasks [ 4 , 9 ] , though a conclusive theoretical result has remained elusive . In particular , it is hard to see the right candidate DPP when , unlike linear regression , the objective function does not necessarily have a geometric interpretation . Our contributions are as follows . We combine continuous DPPs based on orthogonal polynomials [ 10 ] and kernel density estimators built on the data to obtain two gradient estimators ; see Section 3 . We prove that their variance isOP ( p− ( 1+1/d ) ) , where p is the size of the minibatch , d is the dimension of data ; see Section 4 . This provides theoretical backing to the claim that DPPs yield variance reduction [ 4 ] over , say , uniformly sampling without replacement . In passing , the combination of analytic tools –orthogonal polynomials– , and an essentially discrete subsampling task –minibatch sampling– sheds light on new ways to build discrete DPPs for subsampling . Finally , we demonstrate our theoretical results on simulated data in Section 5 . A cornerstone of our approach is to utilise orthogonal polynomials to construct our sampling paradigm , interweaving between the discrete set of data points and the continuum in which the orthogonal polynomials reside . A few words are in order regarding the motivation for our choice of techniques . Roughly speaking , we would like to use a DPP that is tailored to the data distribution at hand . Orthogonal Polynomial Ensembles ( OPEs ) provide a natural way of associating a DPP to a given measure ( in this case , the probability distribution of the data points ) , along with a substantive body of mathematical literature and tools that can be summoned as per necessity . This makes it a natural choice for our purposes . Notation . Let data be denoted by D : = { z1 , . . . , zN } , and assume that the zi ’ s are drawn i.i.d . from a distribution γ on Rd . Assume γ is compactly supported , with support D ⊂ [ −1 , 1 ] d bounded away from the border of [ −1 , 1 ] d. Assume also that γ is continuous with respect to the Lebesgue measure , and that its density is bounded away from zero on its support . While our assumptions exclude learning problems with discrete labels , such as classification , we later give experimental support that our estimators yield variance reduction in that case too . We define the empirical measure γ̂N : = N −1 · ∑N i=1 δzi , where δzi is the delta measure at the point zi ∈ Rd . Clearly , γ̂N → γ in P ( Rd ) ; under our operating assumption of compact support , this amounts to convergence in P ( D ) . For simplicity , we assume that no penalty is used in ( 1 ) , but our results will extend straightforwardly . We denote the gradient of the empirical loss by ΞN = ΞN ( θ ) : = N−1 · ∑N i=1∇θL ( zi , θ ) . A minibatch is a ( random ) subset A ⊂ [ N ] of size |A| = p N such that the random variable ΞA = ΞA ( θ ) : = ∑ i∈A wi∇θL ( zi , θ ) , ( 2 ) for suitable weights ( wi ) , provides a good approximation for ΞN . 2 Background and relevant literature . Stochastic gradient descent . Going back to [ 2 ] , SGD has been a major workhorse for machine learning ; see e.g . [ 1 , Chapter 14 ] . The basic version of the algorithm , applied to the empirical risk minimization ( 1 ) , is to repeatedly update θt+1 ← θt − ηtΞA ( θt ) , t ∈ N , ( 3 ) where ηt is a ( typically decreasing ) stepsize , and ΞA is a minibatch-based estimator ( 2 ) of the gradient of the empirical risk function ( 1 ) , possibly depending on past iterations . Most theoretical analyses assume that for any θ , ΞA ( θ ) in the t-th update ( 3 ) is unbiased , conditionally on the history of the Markov chain ( θt ) so far . For simplicity , we henceforth make the assumption that ΞA does not depend on the past of the chain . In particular , using such unbiased gradients , one can derive a nonasymptotic bound [ 3 ] on the mean square distance of θt to the minimizer of ( 1 ) , for strongly convex and smooth loss functions like linear regression and logistic regression with ` 2 penalization . More precisely , for ηt ∝ t−α and 0 < α < 1 , [ 3 , Theorem 1 ] yields E‖θt − θ ? ‖2 ≤ f ( t ) ( E‖θ0 − θ ? ‖2 + σ2 L ) + C σ2 tα , ( 4 ) where C , L > 0 are problem-dependent constants , f ( t ) = O ( e−tα ) , and σ2 = E [ ‖ΞA ( θ ? ) ‖2|D ] is the trace of the covariance matrix of the gradient estimator , evaluated at the optimizer θ ? of ( 1 ) . The initialization bias is thus forgotten subexponentially fast , while the asymptotically leading term is proportional to σ2/tα . Combined with practical insight that variance reduction for gradients is key , theoretical results like ( 4 ) have motivated methodological research on efficient gradient estimators [ 4 , Section 2 ] , i.e. , constructing minibatches so as to minimize σ2 . In particular , repulsive point processes such as determinantal point processes have been empirically demonstrated to yield variance reduction and overall better performance on ML tasks [ 4 , 9 ] . Our paper is a stab at a theoretical analysis to support these experimental claims . The Poissonian benchmark . The default approach to sample a minibatch A ⊂ [ N ] is to sample p data points from D uniformly , with or without replacement , and take wi = 1/p constant in ( 2 ) . Both sampling with or without replacement lead to unbiased gradient estimators . A similar third approach is Poissonian random sampling . This simply consists in starting from A = ∅ , and independently adding each element of D to the minibatch A with probability p/N . The Poisson estimator ΞA , Poi is then ( 2 ) , with constant weights again equal to 1/p . When p N , which is the regime of SGD , the cardinality of A is tightly concentrated around p , and ΞA , Poi has the same fluctuations as the two default estimators , while being easier to analyze . In particular , E [ ΞA , Poi|D ] = ΞN and Var [ ΞA , Poi|D ] = OP ( p−1 ) ; see Appendix S1 for details . DPPs as ( sub ) sampling algorithms . As distributions over subsets of a large ground set that favour diversity , DPPs are intuitively good candidates at subsampling tasks , and one of their primary applications in ML has been as summary extractors [ 7 ] . Since then , DPPs or mixtures thereof have been used , e.g. , to generate experimental designs for linear regression , leading to strong theoretical guarantees [ 11 , 12 , 13 ] ; see also [ 14 ] for a survey of DPPs in randomized numerical algebra , or [ 15 ] for feature selection in linear regression with DPPs . When the objective function of the task has less structure than linear regression , it has been more difficult to prove that finite DPPs significantly improve over i.i.d . sampling . For coreset construction , for instance , [ 16 ] manages to prove that a projection DPP necessarily improves over i.i.d . sampling with the same marginals [ 16 , Corollary 3.7 ] , but the authors stress the disappointing fact that current concentration results for strongly Rayleigh measures ( such as DPPs ) do not allow yet to prove that DPP coresets need significantly fewer points than their i.i.d . counterparts [ 16 , Section 3.2 ] . Even closer to our motivation , DPPs for minibatch sampling have shown promising experimental performance [ 4 ] , but reading between the lines of the proof of [ 4 , Theorem 1 ] , a bad choice of DPP can even yield a larger variance than i.i.d . sampling ! For such unstructured problems ( compared to , say , linear regression ) as coreset extraction or lossagnostic minibatch sampling , we propose to draw inspiration from work on continuous DPPs , where faster-than-i.i.d . error decays have been proven in similar contexts . For instance , orthogonal polynomial theory motivated [ 10 ] to introduce a particular DPP , called a multivariate orthogonal polynomial ensemble , and prove a faster-than-i.i.d . central limit theorem for Monte Carlo integration of smooth functions . While resorting to continuous tools to study a discrete problem such as minibatch sampling may be unexpected , we shall see that the assumption that the size of the dataset is large compared to the ambient dimension crucially allows to transfer variance reduction arguments . 3 DPPs , and two gradient estimators . Since we shall need both discrete and continuous DPPs , we assume that X is either Rd or D , and follow [ 17 ] in introducing DPPs in an abstract way that encompasses both cases . After that , we propose two gradient estimators for SGD that build on a particular family of continuous DPPs introduced in [ 10 ] for Monte Carlo integration , called multivariate orthogonal polynomial ensembles .

Summary: This paper presents a framework for variance reduction in stochastic gradient descent (SGD) that involves detrimental point processes (DPPs) based on orthogonal polynomials.  This framework uses a class of continuous DPPs based on orthogonal polynomials, and kernel density estimators built from the data, to perform minibatch sampling in SGD.  Theoretical analysis of this approach proves a bound on the variance for two gradient estimators, leading to variance reduction compared to uniform minibatch sampling without replacement.  This approach is empirically validated using experiments performed on synthetic simulated data.