Beyond Smoothness: Incorporating Low-Rank Analysis into Nonparametric Density Estimation

√ n ) in L1 error . In contrast , the standard histogram estimator can converge at a rate slower than 1/ d √ n on the same class of densities . We also introduce a new nonparametric latent variable model based on the Tucker decomposition . A rudimentary implementation of our estimators experimentally demonstrates a considerable performance improvement over the standard histogram estimator . We also provide a thorough analysis of the sample complexity of our Tucker decomposition-based model and a variety of other results . Thus , our paper provides solid theoretical foundations for extending low-rank techniques to the nonparametric setting . 1 Introduction . Nonparametric density estimators are density estimators capable of estimating a density p while making few to no assumptions on p. Two commonly used nonparametric density estimators include the histogram estimator and the kernel density estimator ( KDE ) . A common characteristic of these two estimators is that they make estimation tractable via a hyperparameter that relates to smoothness , namely bin width and bandwidth . Large bin width or bandwidth allows for good control of estimation error * at the cost of increased approximation error via large bin volume for the histogram and smoothing for KDEs . Selection of this parameter is crucial for estimator performance . In fact a recent survey on bandwidth selection for KDEs found at least 30 methods for setting this value along with a few surveys dedicated to the topic [ 15 ] . While nonparametric density estimation has been shown to be effective for many tasks , it has been observed empirically that estimator performance typically declines as data dimensionality increases , a manifestation of the curse of dimensionality . For the histogram and KDE this phenomenon has concrete mathematical analogs . For example , these estimators are only universally consistent† if * For an estimator V restricted space a of densities P , the estimation error refers to the difference between ‖V − p‖ and minq∈P ‖p− q‖ , where p is the target density . This is similar to estimator variance . †A density estimator is universally consistent if it asymptotically recovers any density . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . n → ∞ and h → 0 , with nhd → ∞ , where n is the number of samples , d is the data dimension , and h is the bin width for the histogram and bandwidth parameter for the KDE [ 14 ] . One may then wonder whether there exists some other way to constrain model capacity so as to alleviate this exponential dependence on dimension . In this paper we theoretically analyze the advantages of including a constraint akin to matrix/tensor rank in a nonparametric density estimator . We do so by analyzing histogram-style estimators ( estimators that output a density that are piecewise constant on bins defined by a grid ) that are enforced to have a low-rank PARAFAC or Tucker decomposition . Enforcing this low-rank constraint in the histogram estimator allows for much faster rates on h→ 0 , while still controlling estimation error . This can remove the exponential penalty on rate of convergence that dimensionality produces with the standard histogram estimator . The bulk of this work focuses on analysis of hypothetical estimators that , while offering large statistical advantages over the standard histogram estimator , are not computationally tractable . At the end of this work we include experiments demonstrating that low-rank histograms , computed using an off-the-shelf nonnegative tensor factorization library [ 23 ] , consistently outperform the standard histogram estimator . 1.1 Density Models . Here we introduce the low-rank models that will be used to constrain our estimators . The first is a multi-view model . A multi-view model p is a finite mixture p = ∑k i=1 wifi whose components are separable densities fi = pi,1 ⊗ pi,2 ⊗ · · · ⊗ pi , d ‡ [ 19 , 1 , 37 , 36 ] . A multi-view model has the following form p ( x1 , x2 , . . . , xd ) = k∑ i=1 wipi,1 ( x1 ) pi,2 ( x2 ) · · · pi , d ( xd ) . ( 1 ) A multi-view model with one component is typically called a naive Bayes model . For the estimators we propose , the component marginals pi , j will have the form of one-dimensional histograms and the number of components k will be limited to restrict estimator capacity . When k = 1 the model is equivalent to a naive Bayes model , p ( x1 , x2 , . . . , xd ) = p1 ( x1 ) p2 ( x2 ) · · · pd ( xd ) . When the component marginals pi , j are histograms increasing k expands the set of potential estimates from naive Bayes models when k = 1 to all possible histogram estimates . The models in this paper are motivated by nonnegative tensor factorizations so we term them generally as nonparametric nonnegative tensor factorization ( NNTF ) models . The previous model was related to nonnegative PARAFAC [ 34 ] . Our second model is based on the nonnegative Tucker decomposition [ 22 ] . A density in this model utilizes d collections , F1 , . . . , Fd , of k one-dimensional densities , Fi = { pi,1 , . . . , pi , k } , and some probability measure that randomly selects one density from each Fi . This measure a can be represented by a tensor W ∈ Rk ×d where the probability of selecting ( p1 , i1 , . . . , pd , id ) from F1×· · ·×Fd isWi1 , ... , id . To sample from this model we first randomly select the marginal distributions p1 , i1 , . . . , pd , id according to W , and an observation is sampled according to the d-dimensional distribution p1 , i1 ⊗ p2 , i2 ⊗ · · · ⊗ p2 , i2 . The density of this model is p ( x1 , x2 , . . . , xd ) = k∑ i1=1 · · · k∑ id=1 Wi1 , ... , idp1 , i1 ( x1 ) p2 , i2 ( x2 ) · · · pd , id ( xd ) . ( 2 ) We are unaware of previous works investigating this model for general probability distributions so we will simply term it the Tucker model . Again we will investigate estimators where the component marginals are one-dimensional histograms and small k corresponds to reduced model capacity . We remark that Tucker decompositions typically have a rank vector [ k1 , . . . , kd ] which for our model would mean that each Fi contains ki vectors and W would lie in Rk1×···×kd . This sort of rank restriction could be used in our methods however , for simplicity , we just set k1 = k2 = · · · = kd , k. ‡For two functions f , g let f ⊗ g : ( x , y ) 7→ f ( x ) g ( y ) . This is analogous to the tensor product of L2 functions . 1.2 Overview of Results . The principal contribution of this paper is to analyze the advantage of incorporating rank restriction into density estimation . In Section 2.1 we precisely introduce the multi-view histogram and a histogram based on the Tucker model . With these models we investigate how quickly we can let bin width h go to zero and rank k grow , in relation to the amount of data n , while still being able to control estimation error and select an estimator that is nearly optimal for the allowable set . For the multi-view histogram we show that one can control estimation error so long as k/h is asymptotically dominated by n and for the Tucker histogram we need k/h + kd to be asymptotically dominated by n ( we are omitting logarithmic factors here for convenience ) . This stands in stark contrast to standard space of histogram estimators which requires 1/hd to be asymptotically dominated by n. We furthermore show that these estimators are universally consistent and that these rates can not be significantly improved . For a second style of analysis we provide finite-sample bounds on the convergence of NNTF histogram estimators . We then construct a class of universally consistent density estimators that converge at rate Õ ( 1/ 3 √ n ) § on all densities that are a multi-view model with Lipschitz continuous component marginals . Note that the NNTF histogram estimators we construct do not require the target density to be an NNTF model to function well . Our estimators will select a good estimator so long as there exists any NNTF histogram that approximates the target density well ; so our hypothetical estimators “ fail elegantly ” in some sense . We further show that the standard histogram can converge at a rate ω ( 1/ d √ n ) ¶ on the same class of densities . In Section 3 we experimentally investigate the efficacy of using NNTF histograms on real-world data . In lieu of the computationally intractable methods we investigated in our theoretical analyses , we use an existing low-rank nonnegative Tucker factorization algorithm to fit an NNTF histogram to data . Surprisingly even this method outperforms the standard histogram estimator with very high statistical significance . Lastly we mention that this paper is an extension of [ 42 ] and contains a fair amount of overlap with that text . 1.3 Previous Work . Nonparametric density estimation has been extensively studied with the histogram estimator and KDE being some of the most well-known methods . There do exist , however , alternative methods for density estimation , e.g . the forest density estimator [ 24 ] and k-nearest neighbor density estimator [ 25 ] . The L1 , L2 , and L∞ convergence of the histogram and KDE has been studied extensively [ 14 , 9 , 40 , 16 ] . The KDE is generally regarded as the superior density estimator , with some mathematical justification [ 14 , 35 ] . Numerous modifications and extensions of the KDE have been proposed including utilizing variable bandwidth [ 39 ] , robust KDEs [ 20 , 41 , 44 ] , methods for enforcing support boundary constraints [ 33 ] , and a supervised variant [ 43 ] . The work [ 46 ] investigated using KDEs for nonparametric mixture modeling . Finally we mention [ 21 ] that demonstrated that uniform convergence of a KDE to its population estimate suffered when the intrinsic dimension of the data was lower than the ambient dimension , a phenomenon seemingly at odds with the curse of dimensionality . For our review of NNTF models we also include a general review of tensor/matrix factorizations since both can be viewed being low-rank models . In particular , for the multi-view model we have the following analogy k∑ i=1 wipi,1 ( x1 ) pi,2 ( x2 ) · · · pi , d ( xd ) ∼ k∑ i=1 λivi,1 ⊗ vi,2 ⊗ · · · ⊗ vi , d . ( 3 ) A great deal of work has gone into leveraging low-rank assumptions to improve matrix estimation , particularly in the field of compressed sensing [ 10 , 30 ] . The most basic version of compressed sensing is concerned with estimating a “ tall ” vector x from a “ short ” vector y , Ax where A is a known “ short and fat ” matrix . One can recover y if it is sparse and A satisfies a property known as the restricted isometry property ( RIP ) [ 47 , 7 ] . These methods can be extended to the estimation of matrices when x is a low-rank matrix [ 30 , 26 , 27 ] . In this extension A is an order-3 tensor which acts §fn ∈ Õ ( gn ) ⇐⇒ ∃k s.t . fn ∈ O ( gn log k gn ) ¶fn ∈ ω ( gn ) ⇐⇒ |fn/gn| → ∞ in probability . In d-dimensional space the standard histogram can converge slower than 1/ d √ n. as a linear operator on x and satisfies an adjusted form of RIP . RIP commonly arises from matrices and tensors whose entries are random . Because of this compressed sensing techniques are useful in settings where one wants to estimate x from random linear transforms of x . For example , in matrix completion observing the ( i , j ) -th entry of x can be represented as an inner product of x with an indicator matrix ei , j , i.e . 〈x , ei , j〉F . Thus the random observed indices ( i1 , j1 ) , ( i2 , j2 ) , . . . can be represented as random matrices ei1 , j1 , ei2 , j2 , . . . which are then stacked into A . Now A is a random linear operator that is applied to x to represent the observation of random entries of x and the methods of compressed sensing can be used to recover x. Compressed sensing has also proven useful for multivariate regression and autoregressive models [ 26 , 27 ] . Such techniques don ’ t appear to be extensible to histogram estimation due to the lack of a linear sampling scheme . General matrix/tensor factorization , including nonnegative matrix/tensor factorizations , has been extensively studied despite being inherently difficult due to non-convexity . The identifiability and recovery of mixtures of product distributions was studied in [ 18 , 38 , 13 ] . The special case of mixtures of power distributions has also been investigated [ 29 , 45 ] . The works [ 11 , 5 ] present potential theoretical grounds for avoiding the computational difficulties of nonnegative matrix factorization . One notable approach to tensor factorization is to assume , in the tensor representation in ( 3 ) , that d ≥ 3 and the collections of vectors v1 , j , . . . , vk , j are linearly independent for all j . Under this assumption we are guaranteed that the factorization ( 3 ) is unique [ 1 , 18 ] . In [ 4 ] the authors present a method for recovering this factorization efficiently and demonstrate its utility for a variety of tasks . This work was extended in [ 36 ] to recover a multi-view KDE satisfying an analogous linear independence assumption and theoretically analyze the estimator ’ s convergence to the true low-rank components . In [ 36 ] the authors investigate the sample complexity of their estimator but do not demonstrate that their technique has potential for improving rates for nonparametric density estimation in general . In [ 37 ] it was observed that using low-rank embeddings can improve density estimation . A multi-view histogram was investigated in [ 17 ] where the authors present an identifiability result and algorithm for recovering latent factors of the distribution . The related works [ 2 , 3 ] consider a low-rank characteristic function as an approach to improving nonparametric density estimation . Though earlier works have observed that a low-rank approach improves nonparametric density estimation [ 37 , 36 , 17 , 2 , 3 ] , we are the first to demonstrate this through theoretical analysis of sample complexity . Finally we note that the Tucker decomposition has been utilized in Bayesian statistics [ 32 ] . We are unaware of any literature on factoring functions in Rd → R in a Tucker-inspired as we do in ( 2 ) .

Summary: This work is concerned with the (high-dimensional) density estimation problem.  While most minimax rates  for  the  problem  are  expressed  in  terms  of  some  measure  of  smoothness  that  the  unknown  distribution is assumed to verify, the current approach adds more structure, by requiring that the joint density be tensor-decomposable (in a PARAFAC or in a Tucker sense), with Lipschitz components.  The authors show(theoretical analysis) that this subverts the smoothness-only minimax rates together with their curse of dimensionality. In order to illustrate their method, the authors further perform experiments in which –perhaps controversially– they relax the estimator search to an L2 problem.