Beyond Smoothness: Incorporating Low-Rank Analysis into Nonparametric Density Estimation

√ n ) in L1 error . In contrast , the standard histogram estimator can converge at a rate slower than 1/ d √ n on the same class of densities . We also introduce a new nonparametric latent variable model based on the Tucker decomposition . A rudimentary implementation of our estimators experimentally demonstrates a considerable performance improvement over the standard histogram estimator . We also provide a thorough analysis of the sample complexity of our Tucker decomposition-based model and a variety of other results . Thus , our paper provides solid theoretical foundations for extending low-rank techniques to the nonparametric setting . 1 Introduction . Nonparametric density estimators are density estimators capable of estimating a density p while making few to no assumptions on p. Two commonly used nonparametric density estimators include the histogram estimator and the kernel density estimator ( KDE ) . A common characteristic of these two estimators is that they make estimation tractable via a hyperparameter that relates to smoothness , namely bin width and bandwidth . Large bin width or bandwidth allows for good control of estimation error * at the cost of increased approximation error via large bin volume for the histogram and smoothing for KDEs . Selection of this parameter is crucial for estimator performance . In fact a recent survey on bandwidth selection for KDEs found at least 30 methods for setting this value along with a few surveys dedicated to the topic [ 15 ] . While nonparametric density estimation has been shown to be effective for many tasks , it has been observed empirically that estimator performance typically declines as data dimensionality increases , a manifestation of the curse of dimensionality . For the histogram and KDE this phenomenon has concrete mathematical analogs . For example , these estimators are only universally consistent† if * For an estimator V restricted space a of densities P , the estimation error refers to the difference between ‖V − p‖ and minq∈P ‖p− q‖ , where p is the target density . This is similar to estimator variance . †A density estimator is universally consistent if it asymptotically recovers any density . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . n → ∞ and h → 0 , with nhd → ∞ , where n is the number of samples , d is the data dimension , and h is the bin width for the histogram and bandwidth parameter for the KDE [ 14 ] . One may then wonder whether there exists some other way to constrain model capacity so as to alleviate this exponential dependence on dimension . In this paper we theoretically analyze the advantages of including a constraint akin to matrix/tensor rank in a nonparametric density estimator . We do so by analyzing histogram-style estimators ( estimators that output a density that are piecewise constant on bins defined by a grid ) that are enforced to have a low-rank PARAFAC or Tucker decomposition . Enforcing this low-rank constraint in the histogram estimator allows for much faster rates on h→ 0 , while still controlling estimation error . This can remove the exponential penalty on rate of convergence that dimensionality produces with the standard histogram estimator . The bulk of this work focuses on analysis of hypothetical estimators that , while offering large statistical advantages over the standard histogram estimator , are not computationally tractable . At the end of this work we include experiments demonstrating that low-rank histograms , computed using an off-the-shelf nonnegative tensor factorization library [ 23 ] , consistently outperform the standard histogram estimator . 1.1 Density Models . Here we introduce the low-rank models that will be used to constrain our estimators . The first is a multi-view model . A multi-view model p is a finite mixture p = ∑k i=1 wifi whose components are separable densities fi = pi,1 ⊗ pi,2 ⊗ · · · ⊗ pi , d ‡ [ 19 , 1 , 37 , 36 ] . A multi-view model has the following form p ( x1 , x2 , . . . , xd ) = k∑ i=1 wipi,1 ( x1 ) pi,2 ( x2 ) · · · pi , d ( xd ) . ( 1 ) A multi-view model with one component is typically called a naive Bayes model . For the estimators we propose , the component marginals pi , j will have the form of one-dimensional histograms and the number of components k will be limited to restrict estimator capacity . When k = 1 the model is equivalent to a naive Bayes model , p ( x1 , x2 , . . . , xd ) = p1 ( x1 ) p2 ( x2 ) · · · pd ( xd ) . When the component marginals pi , j are histograms increasing k expands the set of potential estimates from naive Bayes models when k = 1 to all possible histogram estimates . The models in this paper are motivated by nonnegative tensor factorizations so we term them generally as nonparametric nonnegative tensor factorization ( NNTF ) models . The previous model was related to nonnegative PARAFAC [ 34 ] . Our second model is based on the nonnegative Tucker decomposition [ 22 ] . A density in this model utilizes d collections , F1 , . . . , Fd , of k one-dimensional densities , Fi = { pi,1 , . . . , pi , k } , and some probability measure that randomly selects one density from each Fi . This measure a can be represented by a tensor W ∈ Rk ×d where the probability of selecting ( p1 , i1 , . . . , pd , id ) from F1×· · ·×Fd isWi1 , ... , id . To sample from this model we first randomly select the marginal distributions p1 , i1 , . . . , pd , id according to W , and an observation is sampled according to the d-dimensional distribution p1 , i1 ⊗ p2 , i2 ⊗ · · · ⊗ p2 , i2 . The density of this model is p ( x1 , x2 , . . . , xd ) = k∑ i1=1 · · · k∑ id=1 Wi1 , ... , idp1 , i1 ( x1 ) p2 , i2 ( x2 ) · · · pd , id ( xd ) . ( 2 ) We are unaware of previous works investigating this model for general probability distributions so we will simply term it the Tucker model . Again we will investigate estimators where the component marginals are one-dimensional histograms and small k corresponds to reduced model capacity . We remark that Tucker decompositions typically have a rank vector [ k1 , . . . , kd ] which for our model would mean that each Fi contains ki vectors and W would lie in Rk1×···×kd . This sort of rank restriction could be used in our methods however , for simplicity , we just set k1 = k2 = · · · = kd , k. ‡For two functions f , g let f ⊗ g : ( x , y ) 7→ f ( x ) g ( y ) . This is analogous to the tensor product of L2 functions . 1.2 Overview of Results . The principal contribution of this paper is to analyze the advantage of incorporating rank restriction into density estimation . In Section 2.1 we precisely introduce the multi-view histogram and a histogram based on the Tucker model . With these models we investigate how quickly we can let bin width h go to zero and rank k grow , in relation to the amount of data n , while still being able to control estimation error and select an estimator that is nearly optimal for the allowable set . For the multi-view histogram we show that one can control estimation error so long as k/h is asymptotically dominated by n and for the Tucker histogram we need k/h + kd to be asymptotically dominated by n ( we are omitting logarithmic factors here for convenience ) . This stands in stark contrast to standard space of histogram estimators which requires 1/hd to be asymptotically dominated by n. We furthermore show that these estimators are universally consistent and that these rates can not be significantly improved . For a second style of analysis we provide finite-sample bounds on the convergence of NNTF histogram estimators . We then construct a class of universally consistent density estimators that converge at rate Õ ( 1/ 3 √ n ) § on all densities that are a multi-view model with Lipschitz continuous component marginals . Note that the NNTF histogram estimators we construct do not require the target density to be an NNTF model to function well . Our estimators will select a good estimator so long as there exists any NNTF histogram that approximates the target density well ; so our hypothetical estimators “ fail elegantly ” in some sense . We further show that the standard histogram can converge at a rate ω ( 1/ d √ n ) ¶ on the same class of densities . In Section 3 we experimentally investigate the efficacy of using NNTF histograms on real-world data . In lieu of the computationally intractable methods we investigated in our theoretical analyses , we use an existing low-rank nonnegative Tucker factorization algorithm to fit an NNTF histogram to data . Surprisingly even this method outperforms the standard histogram estimator with very high statistical significance . Lastly we mention that this paper is an extension of [ 42 ] and contains a fair amount of overlap with that text . 1.3 Previous Work . Nonparametric density estimation has been extensively studied with the histogram estimator and KDE being some of the most well-known methods . There do exist , however , alternative methods for density estimation , e.g . the forest density estimator [ 24 ] and k-nearest neighbor density estimator [ 25 ] . The L1 , L2 , and L∞ convergence of the histogram and KDE has been studied extensively [ 14 , 9 , 40 , 16 ] . The KDE is generally regarded as the superior density estimator , with some mathematical justification [ 14 , 35 ] . Numerous modifications and extensions of the KDE have been proposed including utilizing variable bandwidth [ 39 ] , robust KDEs [ 20 , 41 , 44 ] , methods for enforcing support boundary constraints [ 33 ] , and a supervised variant [ 43 ] . The work [ 46 ] investigated using KDEs for nonparametric mixture modeling . Finally we mention [ 21 ] that demonstrated that uniform convergence of a KDE to its population estimate suffered when the intrinsic dimension of the data was lower than the ambient dimension , a phenomenon seemingly at odds with the curse of dimensionality . For our review of NNTF models we also include a general review of tensor/matrix factorizations since both can be viewed being low-rank models . In particular , for the multi-view model we have the following analogy k∑ i=1 wipi,1 ( x1 ) pi,2 ( x2 ) · · · pi , d ( xd ) ∼ k∑ i=1 λivi,1 ⊗ vi,2 ⊗ · · · ⊗ vi , d . ( 3 ) A great deal of work has gone into leveraging low-rank assumptions to improve matrix estimation , particularly in the field of compressed sensing [ 10 , 30 ] . The most basic version of compressed sensing is concerned with estimating a “ tall ” vector x from a “ short ” vector y , Ax where A is a known “ short and fat ” matrix . One can recover y if it is sparse and A satisfies a property known as the restricted isometry property ( RIP ) [ 47 , 7 ] . These methods can be extended to the estimation of matrices when x is a low-rank matrix [ 30 , 26 , 27 ] . In this extension A is an order-3 tensor which acts §fn ∈ Õ ( gn ) ⇐⇒ ∃k s.t . fn ∈ O ( gn log k gn ) ¶fn ∈ ω ( gn ) ⇐⇒ |fn/gn| → ∞ in probability . In d-dimensional space the standard histogram can converge slower than 1/ d √ n. as a linear operator on x and satisfies an adjusted form of RIP . RIP commonly arises from matrices and tensors whose entries are random . Because of this compressed sensing techniques are useful in settings where one wants to estimate x from random linear transforms of x . For example , in matrix completion observing the ( i , j ) -th entry of x can be represented as an inner product of x with an indicator matrix ei , j , i.e . 〈x , ei , j〉F . Thus the random observed indices ( i1 , j1 ) , ( i2 , j2 ) , . . . can be represented as random matrices ei1 , j1 , ei2 , j2 , . . . which are then stacked into A . Now A is a random linear operator that is applied to x to represent the observation of random entries of x and the methods of compressed sensing can be used to recover x. Compressed sensing has also proven useful for multivariate regression and autoregressive models [ 26 , 27 ] . Such techniques don ’ t appear to be extensible to histogram estimation due to the lack of a linear sampling scheme . General matrix/tensor factorization , including nonnegative matrix/tensor factorizations , has been extensively studied despite being inherently difficult due to non-convexity . The identifiability and recovery of mixtures of product distributions was studied in [ 18 , 38 , 13 ] . The special case of mixtures of power distributions has also been investigated [ 29 , 45 ] . The works [ 11 , 5 ] present potential theoretical grounds for avoiding the computational difficulties of nonnegative matrix factorization . One notable approach to tensor factorization is to assume , in the tensor representation in ( 3 ) , that d ≥ 3 and the collections of vectors v1 , j , . . . , vk , j are linearly independent for all j . Under this assumption we are guaranteed that the factorization ( 3 ) is unique [ 1 , 18 ] . In [ 4 ] the authors present a method for recovering this factorization efficiently and demonstrate its utility for a variety of tasks . This work was extended in [ 36 ] to recover a multi-view KDE satisfying an analogous linear independence assumption and theoretically analyze the estimator ’ s convergence to the true low-rank components . In [ 36 ] the authors investigate the sample complexity of their estimator but do not demonstrate that their technique has potential for improving rates for nonparametric density estimation in general . In [ 37 ] it was observed that using low-rank embeddings can improve density estimation . A multi-view histogram was investigated in [ 17 ] where the authors present an identifiability result and algorithm for recovering latent factors of the distribution . The related works [ 2 , 3 ] consider a low-rank characteristic function as an approach to improving nonparametric density estimation . Though earlier works have observed that a low-rank approach improves nonparametric density estimation [ 37 , 36 , 17 , 2 , 3 ] , we are the first to demonstrate this through theoretical analysis of sample complexity . Finally we note that the Tucker decomposition has been utilized in Bayesian statistics [ 32 ] . We are unaware of any literature on factoring functions in Rd → R in a Tucker-inspired as we do in ( 2 ) .

Summary: The authors consider the density estimation problem in multiple dimensions. They study the convergence properties of a low-rank estimator: the estimator is the sum of a few tensor products of univariate histograms. They give asymptotic and finite sample bounds for convergence. When the true density is a sum of k tensor products of Lipschitz densities, the authors show an $L_1$ convergence rate of $n^{-⅓}.$