Noise Regularization for Conditional Density Estimation

1 INTRODUCTION . While regression analysis aims to describe the conditional mean E [ y|x ] of a response y given inputs x , many problems such as risk management and planning under uncertainty require gaining insight about deviations from the mean and their associated likelihood . The stochastic dependency of y on x can be captured by modeling the conditional probability density p ( y|x ) . Inferring such a density function from a set of empirical observations { ( xn , yn ) } Nn=1 is typically referred to as conditional density estimation ( CDE ) and is the focus of this paper . In the recent machine learning literature , there has been a resurgence of interest in high-capacity density models based on neural networks ( Dinh et al. , 2017 ; Ambrogioni et al. , 2017 ; Kingma & Dhariwal , 2018 ) . Since this line of work mainly focuses on the modelling of images based on large scale data sets , over-fitting and noisy observations are of minor concern in this context . In contrast , we are interested in CDE in settings where data may be scarce and noisy . When combined with maximum likelihood estimation , the flexibility of such high-capacity models results in over-fitting and poor generalization . While regression typically assumes Gaussian conditional noise , CDE uses expressive distribution families to model deviations from the conditional mean . Hence , the overfitting problem tends to be even more severe in CDE than in regression . Classical regularization of the neural network weights such as weight decay ( Pratt & Hanson , 1989 ) has been shown to be effective for regression and classification . However , in the context of CDE , the output of the neural network merely controls the parameters of a density model such as a Gaussian Mixture or Normalizing Flow . This makes the standard regularization methods in the parameter space less effective and harder to analyze . Aiming to address this issue , we propose and analyze noise regularization , a method well-studied in the context of regression and classification , for the purpose of conditional density estimation . In that , the paper attempts to close a gap in previous research . By adding small random perturbations to the data during training , the conditional density estimate is smoothed and tends to generalize better . In fact , we show that adding noise during maximum likelihood estimation is equivalent to penalizing the second derivatives of the conditional log-probability . Visually , the respective regularization term punishes very curved or even spiky density estimators in favor of smoother variants , which proves to be a favorable inductive bias in many applications . Moreover , under some regularity conditions , we show that the proposed regularization scheme is asymptotically consistent , converging to the unbiased maximum likelihood estimator . This does not only support the soundness of the proposed method but also endows us with useful insight in how to set the regularization intensity relative to the data dimensionality and training set size . Overall , the proposed noise regularization scheme is easy to implement and agnostic to the parameterization of the CDE model . We empirically demonstrate its effectiveness on three different neural network based models . The experimental results show that noise regularization outperforms other regularization methods significantly and consistently across various data sets . Finally , we demonstrate that , when properly regularized , neural network based CDE is able to improve upon state-of-the art non-parametric estimators , even when only 400 training observations are available . 2 BACKGROUND . Density Estimation . Let X be a random variable with probability density function ( PDF ) p ( x ) defined over the domain X ⊆ Rdx . Given a collection D = { x1 , ... , xn } of observations sampled from p ( x ) , the goal is to find a good estimate f̂ ( x ) of the true density function p. In parametric estimation , the PDF f̂ is assumed to belong to a parametric family F = { f̂θ ( · ) |θ ∈ Θ } where the density function is described by a finite dimensional parameter θ ∈ Θ . The standard method for estimating θ is maximum likelihood estimation , wherein θ is chosen so that the likelihood of the data D is maximized . This is equivalent to minimizing the Kullback-Leibler divergence between the empirical data distribution pD ( x ) = 1n ∑n i=1 δ ( ||x − xi|| ) ( i.e. , mixture of point masses in the observations xi ) and the parametric distribution f̂θ : θ∗ = arg max θ∈Θ n∑ i=1 log f̂θ ( xi ) = arg min θ∈Θ DKL ( pD||f̂θ ) ( 1 ) From a geometric perspective , ( 1 ) can be viewed as an orthogonal projection of pD ( x ) onto F w.r.t . the reverse KL-divergence . Hence , ( 1 ) is also commonly referred to as an M-projection ( Murphy , 2012 ; Nielsen , 2018 ) . In contrast , non-parametric density estimators make implicit smoothness assumptions through a kernel function . The most popular non-parametric method , kernel density estimation ( KDE ) , places a symmetric density function K ( z ) , the so-called kernel , on each training data point xn ( Rosenblatt , 1956 ; Parzen , 1962 ) . The resulting density estimate reads as q̂ ( x ) = 1 nhd ∑n i=1K ( x−xi h ) . One popular choice of K ( · ) is a Gaussian K ( z ) = ( 2π ) − d2 exp ( − 12z 2 ) . Beyond the appropriate choice of K ( · ) , a central challenge is the selection of the bandwidth parameter h which controls the smoothness of the estimated PDF ( Li & Racine , 2007 ) . Conditional Density Estimation ( CDE ) . Let ( X , Y ) be a pair of random variables with respective domains X ⊆ Rdx and Y ⊆ Rdy and realizations x and y . Let p ( y|x ) = p ( x , y ) /p ( x ) denote the conditional probability density of y given x . Typically , Y is referred to as a dependent variable ( explained variable ) and X as conditional ( explanatory ) variable . Given a dataset of observations D = { ( xn , yn ) } Nn=1 drawn from the joint distribution ( xn , yn ) ∼ p ( x , y ) , the aim of conditional density estimation ( CDE ) is to find an estimate f̂ ( y|x ) of the true conditional density p ( y|x ) . In the context of CDE , the KL-divergence objective is expressed as expectation over p ( x ) : Ex∼p ( x ) [ DKL ( p ( y|x ) ||f̂ ( y|x ) ) ] = E ( x , y ) ∼p ( x , y ) [ log p ( y|x ) − log f̂ ( y|x ) ] ( 2 ) Corresponding to ( 1 ) , we refer to the minimization of ( 2 ) w.r.t . θ as conditional M-projection . Given a dataset D drawn i.i.d . from p ( x , y ) , the conditional MLE following from ( 2 ) can be stated as θ∗ = arg min θ − n∑ i=1 log f̂θ ( yi|xi ) ( 3 ) 3 RELATED WORK . The first part of this section discusses relevant work in the field of CDE , focusing on high-capacity models that make little prior assumptions . The second part relates our approach to previous regularization and data augmentation methods . Non-parametric CDE . A vast body of literature in statistics and econometrics studies nonparametric kernel density estimators ( KDE ) ( Rosenblatt , 1956 ; Parzen , 1962 ) and the associated bandwidth selection problem , which concerns choosing the appropriate amount of smoothing ( Silverman , 1982 ; Hall et al. , 1992 ; Cao et al. , 1994 ) . To estimate conditional probabilities , previous work proposes to estimate both the joint and marginal probability separately with KDE and then computing the conditional probability as their ratio ( Hyndman et al. , 1996 ; Li & Racine , 2007 ) . Other approaches combine non-parametric elements with parametric elements ( Tresp , 2001 ; Sugiyama & Takeuchi , 2010 ; Dutordoir et al. , 2018 ) . Despite their theoretical appeal , non-parametric density estimators suffer from poor generalization in regions where data is sparse ( e.g. , tail regions ) , causing rapid performance deterioration as the data dimensionality increases ( Scott & Wand , 1991 ) . CDE based on neural networks . Most work in machine learning focuses on flexible parametric function approximators for CDE . In our experiments , we use the work of Bishop ( 1994 ) and Ambrogioni et al . ( 2017 ) , who propose to use a neural network to control the parameters of a mixture density model . A recent trend in machine learning are latent density models such as cGANs ( Mirza & Osindero , 2014 ) and cVAEs ( Sohn et al. , 2015 ) . Although such methods have been shown successful for estimating distributions of images , the probability density function ( PDF ) of such models is intractable . More promising in this sense are normalizing flows ( Rezende & Mohamed , 2015 ; Dinh et al. , 2017 ; Trippe & Turner , 2018 ) , since they provide the PDF in tractable form . We employ a neural network controlling the parameters of a normalizing flow as our third CDE model to showcase the empirical efficacy of our regularization approach . Regularization . Since neural network based CDE models suffer from severe over-fitting when trained with the MLE objective , they require proper regularization . Classical regularization of the parameters such as weight decay ( Pratt & Hanson , 1989 ; Krogh & Hertz , 1992 ; Nowlan & Hinton , 1992 ) , l1/l2-penalties ( Mackay , 1992 ; Ng , 2004 ) and Bayesian priors ( Murray & Edwards , 1993 ; Hinton & Van Camp , 1993 ) have been shown to work well in the regression and classification setting . However , in the context of CDE , it is less clear what kind of inductive bias such a regularization imposes on the density estimate . In contrast , our regularization approach is agnostic w.r.t . parametrization and is shown to penalize strong variations of the log-density function . Regularization methods such as dropout are closely related to ensemble methods ( Srivastava et al. , 2014 ) . Thus , they are orthogonal to our work and can be freely combined with noise regularization . Adding noise during training . Adding noise during training is a common scheme that has been proposed in various forms . This includes noise on the neural network weights or activations ( Wan et al. , 2013 ; Srivastava et al. , 2014 ; Gal & Uk , 2016 ) and additive noise on the gradients for scalable MCMC posterior inference ( Welling & Teh , 2011 ; Chen et al. , 2014 ) . While this line of work corresponds to noise in the parameter space , other research suggests to augment the training data through random and/or adversarial transformations of the data ( Sietsma & Dow , 1991 ; Burges & Schölkopf , 1996 ; Goodfellow et al. , 2015 ; Yuan et al. , 2017 ) . Our approach transforms the training observations by adding small random perturbations . While this form of regularization has been studied in the context of regression and classification problems ( Holmstrom & Koistinen , 1992a ; Webb , 1994 ; Bishop , 1995 ; Natarajan et al. , 2013 ; Maaten et al. , 2013 ) , this paper focuses on the regularization of CDE . In particular , we build on top of the results of Webb ( 1994 ) showing that training with noise corresponds to a penalty on strong variations of the log-density and extend previous consistency results for regression of Holmstrom & Koistinen ( 1992a ) to the more general setting of CDE . To our best knowledge , this is also the first paper to evaluate the empirical efficacy of noise regularization for density estimation .

Summary: The paper considers the problem of parametric conditional density estimation, i.e. given a set of points {(x_n, y_n)} drawn from a distribution $p(x,y)$, the task is to estimate the conditional distribution p(x|y). The paper considers parametric estimation where in given a parametrized family of distributions f_{theta} we wish to minimize the likelihood of seeing the given data over theta. The parametric family in a lot of applications consists of highly expressive families like neural networks, which leads to the issue of overfitting in small data regimes. This has been tackled via regularization over the parameter space which might be hard to interpret as the associated inductive bias is not well understood and depends on the parametric family under consideration. On the other hand the paper proposes to add explicit noise in the examples used during training, i.e. irrespective of the optimization procedure (which could be mini-bath sgd) the paper proposes to draw examples from the data set, explicitly add noise onto the examples and create a proxy objective over the augmented data set.