Noise Regularization for Conditional Density Estimation
Authors: Jonas Rothfuss
Fabio Ferreira
Simon Boehm
Simon Walther
Maxim Ulrich
Tamim Asfour
Andreas Krause
Venue:  (2020)

Abstract: Modelling statistical relationships beyond the conditional mean is crucial in many settings. Conditional density estimation (CDE) aims to learn the full conditional probability density from data. Though highly expressive, neural network based CDE models can suffer from severe over-fitting when trained with the maximum likelihood objective. Due to the inherent structure of such models, classical regularization approaches in the parameter space are rendered ineffective. To address this issue, we develop a model-agnostic noise regularization method for CDE that adds random perturbations to the data during training. We demonstrate that the proposed approach corresponds to a smoothness regularization and prove its asymptotic consistency. In our experiments, noise regularization significantly and consistently outperforms other regularization methods across seven data sets and three CDE models. The effectiveness of noise regularization makes neural network based CDE the preferable method over previous non- and semi-parametric approaches, even when training data is scarce. 

Under review as a conference paper at ICLR 2020
NOISE REGULARIZATION FOR CONDITIONAL DENSITY ESTIMATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Modelling statistical relationships beyond the conditional mean is crucial in many settings. Conditional density estimation (CDE) aims to learn the full conditional probability density from data. Though highly expressive, neural network based CDE models can suffer from severe over-fitting when trained with the maximum likelihood objective. Due to the inherent structure of such models, classical regu- larization approaches in the parameter space rendered ineffective. To address this issue, we develop a model-aggostic noise regularization method for CDE that adds random perturbations to the data during training. We demonstrate that the proposed approach corresponds to smoothness regularization and prove its asymptotic consistency. In our experiments, noise regularization significantly and consistently outperforms other regularization methods across seven data sets and three CDE models. The effectiveness of noise regularization makes neural net- work based CDE the preferable method over previous non- and semi-parametric approaches, even when training data is scarce.
INTRODUCTION
While regression analysis aims to describe the conditional mean E[y[a]] of a response y given inputs r, many problems such as risk management and planning under uncertainty require gaining insight about deviations from the mean and their associated likelihood. The stochastic dependency of on x can be captured by modeling the conditional probability density p(y]ar). Inferring such density function from a set of empirical observations { yr. 1 b. is typically referred to as conditional density estimation (CDE) and the focus of this paper.
In the recent machine learning literature, there has been resurgence of interest in high-capacity density models based on neural networks (Dinh al., 2017 Ambrogioni et al.) 2017) Kingma & Dhariwal 2018). Since this line of work mainly focuses on the modelling of images based on large scale data sets, fver-fitting and noisy observations are of minor concern in this context. In contrast, we are interested in CDE in settings where data may be scarce and noisy. When combined with maximum likelihood estimation, the flexibility of such high-capacity models results in over-fitting and poor generalization. While regression typically assumes Gaussian conditional noise, CDE uses expressive distribution families to model deviations from the conditional mean. Hence, the over- fitting problem tends be even
severe in CDE than in regression.
Classical regularization of the neural network weights such as weight decay (Pratt& & Hanson 1989) has been shown be effective for regression and classification. However, in the context of CDE, the output of the neural network merely controls the parameters of a density model such as a Gaussian Mixture or Normalizing Flow. This makes the standard regularization methods in the parameter space less effective and harder to analyze.
Aiming address this issue, we propose and analyze noise regularization, a method well-studied in the context of regression and classification, for the purpose of conditional density estimation. In that, the paper attempts to close a gap in previous research. By adding small random perturbations to the data during training, the conditional density estimate is smoothed and tends to generalize better. In fact, we show that adding noise during maximum likelihood estimation is equivalent to penalizing the second derivatives of the conditional prg-probability. Visually, the respective regularization term punishes very curved or even spiky density estimators in favor of smoother variants, which proves to be favorable inductive bias in many applications. Moreover, under some regularity conditions, we show that the proposed regularization scheme is asymptotically consistent, converging to the unbiased maximum likelihood estimator. This does not only support the soundness of the proposed

---

Under review as a conference paper at ICLR 2020
method but also endows with useful insight in how to set the regularization intensity relative to the data dimensionality and training set size.
Overall, the proposed noise regularization scheme is easy to implement and agnostic the parameterization of the CDE model. We empirically demonstrate its effectiveness on three different neural network based models. The experimental results show that noise regularization outperforms other regularization methods significantly and consistently across various data sets. Finally, we demonstrate that, when properly regularized, neural network based CDE is able to improve upon state-of-the art non-parametric estimators, even when only 400 training observations are available.
2 BACKGROUND
Density Estimation. Let X be a random variable with probability density function (PDF) p(x) defined over the domain X' S Rd. Given collection D 121,...; of observations sampled from p(z), the goal is to find a good estimate f(x) of the true density function p. In parametric estimation, the PDF f is assumed to belong to a parametric family F = {fo()) E }} where the density function is described by a finite dimensional parameter 0 O. The standard method for estimating θ is maximum likelihood estimation. wherein @ is chosen so that the likelihood of the data D maximized. This equivalent to minimizing the Dallback-Leibler divergence between the empirical data distribution pD = Σi= 0(122 mell) (i.e., mixture of point masses in the observations zz) and the parametric distribution fo:
0" arg max BE0
log fo(2:) = arg min DK (Pollfo) liee
(1)
From a geometric perspective, 1 can be viewed as orthogonal projection of pD(x) onto F w.r.t
the reverse KL--divergence. Hence, (T) is also commonly referred to as an M-projection (Murphy) 2012 Nielsen 2018). In contrast, non-parametric density estimators make implicit smoothness assumptions through a kernel function. The most popular non-parametric method, kernel density estimation (KDE), places symmetric density function K (=). the so-called kernel, on each training data point I' (Rosenblatt]
1956
Parzen 1962). The resulting density estimate reads as ((x)
whd Zi=1 ( yond the appropriate choice of K (-), a central challenge is the selection of the bandwidth parameter
One popular choice of K(-) is Gaussian K(z) (2x) exp (-]=²). Be-
which controls the smoothness of the estimated PDF (Li & Racine 2007).
Conditional Density Estimation (CDE). Let (X, be a pair of random variables with respective domains X' C Rd and y C Rd and realizations x and y. Let p(ylæt) = p(a,, y)/p(x) denote the conditional probability density of y given x. Typically, is referred to a dependent variable (explained variable) and X as conditional (explanatory) variable. Given a dataset of observations D ((x,, yn N drawn from the joint distribution (Zn, Vn) ~ p(x, y). the aim of conditional density estimation (CDE) is to find an estimate f(ylæ)) of the true conditional density p(z[(z).
In the context of CDE, the KL-divergence objective is expressed expectation over p(x):
Ez<p(x) Dri(P(u) (x(() (+)l(())))))))) = ((xx))p((e logp(y)>>) log f(y]=)
(2)
Corresponding to we refer to the minimization of (2) w.r.t. 0 as conditional M-projection. Given a dataset DD drawn i.i.d. from p(z, y), the conditional MLE following from (2) can be stated as
g = arg min
log fo(uilz;)
(3)
RELATED WORK
The first part of this section discusses relevant work in the field of CDE, focusing on high-capacity models that make little prior assumptions. The second part relates our approach to previous regular- ization and data augmentation methods.
Non-parametric CDE. A vast body of literature in statistics and econometrics studies nonpara- metric kernel density estimators (KDE) (Rosenblatt) 1956] Parzen 1962) and the associated band- width selection problem, which concerns choosing the appropriate amount of smoothing (Silverman)
2

---

Under review as a conference paper at ICLR 2020
1982 Hall et al. 1992 Cao al., 1994). To estimate conditional probabilities, previous work pro- poses to estimate both the joint and marginal probability separately with KDE and then computing the conditional probability as their ratio (Hyndman et al. 1996 Li & Racine (007). Other ap-
proaches combine non-parametric elements with parametric elements (Tresp) 2001] Sugiyama & Takeuchi 2010 Dutordoir al., 2018). Despite their theoretical appeal, non-parametric density timators suffer from poor generalization in regions where data is sparse (c.g., tail regions), causing rapid performance deterioration as the data dimensionality increases (Scout & Wand 1991).
CDE based on neural networks. Most work in machine learning focuses on flexible paramet- ric function approximators for CDE. In our experiments, we use the work of [Bishop| (1994) and Ambrogioni et al. (2017), who propose to use a neural network to control the parameters of a mix- ture density model. A recent trend in machine learning are latent density models such cGANS (Mizaa & Osindero 2014) and cVAEs (Sohn et al. 2015). p. Although such methods have been shown successful for estimating distributions of images, the probability density function (PDF) of such models is intractable. More promising in this sense are normalizing flows (Rezende & Mohamed 2013 Dinh et al., 2017 Trippe & Turner 2018), since they provide the PDF in tractable form. We employ a neural network controlling the parameters of a normalizing flow as our third CDE model to showcase the empirical efficacy of our regularization approach.
Regularization. Since neural network based CDE models suffer from severe over-fitting when trained with the MLE objective, they require proper regularization. Classical regularization of the parameters such as weight decay (Pratt & Hanson 1989 Krogh & Hertz 1992; Nowlan & Hinton
1992). 4/l2p-ennaltiss (Mackay) 1992 Ng 2004) and Bayesian priors (Murray & Edwards 1993
Hinton& & Van Camp 1993) have been shown to work well in the regression and classification set- ting. However, in the context of CDE, it is less clear what kind of inductive bias such regular- ization imposes on the density estimate. In contrast, our regularization approach is agnostic w.r.t parametrization and is shown to penalize strong variations of the log-density function.
Regularization methods such as dropout are closely related to ensemble methods (Srivastava et al,, 2014. Thus, they are orthogonal to our work and can be freely combined with noise regularization.
Adding noise during training. Adding noise during training is a common scheme that has been proposed in various forms. This includes noise on the neural network weights or activations (Wan) ct al. 2013 Srivastava et al. 2014; Gal & Uk 2016) and additive noise on the gradients for scalable MCMC posterior inference (Weling & Teh 2011: et al. 2014). While this line of work corre- sponds to noise in the parameter space, other research suggests to augment the training data through random and/or adversarial transformations of the data (Sietsma & Dow 1991 Burges & Schölkopf
1996 Goodfellow et al., 2015; Yuan et al.] (017).. Our approach transforms the training observa- tions by adding small random perturbations. While this form of regularization has been studied in the context of regression and classification problems (Holmstrom & Koistinen 1992a Webb 1994] Bishop 1995 Natarajan et al. 2013: Maaten et al. (013), this paper focuses on the regularization of CDE. In particular, we build on top of the results of Webb (1994) showing that training with noise corresponds to penalty on strong variations of the log-density and extend previous consistency results for regression of Holmstrom & Koistinen (1992a) to the more general setting of CDE. To our best knowledge, this is also the first paper to evaluate the empirical efficacy of noise regularization for density estimation.
4 NOISE REGULARIZATION
When considering expressive families of conditional densities, standard maximum likelihood esti- mation of the model parameters is ill suited. As can be observed in Figure [ simply minimizing the negative log-likelihood of the data leads to severe over-fitting and poor generalization beyond the training data. Hence, is necessary to impose additional inductive bias, for instance, in the form of regularization. Unlike in regression or classification, the form of inductive bias imposed by pop- ular regularization techniques such as weight decay (Krogh & Hertz] 1991] Kukačka et al.) (2017) is less clear in the CDE setting, where the neural network weights often only indirectly control the probability density through a unconditional density model, e.g., a Gaussian Mixture.
We propose to add noise perturbations to the data points during the optimization of the log likelihood objective. This can be understood as replacing the original data points (z,, Jt) by random variables d. = x; + Ex and ÿJ = y/2 E, where the perturbation vectors are sampled from noise distribu-
3

---

Under review as a conference paper at ICLR 2020
noise_sid 0.00
noise_ sld 0.05
noise_ste sid 0,20
webbib denatt
0.8
0.
-2
Figure I: Conditional MDN density estimate (red) and true conditional density (green) for different noise regularization intensities h- {0.0, 0.05, 0.2}. The MDN has been fitted with 3000 samples drawn from a conditional Gaussian.
tions (5,) and (yy) respectively. Further, we choose the noise be zero centered as well as identically and independently distributed among the data dimensions, with standard deviation h:
(() [<] =0 and Bg=K(e) [$E 1²T
This can be seen as data augmentation, where "synthetic" data is generated by randomly perturbing the original data. Since the supply noise vectors technically unlimited, an arbitrary large aug- mented data set can be generated by repetitively sampling data points from D, and adding random perturbation vector to the respective data point. This procedure is formalized in Algorithm [|
For notational brevity, we set 2 == X' x y, 2 == (x y and denote fo(2) = fo(ulz). The presented noise regularization approach is agnostic to whether we concerned with unconditional or conditional MLE. Thus, the generic notation also allows us to generalize the results to both settings (derived in the remainder of the paper).
Algorithm (Conditional) MLE with Noise Regularization -Mini-Batch Gradient Descent
Algorithm (Conditional) MLE with Noise
Regularization Generie Procedure
Require: D =f21, .... 2n) noise intensity h
Require: learning rate mini-batch size m Initialize 0
Require: {21, J. noise intensity h
Require: number of perturbed samples
2: while 0 not converged do
1: for j= j= to do
3: Sample minibatch {E1..... Zm C. D
2: Select i E (1,-..; n with equal prob.
4: for to do
3: Draw perturbation ~ K
5: Draw perturbation &
4: Set žj 24 hg
Set 2, 2j hg
5: return arg mingee
Ej=1 log log fo(2))
7:
+ 0 V プド log (2,)
8: return optimized parameter
When considering highly flexible parametric families such Mixture Density Networks (MDNs) Bishop 1994). the maximum likelihood solution in line 3 Algorithm i no longer tractable. In such case, typically resorts to numerical optimization techniques such mini-batch gradient descent and variations thereof. In this context, the generic procedure Algorithm formed into simple extensions of mini-batch gradient descent the MLE objective (see Algorithm 2}. Specifically, each mini-batch is perturbed with i.i.d. noise before computing the MI.E objective function (forward pass) and the respective gradients (backward pass).
[ can be trans-
4.1 VARIABLE NOISE AS SMOOTHNESS REGULARIZATION
Intuitively, the previously presented variable noise can be interpreted as "smearing" the data points during the maximum likelihood estimation. This alleviates the jaggedness of the density estimate arising from an un-reuularized maximum likelihood objective in flexible density classes. We will now give this intuition a formal foundation, by mathematically analyzing the effect of the noise perturbations.
Before discussing the particular effects of randomly perturbing the data during conditional maxi- mum likelihood estimation, we first analyze noise regularization be a loss function over set of data points D {&1,... .... 2n which can be partitioned into a sum of losses ((D) = En= 1(2;), corresponding to each data point र5: The expected loss ((2;+)), resulting from adding random perturbations, can be approximated by second order Taylor expansion around
more general case. Let 4D)
4

---

Under review as a conference paper at ICLR 2020
2,. Using the assumption about & in (4). the expected loss an be written
h? + +O(E x))()))) ur(H()
K(E) [1= E)] ((2,) E K(s)
H H
(5)
(z) l. the Hessian of w.r.t 2, evaluated at Z4. Assuming that the noise & is small in its magnitude, ( is negligible. This effect has been observed earlier by Webb (994) and Bishop 1994). See Appendix A for derivations.
where ((zi) is the loss without noise and H
When concerned with maximum likelihood estimation of a conditional density fo(ulæ)). the loss function coincides with the negative conditional log- geg-likelihood [y,, 24) log fo (1i(2;) Let the standard deviation of the additive data noise εx. Ey be h, and h, respectively. Maximum likelihood estimation (MLE) with data noise is equivalent to minimizing the loss
h 2
dy
82 log fo(y)x)
h
de
8² log fo( ax()dz6)
((D) = Σ log (o(l))
Ty())y()
2
In that, the first term corresponds the standard MLE objective, while the other two terms constitute a form of smoothness regularization. The second term in (6) penalizes large negative second deriva- tives of the conditional log density estimate log (ol w.r.t. As the MLE objective pushes the density estimate towards high densities and strong concavity in the data points yy, the regularization term counteracts this tendency to over-fit and overall smoothes the fitted distribution. The third term penalizes large negative second derivatives w.r.t the conditional variable x, thereby regularizing the sensitivity of the density estimate to changes in the conditional variable. The intensity of the noise regularization can be controlled through the variance (M2 and (.) of the random perturbations.
Figure T illustrates the effect of the introduced noise regularization scheme MDN estimates. Plain maximum likelihood estimation (left) leads to strong over-fitting, resulting in a spiky distribution that generalizes poorly beyond the training data. In contrast, training with noise regularization (center and right) results in smoother density estimates that are closer to the true conditional density.
4.2 CONSISTENCY OF NOISE REGULARIZATION
We now establish asymptotic consistency results for the proposed noise regularization. In particular, we show that, under some regularity conditions, concerning integrability and decay of the noise regularization, the solution of Algorithm converges to the asymptotic MLE solution.
Let fo(2): Rdd x θ + (0, oc) continuous function of and 6. Moreover, we assume that the parameter space is compact. In the classical MLE setting, the idcalized loss, corresponding to a (conditional) M-projection of the true data distribution onto the parametric family, reads as
1(0) Eppx) [lo fo()
As we typically just have finite number of samples from p(=), the respective empirical estimate (,())) - log fo(),, 2; i.id p(2) used as training objective. Note that we now define the loss as function of 0, and, for fixed 6, treat 4, (0) as a random variable. Under some regularity conditions, one can invoke the uniform law of large numbers to show consistency of the empirical
ML objective in the sense that supece (0) I(0) a.s, 0 (see Appendix B for details).
In case of the presented noise regularization scheme, the maximum likelihood estimation is per- formed using on the augmented data } rather than the original data {z,). For our analy- sis, we view Algorithm [ from slightly different angle. In fact, the data augmentation proce- dure of uniformly selecting a data point from (=1,..,, 2n} and perturbing with a noise vector
drawn from K can be viewed as drawing i.i.d. samples from kernel density estimate q! (z) Dn
1 K ( (2-2)) Hence, maximum likelihood estimation with variable noise can be under- Sist hag stood as
forming a kernel density estimate ( of the training data
2. followed by a (conditional) M-projection of ((h) onto the parametric family.
5

---

Under review as a conference paper at ICLR 2020
In that, step 2 aims to find the l that minimizes the following objective:
t (0) (c) [Lo
(z)
(8)
Since 8 is generally intractable, samples are drawn from the kernel density estimate, forming the following Monte Carlo approximation of (8) which corresponds to the loss in line Algorithm
log fo(g)) Eg an (z)
We are concerned with the consistency of the training procedure in Algorithm similar to the classical MLE consistency result discussed above. Hence, we need to show that supace (0) - [(0) a.8, O as n, r + co. We begin our argument by decomposing the problem
into easier sub-problems. particular, the triangle inequality is used to obtain the following upper bound:
sup OE
(00) (0) 1(0) - sup O
f(h) n. (0)
1/42 (0) + sup oE
(0) [(0]
(10)
Note that (0) is based on samples from the kernel density estimate, which are obtained by adding random noise vectors & ~ K(.) to our original training data. Since we can sample an unlimited amount of such random noise vectors, r can be chosen arbitrarily high. This allows us to make supsge ((()) (0) (0)) arbitrary small by the uniform law of large numbers. In order to make supace (1) (0) I(0) small in the limit - oo, the sequence of bandwidth parameters h, needs to chosen appropriately. Such results can then combined using a union bound argument. In the following we outline the steps leading us to the desired results. In that, the proof methodology similar to Holmstrom & Koistinen (199b)). While they show consistency results for regression with quadratic loss function, our proof deals with generic and inherently unbounded ligeliild objectives and thus holds for much more general class of learning problems. The full proofs can be found in the Appendix.
(0)
Initially, we have to make asymptotic integrability assumptions that ensure that the expectations in & (0) and ((0) are well-behaved in the limit (see Appendix C for details). Given respective integrability, we are able to obtain the following proposition.
Proposition 1 Suppose the regularity conditions
28 and (9) are satisfied, and that
lim
hn
lim
n mh,,)) d oc
(11)
12 1-000
n n=000
Then,
lim sup AE0
(8) - ((0) m
0
(12)
almost surely.
In we find conditions on the asymptotic behavior of the smoothing sequence (h₂)). These con- ditions also give valuable guidance on how to properly choose the noise intensity in line 4 of Algorithm| (see Section 4.3 for discussion). The result in (12) demonstrates that, under the dis- cussed conditions, replacing the empirical data distribution with a kernel density estimate still results in an asymptotically consistent maximum likelihood objective. However, as previously discussed,
(1)) (0) is intractable and, thus, replaced by its sample estimate ((A). Since we can draw an arbitrary amount of samples from a(ah) we can approximate 1(hh) (0) with arbitrary precision. Given fixed data D of size n nlg, this means that lim, supace (. (
(0) (0)
0 almost surely, by
and the uniform law of large numbers. Since our original goal was to also show consistency for n1 0o, this result is combined with Proposition] T obtaining the following consistency theorem.
Theorem Suppose the regularity conditions compact. Then,
28 and 29 are satisfied, hn fulfills 1 and θ is
lim Iim sup
((h) R.r (0) 1(0) =
(13)
almost surely.

---

Under review as a conference paper at ICLR 2020
In that, lim used to denote the limit superior ("lim sup") of sequence.
Training a (conditional) density model with noise regularization means minimizing (0)
w.r.t. 8. As result of this optimization, one obtains parameter vector 0%h) which we hope is close to the minimizing parameter 0 of the ideal objective function ((0). In the following, we establish asymp- totic consistency results, similar to Theorem [ in the parameter space. Therefore we first have to formalize the concept of closeness and optimality in the parameter space. Since a minimizing pa- rameter 0 of I(0) may not be unique, we define OH {0" [((0*) S 1(0) vo E } as the set of global minimizers of I(6), and d(é, θ*) ming ge [|le - 11 as the distance of an arbitrary parameter 0 to G Based on these definitions, it can be shown that Algorithm I asymptotically consistent in a sense that the minimizer of 0 converges almost surely to the set of optimal parameters Or.
Theorem Suppose the regularity conditions 28 and 29 are satisfied, hn fulfills TT and θ is compact. For r > 0 and n > na, let (0h) E θ be a global minimizer of the empirical objective i()) Then
lim lim d(0th) 12-000 1-000 n,F! (-) =0
(14)
almost surely.
Note that Theorem 2 considers global optimizers, but equivalently holds for compact neighborhoods of local minimum & (see discussion in Appendix C).
4.3 CHOOSING THE NOISE INTENSITY
After discussing the properties of noise regularization, we are interested in how to properly choose the noise intensity h. for different training data sets. Ideally, we would like to choose h so that 1 (0) - I(0) is minimized, which is practically not feasible since I(0) is intractable. Inequality upper bound on this quantity, suggesting to minimize l distance between the ker- nel density estimate (( and the data distribution p(z). This is in turn a well-studied problem in the kernel density estimation literature (see e.g., Devroye & Luc (1977)) Unfortunately, general solutions of this problem require knowing p(z) which is not the case in practice. Under the assump- tion that p(2) and the kernel function K Gaussian, the optimal bandwidth can be derived as 1.06ôn 4-a (Silverman 1986). In that, & denotes the estimated standard deviation the data, the number of data points and d the dimensionality of Z. This formula is widely known as the rule of thumb and often used as a heuristic for choosing h.
30
gives
In addition, the conditions in
give further intuition. The first condition tells us that hn needs to decay towards zero as n becomes large. This reflects the general theme in machine learning that the more data is available, the less inductive bias/ / regularization should be imposed. The second condition suggests that the bandwidth decay must happen rate slower than n à. For instance, the rule of thumb fulfills these two criteria and thus constitutes a useful guideline for selecting h. However, for highly non- non-Gaussian data distributions, the respective h, may decay too slowly and a faster decay rate such as n Iea may be appropriate.
5 EXPERIMENTS
This section provides a detailed experimental analysis of the proposed method, aiming to empiri- cally validate the theoretical arguments outlined previously and investigating the practical efficacy of our regularization approach. In all experiments we use Gaussian pertubations of the data, i.e., K{{) = (0, (0, Since one of the key features of our noise regularization scheme is that is ag- nostic to the choice of model, we evaluate its performance on three different neural network based CDE models: Mixture Density Networks (MDN) (Bishop| 1994), Kernel Mixture Networks (KMN) Ambrogioni et al. 2017) and Normalizing Flows Networks (NFN) (Rezende & Mohamed 2015 Trippe & Turner 2018). In our experiments, we consider both simulated as well real- data sets. In particular, we simulate data from 4-dimensional Gaussian Mixture (d, = 2, d, = 2) and Skew-Normal distribution whose parameters are functionally dependent on (d, = 1, d, = 1). In terms of real- wolldwrldd data, we use the following three data sources:
returns of the Euro Stoxx 50 index conditioned on various stock return factors relevant in finance (d,, 14, d, =)).
Euro Stoxx: Daily
7

---

Under review as a conference paper at ICLR 2020
Mixture Density Network
Kernel Mixture Network
Normalizing Flow
-1.00
1935 Mixture
-3.5
-1.25
-3.00
1.50
-3.75
-3.75
-3.7
-4.00
4.50
-4.25
-422
1.25
4.90
4.50
4.50
10
oumber of tran asservalona
10
100
100
107
107
1.00
1660
1.40
Skow Normal
1.55
1.55
1.55
1.90
150
1.50
1.40
140
1.40
1.35
133
1.35
107
122
10'
13a
207
10a
10
10a
107
10'
10t
201
Figure 2: Comparison of different noise intensity schedules hn and their implications on the perfor-
mance of various CDE models across different training set sizes.
NYC Taxi: Drop-off locations of Manhattan taxi trips conditioned on the pickup location, week- day and time (dg = 6, dy 2).
UCI datasets: Standard data sets from the UCI machine learning repository (Dua & Graft 2017 In particular, Boston Housing (d, = 13, dy = 1), Concrete (dg = 8, dg 1), Energy (d2 9, dy 1).
The reported scores are test log- lieeliiesssodss averaged over at least 5 random seeds alongside the respective standard deviation. For further details regarding the data sets and simulated data, we refer to Appendix E The experiment data and code is available at TODO
5.1 NOISE INTENSITY SCHEDULES
We complement the discussion in 4.3 with an empirical investigation of different schedules of hn-- In particular, we compare a) the rule of thumb hn x b) square root decay schedule h,, x constant bandwidth hn const. (0, oo) and d) no noise regularization, i.c. hn 0. Figure 2 plots the respective test log-likelihoods against an increasing training set size for the two simulated densities Gaussian Mixture and Skew Normal.
TF7
First, we observe that bandwidth rates which conform with the decay conditions seem to converge in performance to the non-lagularized maximum likelihood estimator (red) as n becomes large. This reflects the theoretical result of Theorem [ Second, fixed bandwidth across (green), violating [IT. imposes asymptotic bias and thus saturates in performance vastly before its counterparts. Third, hypothesized, the relatively slow decay of h, through the rule of thumb works better for data distributions that have larger similarities to a Gaussian, i.c., in our case the Skew Normal distribution. In contrast, the highly non-Gaussian data from the Gaussian Mixture requires faster decay rates like the square root decay schedule. Most importantly, noise regularization substantially improves the estimator's performance when only little training data available.
5.2 REGULARIZATION COMPARISON
We investigate how the proposed noise regularization scheme compares to classical regular- ization techniques. In particular, we consider an I and le-penalty on the neural network weights as regularization term, the weight decay technique Loshchilov & Hutter 22099 as well Bayesian neural network (Neal 2012) trained with variational inference using a Gaussian prior and posterior Blei et al. 2017).
First, we study the performance of the regularization techniques on our two simulation benchmarks. Figure3 the respective test log-likelihood across different training sizes. For each regu- larization method, the regularization hyper- parameter has been optimized via grid search.
As one would expect, the importance regularization, i.c.. performance difference to un-regularized model, decreases as the amount of training data becomes larger. The noise regularization scheme
Note that an 12 regularizer and weight decay not equivalent since use the adaptive learning rate technique Adam. Loshhhilr & Hutter (2019) for details.
8

---

Under review as a conference paper at ICLR 2020
Mixture Density Network
Keme! Mixture Network
Normalizing Flow
Gauzzien Mature
aed sggiblii hood
10
10P
107
10a
104
10a
number oftai osservation
1.6
1.6
Skew Nomal
1.4-
14-
14
ralie reg. lours)
na reg.
1.2
12
12
reg.
weight decay Veristional Bayen
1.0
10a
16a
107
10'
10a
1.0
10'
Figure 3: Comparison of various regularization methods for three neural network based CDE mod-
els. The models are trained with simulated data sets of different sizes.
Euro Stoxx
NYC Taxi
Boston
Concrete
Energy
noise (ours)
0.03
5.2510.04
-2.49:1..11
-2.92:10.08
-1.04:0.009
MDN
weight decay
3.78=0.06
5.07
-3.29-L0.22
-3.33:10.14
-1,21
11 reg.
3.19=0.19
5.00=0.05
.4.0120.36
-3.8710.29
-1.44:0.22
12 reg.
3.16=0.21
4.99=0.04
4.6440.52
-3.8410.26
-1.55+0..2
Bayes
3.26-0.43
5.08++0.33
-.446+0.47
-3.19+0.21
-1.25+0..3
noise (ours)
3.92-0.01
5.3910.02
-2.22.0.08
3.09-0.006
-..22...06
KMN
weight decay
3.05
5.31
2.69
3.15+00.06
-1.7900.12
11 reg-
3.76=0.04
5.39
-..55+0.13
-..25+0.07
0..82
12 reg.
3.71=0005
5.37
-2.66+0.13
-3.1:10.07
-1.79.0.13
Bayes
3.33=0.02
4.47
-3.40+t..11
4.08
-3.65
noise (ours)
3.90=0.01
5.2010.03
2..810.111
3.0310.13
-..1
weight decay
3.02
5.19+0.03
-3.12+0.39
-.112...14
11.22
NFN
I1 reg.
3.50-0.10
5.12+0.05
12.78
+..52
-1.2940.16
reg.
3.50=0.09
5.13
-14.2249.60
-3,99-0.66
-1.34
Bayes
3.34-0.33
5.10-10.3
-5.9942.45
-3.55:10.4
-1.1111.22
Table Comparison of various regularization methods for three neural network based CDE models across 5 data sets. We report the test log-likelihood and its respective standard deviation (higher
log-likelihood values are better).
yields similar performance across the different CDE models while the other regularizers vary greatly in their performance depending the different models. This reflects the fact that noise regulariza- tion is agnostic to the parameterization of the CDE model while regularizers in the parameter space dependent the internal structure of the model. Most importantly, noise regularization per- forms well across all models and sample sizes. In the great majority of configurations it outperforms the other methods. Especially when little training data is available, noise regularization ensures moderate test eтror while the other approaches mostly fail to do so.
Next, we consider real world data Since now the amount data we can use for dlper--aramen selection, training and evaluation is limited, we use 5-fold cross-validation to select the parameters for each regularization method. The test log-likelihoods, reported in Table [ are averages over different train/test splits and seeds each for initializing the neural networks. The held out test set amounts 20% of the overall data sets. Consistent with the results of the simulation study, noise regularization outperforms the other methods across the great majority of data sets and CDE models.
5.3 CONDITIONAL DENSITY ESTIMATOR BENCHMARK STUDY
We benchmark neural network based density estimators against state-of-the art CDE approaches. While neural networks are the obvious choice when a large amount of training data is available, we pose the questions how such estimators compete against well-established non-maraettric methods in small data regimes. In particular, we compare to the three following CDE methods:
Conditional Kernel Density Estimation (CKDE). Non-parametric method that forms a KDE of
both p(z, y) and p(x) compute its estimate as p(y|z) == p(x, v)/(zx) (i & Racine 2007).

---

Under review as a conference paper at ICLR 2020
Euro Stoxx
NCY Taxi
Boston
Conrete
Energy
train obs.
2536
8000
405
824
615
MDN
4.00:10.03
5.41
-2.39.00.0
-2.89.10.00
-1.04.10.05
KMN
3,98+0.03
5.42:10.02
-2.4410.02
-3.06+0.003
-1.59:0.09
NFN
4.00
5.12=0.03
-2.40:10.04
-2.93+0.02
-1.23+0.066
LSCDE
3.44
4.85+:002
-2.78.0.00
-3.6340.00
-2.66:0.02
CKDE R.O.T.
3.36+:0.01
4.87
-3.12+1.03
-3.78+0.02
-2.90
CKDE CV-ML
3.87+0.01
5.27
-2.76+0..6
-3.35+00.13
-1.14:00.02
NKDE R.O.T
3.16+0.02
0.34
-3.52+0.05
4.08
-3.35+0.03
NKDE CV-ML
3.41
4.08
-3.34
-3.93
-2.21
Table 2: Comparison of conditional density estimators across 5 data sets. Reported is the test log- likelihood and its respective standard deviation (higher log-likelihood values are better).
e-Neighborhood kernel density estimation (NKDE). Non-parametric method that considers only a local subset of training points to form a density estimate.
Least-Squares Conditional Density Estimation (LSCDE). Semi-parametric estimator that com- putes the conditional density linear combination of fixed kernels (Sugiyama Takeuchi 2010).
For the kernel density estimation based methods CKDE and NKDE, we perform bandwidth selection via the rule of thumb (R.O.T) (Silverman 1982, Sheather & Jones [991) and via maximum likeli- hood leave-one-our cross- rross-validation (CV-ML) (Rudemo 1982 Hall et 1992). In case of LSCDE, MDN, KMN and NFN, the respective hyper-parameters are selected via 5-lold cross-validation grid search on the training set. Note that, in contrast to Section 5.2 which focuses on regularization pa- rameters, the grid search here extends to more dyper-parametee
The respective test log- -igliikeldoood scores are listed in Table 2 For the majority of data sets, all three neural network based methods outperform all of the non- and semi- parameeric methods. Perhaps surprisingly, it can be seen that, when properly regularized, neural network based CDE works well even when training data is scarce, such as in case of the Boston Housing data set.
CONCLUSION
This paper addresses conditional density estimation with high-capacity models. In particular, we propose to add small random perturbations to the data during training. We demonstrate that the sulting noise regularization method corresponds to a smoothness regularization and prove its asymp- totic consistency. The experimental results underline the effectiveness of the proposed method, demonstrating that it consistently outperforms other regularization methods across various condi- tional density models and data sets. This makes neural network based CDE the preferable method, even when only little training data is available. While we assess the estimator performance in terms of the test log-likelihood, an interesting question for future research is whether the noise regular- ization also improves the respective uncertainty estimates for downstream tasks such as safe control and decision making.
REFERENCES
Luea Ambrogioni, Umut Güçlü, Marcel A. van Gerven, and Eric Maris. The Kernel Mixture Network: A Nonparametric Method for Conditional Density Estimation of Continuous Random Variables. 2017. URL http:://xxx http://wwww..rgaa////aoa//// 07111
Anděl, Ivan Netuka, and Karel Zvára. On Threshold Autoregressive Processes. Kybernet- ica, 20(2):89-106, 1984. htthttps:: / /mmll
dzlez/
924493//ybbrnettkke ika 20 1984-298222 pdd
Chris M. Bishop.
Training with Noise is Equivalent to Tikhonov Regularization. Neural Com- putation, 1(1):108-116, 1995. ISSN 0899-7667. doi: 16./nee/oeco. 1995.7.1.108.
URL
http:/ / // //www/wwww///..tt/////to mi
1160 neco. 1995. 7.1.108
Christopher M Bishop. Mixture Density Networks. 1994.
10

---

Under review as a conference paper at ICLR 2020
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational Inference: A Review for Statis- ticians. Journal of the American Statistical Association. 1126518):85--8777 877, 2017. doi: 10.
(080016224592217.12887777
URL
1285773
Chris J. C. Burges and Bernhard Schölkopf. Improving the accuracy and speed of support vector ma- chines. In NIPS, pp. 375-381. MIT Press, 1996. /tttttsss///d...mmm wtt.:///rg//ciiitioooooooo
cfm?id-2959034
Ricardo Cao, Antonio Cuevas, and Wensceslao González Manteiga. A comparative study of several smoothing methods in density estimation. Computational Statistics & Data
Analysis, 17(2):153-176 2 1994. ISSN 0167-9473. doi: 10.10160016--947302) 00066-Z.
URL https: /ww.. http://wwwcciecceeeece rect com ccmm/sciennce/atticle/ppp
01
Tianqi Chen, Emily B Fox, and Carlos Guestrin. Stochastic Gradient Hamiltonian Monte Carlo. In ICML, 2014.
Luc Devroye. The equivalence of weak, strong and complete convergence in LI for kernel density estimates. Annals of Statistics, 11(3):896-904, 1983. URL
pdf
Luc. Devroye and Luc. A course in density estimation. Birkhauser, 1987. ISBN 0817633650. URL https://dl. wwtp:/iatioo.no.cooo/./. wtw.:aaiii.....mmmm2m.7555
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In
Proceedings of the International Conference on Learning Representations, 2017. URL http:
7/ //arxiv.oro org. wwwgg//aa//00..000
Dheeru Dua and Casey Graff.
{UCI} Machine Learning Repository, 2017. URL http://
archive. ics www.uciieeee. aoc.eedummm
Vincent Dutordoir, Hugh Salimbeni, Mare Peter Deisenroth, and James Hensman. Gaussian Pro-
cess Conditional Density Estimation. In NeurIPS, 2018. BtRLhttps:///ptt //aperss//pppp.ee./em/ www.nnpp.ccoo.mm
Saaper7750-gguuet -aw.sssiiapoosesssssssesssssssee
process pot.raanooooosssscoonoo...ooooo oy
Yarin Gal and Zg201 @cam Ac Uk. Dropout as a Bayesian Approximation: Representing Model
Uncertainty in Deep Learning Zoubin Ghahramani. In ICML, 2016. URL/tttp://yarin.
Nicolas Gilardi, Samy Bengio, and Mikhail Kanevski. Conditional Gaussian Mixture Models for Environmental Risk Mapping. In NNSP, 2002. UtL/htpp://wwt idi idiap.c... ch
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and Harnessing Adversarial Examples. In ICLR, 2015. URL http://pp/aaarii..rr////////////
Hsi Guang Sung. Gaussian Mixture Regression and Classification. PhD thesis, 2004. URL http:
7/www. stat. rice. www.:ric.eeeueeeeeeeooeeeeee
Peter Hall, IS Marron, and Byeong U Park. Smoothed cross-validation. Probability The-
ory, 92:1-20, 1992. URL htttss://iing //inkk/ www.a/......igee....ooooottataaa
SFBF0120522 pdf
Geoorey E Hinton and Drew Van Camp. Keeping Neural Networks Simple by Minimizing the Description Length of the Weights. In COLT, 1993. URL http:://ww/w. cs. toronto................ edu
"fritz/abspp http:/llttt3.....
I. Holmstrom and P. Koistinen. Using additive noise in back-propagation training. IEEE Transac- tions on Neural Networks, 3(1):24-38, 38, 1992a. ISSN 10459227, doi: 10.1109/72.105155. URL
htttp://ieeexp/e/
ieee org http://acumeett1151111.1
Lasse Holmstrom and Petri Koistinen. Using Additive Noise in Back Propagation Training. IEEE
Transactions on Neural Nenworks, 3(1):24-38, 1992b. ISSN 19410093, doi: 10.1109/72.105415.
URL http::ttpp/eeee ore. ieee. wtt.:eeeorr/oooomeee//////./...e

---

Under review as a conference paper at ICLR 2020
Rob J. Hyndman, David M. Bashtannyk, and Gary K. Grunwald. Estimating and Visualizing Con- ditional Densities. Journal of Computational and Graphical Statistics, 5(4):115, 1996. ISSN 10618600. doi: 10.2307/1390887.. URLhttps::
wwww. jator wttsss/aabeeeeeee
ori ariginccrossee:
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible Ixl Convolu- tions. Technical report, 2018. http:://ax.....///aaaa/aaaa///// 000033337770390
Anders Krogh and John A Hertz.
A Simple Weight Decay Can Improve Gen-
eralization. In NIPS, 1991. URL https: mippeerrr.nnii...mmmm nips. cc/paper/ wei
Anders Krogh and John A Hertz. A Simple Weight Decay Can Improve Gener- alization. Technical report, 1992. URL lbttps://papes..nt cc/paper/ 563 a simple e weight decay can ptw.booeegggeeraliiiiiio...ooo.
Jan Kukačka, Vladimir Golkov, and Daniel Cremers. Regularization for Deep Learning: A Taxon- omy. Technical report, 10 2017. URL http://arxxx wttp:///wrrgaab//////////// 1068
Qi Li and Jeffrey S. Racine. Nonparametric econometrics theory and practice. Princeton Univer- sity Press, 2007.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In ICLR, 2019. URL https:// http:///gtttuuu...ooooo..s..t www..coo/mlccccc. wowschhi//aaamiiiiiiiooooo Adamm and SGDW
Laurens Maaten, Minmin Chen, Stephen Tyree, and Kilian Weinberger. Learning with marginalized corrupted features. In International Conference on Machine Leurning, pp. 410-418, 2013.
David J C Mackay. A Practical Bayesian Framework for Backprop Networks. Neural Computa- fion, 1992. URL httl:ttppp//eseeeeeeeeee wit.sisstsssee. pau.
1. 1. 29.
Mehdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. Technical report, 11 2014. URRpattp://axxiv http://awxww..ggb///////ss /t///1111.1111 1784
Kevin P. Murphy, Machine Learning: A Probabilistic Perspective. 2012.
Alan Murray and Peter Edwards. Synaptic Weight Noise During MI. Learning Enhances Fault-
Tolerance, Generalisation and Learning Trajectory. In NIPS, 1993. URL httpp:://ddfff
pdf
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204, 2013.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Andrew Y Ng. Feature selection, L 1 vs. L 2 regularization, and rotational invariance. In ICML, 2004. URL PRLphtpp///a..oee stw.forsssttffood...rrooooo edu/ ang.ppppes//innntmm.c.a paappperrr//mmmaa.m http:/mmm111111211111 pdf
Frank Nielsen. What an information projection? Notices of the AMS, 2018 doi: 10.1090/
noti1647. bttp://// doi
Steven J. Nowlan and Geoffrey E. Hinton. Simplifying Neural Networks by Soft Weight Sharing.
Neural Computation, 1992. URL http::/wwww......oooooo.ee////t/
surspots.pde
Emanuel Parzen. On Estimation of Probability Density Function and Mode. The Annals of Mathe-
matical Statistics, 1055--666, 9 1962. ISSN 0003-4851, doi: 10. fttooms/1177744:1 177704472.
URL http:://prooeeteuucceeeee hwwpelidddrrrrreeeeeee wwwg/eecciiiio aoms/ www.//os1011. /11777044777
Lorien Y. Pratt and Stephen J. Hanson. Comparing Biases for Minimal Network Construction with Back Propaprroaa,t.on. In NIPS, 1989.
12

---

Under review as a conference paper at ICLR 2020
Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In Proceedings of the 32nd International Conference on Machine Learning, 2015. URL http:
//arxiv. abs httpg//aaa/10..... 05770
Murray Rosenblatt. Remarks on Some Nonparametric Estimates of a Density Function. The An-
nals of Mathematical Statisticcs, 27:3) 837, 9 1956. ISSN 0003-4851. doi: 10.1214/aoms/ 1d. org/ /uc: 1 id. acms /
1177728190. htt.htttpp//proojcteeeeee
Jonas Rothfuss, Fabio Ferreira, Simon Walther, and Maxim Ulrich. Conditional Density Estimation
with Neural Networks: Best Practices and Benchmarks. Technical report, 2019. URL http:
//arxiv. harxiv.crg/abst... ww..or///aaoo wwwgg//b///a 00954
Mats Rudemo. Empirical Choice of Histograms and Kernel Density Estimators, 1982. URL
https: / www. jstor wwtp:orrretttbeebreeereee.
David W. Scott and M. P. Wand. Feasibility of Multivariate Density Estimates. Biometrika, 78(1);
197. 3 1991. ISSN 00063444. doi: 10.2307/2339910. URL httpps:/wwwt jstor org/
stable//333911//rlgiiiis
S. J. Sheather and M. C. Jones. A Reliable Data- Bandwidth Selection Method for Kernel Density Estimation. Journal of the Royal Statistical Society, 33:683-690, 1991. doi: 10.2307/ 2345597. URL https:
www. stor wttp:o/tttateeee55559/999
Jocelyn Sietsma and Robert J.F. Dow. Creating artificial neural networks that generalize. Neural Networks, 4(1):67-79, 1991. ISSN 0893-6080. doi: 10.101660993-60800911 90033-2. URL https:/ wwww. scwencedi com hton://cceence/atticl//pppp.. 0893608091 900332
B. Silverman. On the estimation of a probability density function by the maximum penal-
ized likelihood method. The Annals of Statistics, 10(3):795 810, 1982. URL https://
pzojjetteocldd..rrr/aaalllaaooa heteddfoeeuucdddooo aos a0s111634555 /
B. Silverman. Density estimation for statistics and data analysis. Monographs on Statistics and Applied Probability, 1986.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning Structured Output Representation using Deep Conditional Generative Models. In Advances in Neural Information Processing Systems, pp. 3483 1493,,49,,222 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Ma- chine Learning Research, 15:1929-1955 1958, 2014. URL http::/ wttp://..rgpaaaarsssss..
Sri rrvatttavll1 .4.htmll
Masashi Sugiyama and Ichiro Takeuchi. Conditional density estimation via Least-Squares Density
Ratio Estimation. In Proceedings of the Thirteenth International Conference Artificial Intel-
ligence and Statistics, volume 9, pp. 781-788, 2010. URL http: / eeannnininneennnnnnnennnnnnnn
wust w.l.edu/nneg Btst://mppaprrr//ppppap
f ws/AISTATS200 00g.AA//5000000ggggaaaa1t440
Volker Tresp. Mixtures of Gaussian Processes. In NIPS, 2001. URL
00/ /cc/paper/11 900 wwwues-oo--gattee qauss an www.ppoccssssseem pd
Brian Trippe and Richard Turner. Conditional Density Estimation with Bayesian Normalising Flows. Technical report, 2 2018. httl:/tttpp//arxii.rorr/////
04908
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of Neural Net- works using DropConnect. In ICML, pp. (058-1066, 2 2013. URL http:ttt::/prroeedii mlr. wtt.pprss/v////wnnnnn htmmmll
A.R. Webb. Functional approximation by feed-forward networks: least-squares approach gen- cralization. IEEE Transactions on Neural Nenworks, 5(3):363-371, 1994. ISSN 10459227. doi: 00.110977..86608. URL W.RL.ttp:... htt.th::///eeeeellloo/// feee. aanee..ogg//oceee wwt..o///w..oeeee///00/..0
13

---

Under review as a conference paper at ICLR 2020
Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynam-
ics. In ICML, 2011. URL https: www. www.ics... wci. www. www.o/////////o//icaca..ooa.aa
papers/stoolangeviin v6.pdi.
Halbert White. Learning in Artificial Neural Networks: A Statistical Perspective. Neural Com-
putation, 1(4):425-464, 12 1989. ISSN 0899-7667. doi: 10. 198....999........ 1.889.1...440 1,44425. URL
http://www
htt.://awaa..oao.o.oo..////////a 198...9999..0 1. 425
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial Examples:: Attacks and Defenses for Deep Learning. Technical report, 12 2017. URL http:///arav..orrra////12222211
14

---

Under review as a conference paper at ICLR 2020
A DERIVATION SMOOTHNESS REGULARIZATION
Let I(D) be a loss function over a set of data points D {21, 2n3, which can be partitioned into sum of losses corresponding to each data point Zn
(D( - Σ ((zz)
(15)
Also, each 2; be perturbed by random noise vector & ((E) with zero mean and i.i.d. elements, i.e.
(e) [<] 0 and Eg [EnE]] = -²1
(16)
The resulting loss 1(2; + {) can be approximated by a second order Taylor expansion around 2;
U( + {) ((x)) + 5 vz)( + 2 Ve((=)), + O(E))
(17)
Assuming that the noise & is small in its magnitude, ( may be neglected. The expected loss under (E) follows directly from (7):
(() [[= + E)] = (z)) + K(s)
{ vvl(()))
5. 2 [
$ (
(18)
Using the assumption about g in (6) we can simplify (18 as follows:
E() [(= + {)] = ((z)) + E K(E [S]
T V=((2)
2 K(E)
5 [V 5
(19)
= =((z)) +
2 B
(TH(0))
(20)
821(2)
= =((z)) + B(
ΣΣ
(21)
2
020))2(()
= ==x=)) +
Bg [5]]
821(=)
2
1
ΣΣ
Be [CSEa]
02(())
2
Oz(0)
02(002())
(22)
=1(z;) + 112 Σ
8P1(2)
(23)
2
Oz0)az
n)²
[22;) + (r(H
(24)
2
In that, I(z,) is the loss without noise and H = V2((=)1,, the Hessian of at 2, With z(1 we denote the elements of the column vector 2.
B VANILLA CONDITIONAL MLE ORJECTIVE IS UNIFORMLY CONSISTENT
The objective function corresponding to a conditional M-projection.
1(0) Ep(r.a)) [og fo(l)()))))
(25)
The sample equivalent:
(0)
Σ log (o(((:))) (2),11 iid P(X,Y)
(26)
Corollary Let θ be a compact set and and fo RR x Rm x + (0, 00) continuous in 8 for all (r,y) E ER x R such that Ep(c.)) supuce log fo(y/z) < 00. Then, as n + oo, we have
sup eee
[0) 1(0) a...,
(27)
Proof. The corollary follows directly from the uniform law of large numbers.
15

---

Under review as a conference paper at ICLR 2020
C
CONSISTENCY PROOFS
Lemma Suppose for some c. > 0 there exists a constant BBd) such that
/ [loo,f(())) p(=)dz s BB) < V0
(28)
and there exists an no such that for all no there exists a constant B. such that
/ | fa (z)dz S BB < 00 VeE
(29)
almost surely. Then, the inequality
110 1(0) Sc. (f (1) p(z) dz
sup
(30)
where C. is a constant holds with probability for all no-
Proof of Lemma Using Hoelder's inequality and the nonnegativity of and (h)
we obtain
/ log fo(=) (z) - -p()))dz
[ (0) - -(()
/ log fo(2) (x p(=)ld=
/ fo()) (x) p(z)] गों (4th(() (z) - p(=)
( log [= (z) p(z)|dz
+
T
10th (2) (s)- p(=)(dz
/ log [=()) (=)dz
(=)dz
+ / log +(2))
+ (flaf(()) -((())() -D)
(x)(()))
Employing the regularity conditions 28 and (2 and writing C(e) BBx)) B²:) it follows that
Eno such that Vn > no
11b) (0) 1(0) < (3(c)) + B!( ( (2) - p(=)ldz) (=)=)()=))dd)
+
sup AEe
(flah())) p(z)|dz)
Ifc
c(c)
with probability 1.
Lemma 1 states regularity conditions ensuring that the expectations in ))))))
(0) and ((0) are well- behaved in the limit. In particular, (28) and (29) imply uniform and absolute integrability of the log-
likelihoods under the respective probability measures induced by p and g!th) Since we are interested in the asymptotic behavior, it is sufficient for (9) to hold for large enough with probability 1.
Inequality (30) shows that we can make (1(h)(()) (0) I(0) small by reducing the li-distance between the true density p and the kernel density estimate g(a²) There exists already a vast body of liter- ature, discussing how to properly choose the kernel K and the bandwidth sequence (h,, so that (ha) (z) - p(2)ldz + O. We employ the results in Devroye (188) for our purposes, leading us to Proposition
16

---

If we fix (n) to be equal to an arbitrary sequence 2(m), then g% is fixed and we can treat Jn.r Under review as a conference paper at ICLR 2020
Proof of Proposition Let A denote the event that Eng ng inequality (30) holds for some constant C(+), From our regularity assumptions it follows that P((A)) 0. Given that A holds, we
just have to show that f [ (z) - p(z) " O. Then, the upper bound in (30) tends zero and we can conclude our proposition.
For any 6 let B,, denote the event
/ ( (=)-p(=))dds <
(31)
wherein 4% (2) is kernel density estimate obtained based on samples from p(2). Under the conditions in TT) we can apply Theorem 1 of Devroye (1983), obtaining an upper bound on the probability that (31) does not hold, i.e. Eu, mo such that IP ( e-nn for all me-
Since we need both and B, for n - oo to hold, we consider the intersection of the events (An B.). Using union bound argument it follows that 3kg such that Vn ko : P((An n B,))) S P(A9) + P(B)) hold. Hence, -x P((An (.)) < Σx= e um c < oo and by the Borel-Cantelli
0 e e e-un Note that we can simply choose kn- = max (von, mo for this
lemma we can conclude that
lim sup (()) - 1(0)
(32)
n 200 Ice
holds with probability 1.
Proof of Theorem
I The inequality in (10 implies that for any n > no.
hm sup lee
(00] n.r (0) 1(0) -
fim sup dee
Ri - 14th) (0)
+ sup oce
1h0 (0) 1(0)
(33)
Let no be fixed but arbitrary and denote
Jn,r
sup d
((h) (0)
r no
(34)
It is important to note that Jn,- is random variable that depends on the samples Z(n) = (Z .... Z,)
well as on the randomness inherent in Algorithm We define I(*) (4,,...I,) as the indices sampled uniformly from fI....,n} and E(r) (Eir).))))) as the sequence of perturbation vectors sampled from K. Let ())(")), P(I and P(E(r) be probability measures of the respective random sequences.
as the regular difference between sample estimate and expectation under a(A)) By the regularity
condition (29 the compactness of θ and the continuity of fe in 0, we can invoke the uniform law of large numbers to show that
lim Jn = lim sup (1) (0) = 100
(35)
r Aee.
with probability 1.
Now we want to show that 35
also holds with probability 1 for random training samples (()
First, we write Jn.r deterministic function of random variables:
Jn.r J(Z (m) {()))
(36)
This allows us to restate the result in 35 as follows:
Pin 48 > 0 3ro vr ro ((z = ((n),(()) 100,7 < <6)
- =/
48 0 3ro Ar ru ((Z(") (()),(()) E(r) < <6) ((),))))))))
aP(EW))PP(("))
(37)
=1
17

---

Under review as a conference paper at ICLR 2020
In that (A) denotes indicator function which returns 1 if A is true and else. Next consider the probability that the convergence in (35 holds for random (())
P (0 10) g00) g 46 > 0 Fro Vr
1 E(r) < 6)
48 1880030 Vr
1 - < 6) dP(E))
=
i dp(z(m) (
1 24 (v6>0 Wr J(Z 1o = < <6)
<) aP(E
=1
Note that we can d!(Z(") move outside of the inner integrals, since (()) is independent from ((r) and E Hence, we can conclude that (35) also holds, which we denote as event A, with probability for random training data.
From Proposition 1 we know, that
lim sup n=00 age
(0) - 1(0)
(38)
12
with probability 1. We denote the event that (38) holds as B. Since P(49)
P(BB) = 0, we can
union bound argument to show that P( B) 1. From 335) and 33 it follows that for any
n > Ylo.
him sup age
((h) (0) ((0)
sup #c
((h) (0) 1(0)
(39)
with probability 1. Finally, we combine this result with
38
obtaining that
lim Tim sup Bee
((h) (0) I(0) = 0
(40)
almost surely, which concludes the proof.
Proof of Theorem 2 The proof follows the argument used in Theorem 1 of White 1989). In the
following, assume that (13) holds. From Theorem [ we know that this is the case with probability 1. Respectively, we only consider realizations of our training data ((n) and noise samples ((r), =(r)
for which the convergence (13) holds (see proof of Theorem for details on this notation).
For such realization, let (o()) be minimizers of (h)) Also let (n;), and for any i, (Fis)s be increas-
ing sequences of positive integers. Define v. == 0(h) and (t, = mn...., (0). Due to the compactness of θ and the Aalzano-Weierstrass property thereof, there exists a limit point 0 E θ and increasing subsequences (ix)&, (fu)ı: so that Vi 1 as k cc.
From the triangle inequality, it follows that for any e > there exists ko so that Vk > kio
Mi (Ui ) 1(09) < Miieie (Dir I(Vig..ia) )
[((Vegain)) 1(09) 2e
(41)
given the convergence established in Theorem I and the continuity of in A. Next, the result above extended to
1(80) - 1(0) = (1(0P) - Mis..i (Via [pi (Wiese) - Min.is (0)] + juin (0) I(0) < 3e (42)
which again holds for à: large enough. This due to 41p, Miaii (Uig ) Min in (0) S since Vic is the minimizer of Minsin and Mixesi (0) (()) < by Theorem Because e can be made arbitrarily small, 1(8%) < I(0) as lo + co. Because E θ is arbitrary, 8 must be in e*. In turn, since (n.).. (ra.); and (ix)s: k: (jx)s were chosen arbitrarily, every limit point of sequence (Wi ) must be in O
In the final step, we proof the theorem by contradiction. Suppose that (4) does not hold. In this case, there must exist an > 0 and sequences (n:),, (ras); and (ix) (ie)s such that [[ )a -0 > for all & and E Ox. However, by the previous argument the limit point of the any sequence (Ui ) must be in O That is contradiction to (K 0 V k, O+. Since
18

---

Under review as a conference paper at ICLR 2020
the random sequences ((), , E where chosen from set with probability mass of 1, we conclude our proposition that
lim lim d(O ( n,r θ*) = 0
almost surely.
Discussion of Theorem Note that, similar 0", 0/.0) does not have to unique. In case there are multiple minimizers of itt). we can chose one of them arbitrarily and the proof of the theorem still holds. Theorem considers global optimizers over a set of parameters Ô, which may not be attainable in practical settings. However, the application of the theorem to the context of local optimization is straightforward when is chosen as a compact neighborhood of a local minimum 0r of (Holmstrom & Koistinen, 1992b). If we set O* = { and restrict minimization over ((k) the local region, then 0 converges O* as n, r
in the sense of Theorem
D CONDITIONAL DENSITY ESTIMATION MODELS
D.1 MIXTURE DENSITY NETWORK
Mixture Density Networks (MDNs) combine conventional neural networks with mixture density model for the purpose estimating conditional distributions p(y( (Bishop 1994). In particular, the parameters of the unconditional mixture distribution p(y) are outputted by the neural network, which takes the conditional variable æ as input.
For our purpose, employ Gaussian Mixture Model (GMM) with diagonal covariance matrices as density model. The conditional density estimate p(y|z) follows as weighted sum of K Gaussians
K
p(y|>)
Σ (x(x;))N (1))
(43)
wherein (x;x;;) 0) denote the weight, (1; 0) the mean and (x; 0) the variance of the k-th Gaus- sian component. All the GMM parameters governed by the neural network with parameters and input x.
(x;;
The mixing weights wy (z; 0) must resemble categorical distribution, i.c. it must hold that Σk_, wxx (x; = 1 and wx (x; 0) 2 Vk. satisfy the conditions, the softmax linearity used for the output neurons corresponding to wx(x;; 0). Similarly, the standard deviations σ (2) must be positive, which is ensured by a sofplus non-linearity. Since the component means μa (z;; 0) are not subject to such restrictions, output neurons.
linear output layer without non-linearity for the respective
For the experiments in 5.2 and 5.1 we set K 10 and use neural network with two hidden layers of size 32.
D.1.1 KERNEL MIXTURE NETWORK
While MDNs resemble a purely parametric conditional density model, a closely related approach, the Kernel Mixture Network (KMN), combines both non-parametric and parametric elements (Am- brogioni et al. 2017). Similar to MDNs, a mixture density model of p(y) is combined with a neural network which takes the conditional variable as input. However, the neural network only controls the weights of the mixture components while the component centers and scales are fixed w.r.t. z. For each of the kernel centers, M different scale/bandwidth parameters o. are chosen. As for MDNs, we employ Gaussians as mixture components, wherein the scale parameter directly coincides with the standard deviation.
Let K be the number of kernel centers be and M the number of different kernel scales am. The KMN conditional density estimate reads follows:
K
M
p(ylx) = Σ
Wkmm (x; ()N 0
(44)
k=
19

---

Under review as a conference paper at ICLR 2020
As previously, the weights Wk,m correspond to softmax function. The M scale parameters o. learned jointly with the neural network parameters 8. The centers Pa are initially chosen by k-means clustering the [vi) in the training data set. Overall, the KMN model is more restrictive than MDN as the locations and scales of the mixture components are fixed during inference and cannot be controlled by the neural network. However, due to the reduced flexibility of KMNs, they less prone to over-fit than MDNs.
For the experiments in 5.2 and 5.1 set K -50 and M =2 The respective neural network has two hidden layers of size 32.
D.2 NORMALIZING FLOW NETWORK
The Normalizing Flow Network (NFN) is similar the MDN and KMN in that a neural network takes the conditional variable its input and outputs parameters for the distribution over For the NFN, the distribution is given by Normalizing Flow (Rezende & Mohamed] 2013). It works by transforming simple base distribution and an accordingly distributed random variable Zo through a series of invertible, parametrized mappings f = SNN o ..... o ʃi into a successively more complex distribution p( (Zo)). The PDF of samples nN p(f(Zo)) can be evaluted using the change-of-
variable formula:
N
log det 0fn
log p(z)(=)>>> = p(z))
(45)
dzn-1
introduced in the context of pos- terior estimation in variational inference. They are optimized for fast sampling while the likelihood evaluation for externally provided data is comparatively slow. To make them useful for CDE, invert the direction of the flows, defining mapping from the transformed distribution p v(Zx) to the base distribution p(Zo) by setting f. ((z,) fi(z)..
The Normalizing Flows from Rezende & Mohamed (0155)
We experimented with three types of flows: planar flows, radial flows as parametrized by Trippe & Turner (2018) (011) and affine flows (z) exp(a)=++ We have found that one affine flow combined with multiple radial flows performs favourably in most settings.
For the experiments in 5.2 and 5.1 used standard Gaussian the base distribution that is transformed through one affine flow and ten radial flows. The respective neural network has two hidden layers of size 32.
E SIMULATED DENSITIES AND DATASETS
E. SKEWNORMAL
The data generating process (z, y) p(x, y) resembles a bivariate biittdistribbttinn wherein x E R follows normal distribution and y E ER a conditional skew-normal distribution (Andël et al. 1984). The parameters (E, w, a) of the skew normal distribution are functionally dependent on r. Speecfically, the functional dependencies are the following:
x N~N N (
p 1--00-1- 1
(46)
{(x) a a.bER
(47)
w(x) x² +
c,
(48)
a(x) Qlom 1te- (Onigh - aoww
(49)
y- SkewNormall (E()) a(x))
(50)
Accordingly, the conditional probability density p(y(x) corresponds to the skew normal density function:
2 y ( +b
{(x)
p(y]z)
N
[(())
(51)
w(x) w(x)
w(x)
In that, (-)-) denotes the density, and (-) the cumulative distribution function the standard nor- mal distribution. The shape parameter (I)) controls the skewness and kurtosis of the distribution.
20

---

Under review as a conference paper at ICLR 2020
x=0.00
0.50
0.6
x=0.00
x- 1,50
conditionl prob denstty
condtonnl probubbiity demalyy
125
0.-
01
0.-
0.2
(a) SkewNormal
(h) Gaussian
Figure 4: Conditional density simulation models. Conditional probability densities corresponding the different simulation models. The coloured graphs represent the probability densities p(y (z), conditioned on different values of xr
We Otow -4 and Ohiigh O, giving ply|æ)) a negative skewness that decreases as æ increases. This distribution will allow us to evaluate the performance of the density estimators in presence of skewness, a phenomenon that we often observe financial market variables. Figure 4a illustrates the conditional skew normal distribution.
E.2 GAUSSIAN MIXTURE
The joint distribution p(z, y) follows a Gaussian Mixture Model IR with 5 Gaussian components, R2 can be factorized, i.e.
ie. K 5. We assume that IR² and
K
p(z 3)
Σ
wk N(yluy,
Ez,k)
(52)
When x and y can be factorized as in form:
52
the conditional density p(y(x) can be derived in closed
K
p(y]=) Σ
We(w)
W Eyykk)
(53)
wherein the mixture weights are function of x::
Wx(x) WK (a Ex,k)
(54)
Σ/ Wi ((rlazj Ens)
For details and derivations we refer the interested reader to Guang Sung 2004 and Gilardi et al. (2002). The weights wk sampled from uniform distribution U(0, 1) and then normalized to to one. The component means are sampled from spherical Gaussian with zero mean and standard deviation of 1.5. The covariance matrices yy,k) R) and Σy,k) are sampled from Gaussian with mean 1 and standard deviation 0.5, and then projected onto the cone of positive definite matrices.
Since we can hardly visualize 4-dimensional GMM, Figure 4b depicts 2-dimensional equivalent, generated with the procedure explained above.
E.3 EURO STOXX 50 DATA
The Euro Stoxx 50 data comprises 3169 trading days, dated from Jamuary 2003 until June 2015. The goal is to predict the conditional probability density of I-day log-returns, conditioned 14 explanatory variables. These conditional variables comprise classical return factors from finance well option implied moments. For details, refer to Rothfuss et al. 2019). Overall. the target variable is one-dimensional, i.c. y E y C R, whereas the conditional variable constitutes a 14-dimensional vector, i.e. X C 11
21

---

Under review as a conference paper at ICLR 2020
E.4 NYC TAXI DATA
We follow the setup in Dutordoir et al. (2018). The dataset contains records of taxi trips in the Manhattan operated in January 2016. The objective predict spatial distributions of the drop- off location, based on the pick-up location, the day of the week, and the time of day. In that, the two temporal features are represented as sine and cosine with natural periods. Accordingly, the target variable is 2-dimensional (longitude and latitude of dropoff-location) whereas the conditional variable 6-dimensional. From the ca. million trips, we randomly sample 10,000 trips to serve as training data.
E.5 UCI
Boston Housing Concerns the value of houses the suburban area of Boston. Conditional variables mostly socio-economic well as geographical factors. For more details
Concrete The task is predict the compressive strength of concrete given variables describ- ing the conrete composition. For more details see dhabasesccncettccomppssve
Energy Concerns the energy efficiency of homes. The task is to predict the cooling load based on features describing the build of the respective house. For more details see
22


Reviews:
[{"review_id": "rJevBN3g9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the problem of parametric conditional density estimation, i.e. given a set of points {(x_n, y_n)} drawn from a distribution $p(x,y)$, the task is to estimate the conditional distribution p(x|y). The paper considers parametric estimation where in given a parametrized family of distributions f_{theta} we wish to minimize the likelihood of seeing the given data over theta. The parametric family in a lot of applications consists of highly expressive families like neural networks, which leads to the issue of overfitting in small data regimes. This has been tackled via regularization over the parameter space which might be hard to interpret as the associated inductive bias is not well understood and depends on the parametric family under consideration. On the other hand the paper proposes to add explicit noise in the examples used during training, i.e. irrespective of the optimization procedure (which could be mini-bath sgd) the paper proposes to draw examples from the data set, explicitly add noise onto the examples and create a proxy objective over the augmented data set. \n\nThe paper establishes two theoretical results. First is a simple taylor approximation based analysis to highlight the effect of the variance of noise. The conclusion is that higher variance penalizes the high curvature areas of the resultant density and hence this kind of noise addition could be seen as making resulting density smoother. The second contribution is to show that this procedure is asymptotically consistent, i.e. as n goes to infinity and the number of augmented data points go to infinity, the resulting density converges to the target density. This of course requires the noise variance to follow a decreasing schedule to 0. \n\nThe main merit of the idea is in its agnostic nature, as it can be applied to any parametric family and the experiements show that it seems to uniform improvement across models,  The basic idea proposed by the error, has existed in the space of deep learning based methods forever. This is the same idea behind image augmentation which forms a crucial part of training supervised models in vision. The authors claim that this idea is novel in the space of parametric density estimation, however I do not know enough about the area to verify the claim. It would surprise that this very natural idea has not been tried before. \n\nI have gone through the theoretical derivations in the paper and they look sound to me. However the results are all asymptotic in nature without establishing explicit rates which is a little bit of disappointment. Since I am not completely familiar in nature, but I guess such asymptotic consistency might be achievable using other forms of regularization under suitable assumptions. In that light, the theoretical contributions while being sound did not lend much intuition about why such a method might outperform others. Intuition does arise from the derivation for the effect of noise on the objective which helps understand the nature of the noise, but one wonders if similar intuitions could be derived for other forms of regularization as well. It would be great to see this derivation being extended to some concrete scenarios under well understood parametric families and seeing the effect explicitly. \n\nRegarding the experiments - The experiments definitely look promising as the improvments seem uniformly good across the cases considered. I am not an expert however in this setting so it is hard for me to judge the quality and significance of the benchmarks. The experiment methodology nevertheless looks sound. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Syg9trEf5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a regularization technique for conditional density estimation. The method is simple: adding noise to the data points, and training on the noisy data points. The paper also further gives an interpretation of the method, as a form of smoothing the curvature of the density function. It further proves the consistency of the method.\n\nPros:\n(i) The paper is well written.\n(ii) The method is simple to implement.\n(iii) The authors demonstrate a clear intuition of what the method does, i.e., as a form of smoothing.\n\nCons:\n(i)  The method itself is not novel. Adding noise to for regularization is a quite common technique used in many different applications.\n(ii) The experiments performed in the paper are all very small scaled. Not very convincing.\n(iii) Continuing the last point, it is unclear whether such technique can be scaled up. The method in another view is to replace the empirical distribution with a kernel density estimate. We know kernel methods don't scale. It's hence questionable whether the method can give any benefits in large scale dataset.\n(iv) If the use of noise regularization is to smooth the density function, how does it compare to methods that enforces Lipschitz continuity? e.g., gradient penalty, or construct a lipschtz constrained network such as in [1].\n\n[1] Sorting out lipschitz function approximation. Anil et al.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgzQj3b9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a noise regularization method which adds noise on both x and y for conditional density estimation problem (e.g., regression and classification). The writing is good and the whole paper is easy to follow. However, I vote for reject, since the novelty is somehow limited, the claims made in the paper is not well supported and experiments are not very convincing. \n\n1. Adding noise on x (e.g., [1]), y (e.g., [2]) is not new. Though it is claimed that this paper extends previous results on classification/regression to conditional density estimation which is a more general case. This claim is not well supported. Experiments are still evaluated in classification/regression tasks.\n\n2. Theorem 1 & 2 in Sec 4.2 only show the asymptotic case, which are quite obvious and seems helpless in understanding the advantage of adding noise regularization in conditional density estimation.\n\n3. Sec 4.1. The explanation that Page 5, ```\"The second term in (6) penalizes large negative second derivatives of the conditional log density estimate...\". It is hard for me to understand. Large positive second derivatives also lead to poor smoothness.\n\n[1] Learning with Marginalized Corrupted Features, ICML 2013\n[2] Learning with Noisy Labels, NIPS 2013", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has a mixture of weak reviews, the majority of which lean towards reject. All reviews mention a lack of novelty, and 2 of 3 a lack of support in experiments. While the authors argue, perhaps legitimately, for the novelty of the paper with respect to current literature, this is not convincing in the exposition. I recommend that the authors improve the justification for the novelty of their methodology, and strengthen the experiments to convince reviewers. As it stands, this paper is not quite ready for publication.
Decision: Reject