Variational Weighting for Kernel Density Ratios
Authors: Sangwoong Yoon
Frank C. Park
Gunsu S YUN
Iljung Kim
Yung-Kyun Noh
Venue: NeurIPS 2023 poster (2023)
Keywords: Kernel Density Estimation
KL-divergence
Density Ratio
TL;DR: Variational Weighting for Kernel Density Ratios

Abstract: Kernel density estimation (KDE) is integral to a range of generative and discriminative tasks in machine learning. Drawing upon tools from the multidimensional calculus of variations, we derive an optimal weight function that reduces bias in standard kernel density estimates for density ratios, leading to improved estimates of prediction posteriors and information-theoretic measures. In the process, we shed light on some fundamental aspects of density estimation, particularly from the perspective of algorithms that employ KDEs as their main building blocks.

Variational Weighting for Kernel Density Ratios
Sangwoong Yoon Korea Institute Advanced Study swyoonākias.. re.kr
Frank C. Park
Seoul National University Saige Research fcp@snu.ach kc.
Gunsu Yun POSTECH
Iljung Kim
Hanyang University
gunsu@postech.acc ac
Ha.jung811@@hanyee ac. kr
Yung-Kyun Noh
Hanyang University/ Korea Institute for Advanced Study nohyungShanyang. kc.kr
Abstract
Kernel density estimation (KDE) integral range of generative and discrimi- native tasks in machine learning. Drawing upon tools from the multidimensional calculus of variations, standard kernel density estimates for density ratios, leading to improved estimates of prediction posteriors and information-theowe shed light on some fundamental aspects of density estimation, particularly from the perspective of algorithms that employ KDEs as their main building blocks.
derive optimal weight function that reduces bias
measures. In the process, we
Introduction
One fundamental component for building many applications in machine learning is a correctly estimated density for prediction and estimation tasks, with examples ranging from classification
21 anomaly detection [3], and clustering to the generalization of value functions [5]. policy
evaluation (6), and estimation of various information-theoreti measures 7 9 Nonparametric density estimators, such as the nearest neighbor density estimator or kernel density estimators
(KDEs), have been used substitutes for the probability density component within the equation of the posterior probability, or the density-ratio equation, with theoretical guarantees derived in part from the properties of the density estimators used [10 [T].
Given specific task which uses the ratio between two densitics, P1 (x) and P2(x) at point x E IR., consider the ratio handled by the ratio of their corresponding two KDEs, P1 (x) and (x)):
P (x) (x
(x)
(I)
Estimate
P2 (x)
Each estimator is a KDE which counts the effective number of data within small neighborhood of x by averaging the kernels. The biases produced by the KDEs the nominator and denominator
12 Theorem 6.28] are combined to produce single bias of the ratio, as demonstrated in Fig. [
For example, the ratio (x))
at X in Fig. a) is clearly expected be underestimated because of
[x(x]
the dual effects Fig. Ib::: the underestimation of the nominator P1 (xo) and the overestimation
of the denominator Pa (Xn). The underestimation is attributed the concavity of p1 (x) around
Xy which leads to a reduced number data being generated compared to a uniform density. The
underestimation of (x( can be explained similarly. The second derivative- Laplacian that creates
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Pr (x0)
Use weight function data a(x).
Pi (Xo)
(x()
(x0)
P2(x)
(x(
P2 (2(x)
D2 (xx)
Xn
p2 (x0)
P1 (xo)
P1 (xo)
P2 (x0)
P2 (Xo)
XD
(a)
(b)
(c)
Figure 1: Estimation of the density ratio (x(Xa) px(xa) and bias correction using the weight function (xx).
(a) Two density functions, P1 (x) and (x(x). and the point of interest Xy for ratio estimation. Two regions delineated by dashed lines are magnified in (b). (b) The concavity and convexity of the density functions around X. and their KDEs. Concave density P1 (x) generates less data than the uniform density of p1 (Xo) around Xn resulting in an underestimation. For a similar reason, convex density P2 (x) results in an overestimation. The two biases are combined into an underestimation of the ratio. (c) KDE augmented with a nonsymmetrie weight function (xx) can alleviate this bias by
transforming the bias of i2 (x) to an appropriate underestimation from an overestimation.
the concavity or convexity of the underlying density is a dominant factor that causes the bias in this example.
The Laplacian of density has been used to produce equations in various bias reduction methods, such
as bias correction [3 14 [S] and smoothing of the data space 16. 17.. However, the example in Fig. [ motivates a novel, position-dependent weight function (xx) ) to be multiplied with kernels in order to alleviate the bias. For example, to alleviate the overestimation of p2 (x(x)). we can consider the (xx) shown in Fig. [(c) that assigns more weight on the kernels associated with data located to the left of Xo. which a low-density region. When the weighted kernels are averaged, the overestimation of (x(x)) can be mitigated or potentially even underestimated. Meanwhile, the bias of F1 (xo) remains unchanged after applying the weights since p1 (x) is symmetric around Xo. This allows the reversed underestimation of P2(x)) from the initial overestimation to effectively offset or counterbalance the underestimation of p1 (xx) within the ratio.
We derive the (xx function that performs this alleviation over the entire data space. The appropriate information for (x) comes from the geometry of the underlying densities. The aforementioned principle of bias correction leads to novel, model-based and model-free approaches. Based on the assumption of the underlying densities, we learn the parameters for the densities' first and second derivatives and then variationally adjust (xx for the estimator to create the variationally weighted KDE (VWKDE). We note that the model for those densities and their derivatives need not be exact because the goal is not to achieve precise density estimation but rather to accurately capture the bell-behaved (xx) for the KDE ratios.
Applications include classification with posterior information and informatioo-thhoreeti measure estimates using density-ratio estimation. Calibration of posteriors 18] has been of interest to many researchers, in part, to provide ratio of correctness of the prediction. Plug-in estimators of information-theore theoretic measures, such as the -eibbeck-Leiblrr (K-L) divergence, can also be advantageous. For K-L divergence estimation, similar previous formulations for the variational approach have included optimizing a functional bound with respect to the function constrained within the reproducing kernel Hilber space (RKHS) [19 20 21). These and other methods that use weighted kernels (c.g.. [22 23 24) take advantage of the ficxibility offered by universal approximator functions in the form of linear combinations of kernels. These methods, however, do not adequately explain why the weight optimization leads to an improved performance. Based on a derivation of how bias is produced, we provide an explicit modification of weight for standard kernel density estimates, with details of how the estimation is improved.
The remainder of the paper is organized as follows. In Section 2. we introduce the variational formulation for the posterior estimator and explain how to minimize the bias. Section shows how a weight function can be derived using the calculus of variations, which is then extended to general density-ratio and K-I. divergence estimation Section 4. Experimental results are presented in Section 5. Finally, we conclude with discussion in Section 6.

---

Variationally Weighted KDE for Ratio Estimation
KDE p(x) = N (x, is conventionally the average of kernels. The average roughly
represents the count of data within a small region around x, the size of which is determined by bandwidth parameter h. The amount of convexity and concavity inside the region determines the bias of estimation, as depicted in Fig. (b).
2.1 Plug-in estimator of posterior with weight
We consider weighted KDE as plug-in component adjusted for reliable ratio estimation using a positive and twice differentiable weight function: a(x) E A with A = {a : RD R a e Ca (). For two given sets i.id. samples, D1 {x)= ~ (x) for class = and NN D2 {x: (x+1 px) for class 2, we use the following weighted KDE formulation:
N N1 Σ (xxx)) (x, x2), (x,,,,,
N1-N2
1
(x)
(2(x)
Σ
xx,),)
N2
N1+
Here, the two estimators use a single (x).. The kernel function (x, x') is a positive, symmetric, normalized, isotropic, and translation invariant function with bandwidth h. The weight function (xx) informs the region that should be emphasized, and a constant function (xx) = reproduces the ratios from the conventional KDE. We let their plug-in posterior estimator be f fx). and the function can be calculated using
(x)
f(x) = P(y 1|x)
(3)
(x) + (x)
with constant R determined by the class-priors.
2.2 Bias of the posterior estimator
We interested in reducing the expectation of the bias square:
E[Bias(x)) (x)* / (f(x) Eo, p(x)dx.
(4)
The problem of finding the optimal weight function can be reformulated the following equation in Proposition
With small h, the expectation of the bias square in Eq. that eliminates the following function
Proposition 1.
is minimized by any axx)
Beeps xp (x) (Vlog h(x) + gxx),
(5)
at every point x. Here, h(x) = Vp Laplacian operators, and 2 respectively. All derivatives are with respect to x.
Vp2 and g(x) = 3
(2'm
42p2
with gradient and
The derivation of Eq. begins with the expectation the weighted KDE:
Ep, [x) Ex (xx<px(x)/a(x (x(x' (x, = /
(x,
(6)
2 (x))) + (la(())p (x)) + +(13))
(7)
2
Along with the similar expansion for ED, (x)), the following plug-in posterior can be perturbed with small h:
ED, [i (x)]
ED, D. [f(x)]
(8)
ED, (x)) + +,ED, [x))
h² (x) (x)
(x(x())p(xx))
Vl((())2(xx]
f(x)
O(h3)
(xi(x) + (x))
(xx))x((x)
a(x)p2(x)
h²
f(x)
Ply P Bl (x) O(h)),
(9)
3

---

Poste
622
02
811
510
03
371
10 31 110
57
(b)
(c)
Figure 2: Posterior estimates with KDEs and VEKDEs for two 20-dimensional homoscedastic Gaussian densities. (a) Bias corrected by VWKDE. (b) Bias and variance of posterior estimates depending on the bandwidth for standard KDE and for (c) VWKDE.
with the substitution
(((())p(xx)) V (xxx(x)22(x) a(x)p1185
Bagpi p2 (x)
(10)
a(x)p2(x)
The point-wise leading-order bias can be written as
h²2
Bias (x) - = 21x) B Boaps (x).
(11)
Here, the Ba pa (x) includes the second derivative of a(x)pı (x) and (xx)p2(x). Because two classes share the weight function a(x), Eq. (10 can be simplified into two terms without the second derivative of (xx) as
VTal=
Vpilx
Vpal (2(x)
1
V p1
V'pals (x(x)
B Bag p2 (x)
(12)
axx)
Pi (xx
(x)
which leads to Eq. Appendix A.
5 in the Proposition.
The detailed derivation this section can be found in
2.3 Plug-in estimator of K-L divergence with weight
In order to estimate KT(pillps)) = Ex [log (x) p2(x)]
we consider the following plug-in estimator:
N₂
P1 (x,)
KL(pillp2) = NI
log
(13)
P2 (2(x))
using x; in X; D2 for Monte Carlo averaging. When calculate Pi at x,, exclude x; from the KDE samples. We use pi (x,) x Σド= (x,, with the indicator function I(z). which is 1 I is and 0 otherwise.
Proposition 2. With small h, the expectation of the bias square is minimized by finding any a(x) that eliminates the same function as Basp (x) in Eq. (5) in Proposition [ at each point x.
In the task of estimating the K-I. divergence, Proposition ellaim that obtain the equivalent
Basei p2 (x) to Eq. (5) during the derivation of bias. The pointwise bias in the K-I. divergence
estimator be written
1i2 Px(x)
Bias(x)
Busph.pa (x),
(14)
p2(x)
with Bappisn (x) equivalent to Eq. 5
Variational Formulation and Implementation
Now consider the a(x) that minimizes the mean square error for the estimation:
min a(x)aA / ((Vlogak)) l((x) + g(x)) 2
rxx)dx,
(15)

---

with
h(x) ((Er P1
Fp1
Vpa D2
g(x)
=
F' P1
F2p2)
The r(x) function depends on the problem:
P2
r(x) = 1()=1
p(x) for posterior estimation and r(x)
p2(x] p(x) for K-L.
(x(x)
divergence estimation, with the total density, p(x) = (pı (x) + YP2 (x)) (yyyy111 1). The calculus of
variation for optimizing the functional in Eq. (15) provides an equation that the optimal a(xx) should satisfy:
[r((V log a) h + 9)h] 0.
(16)
The detailed derivation of this equation can be found in Appendix B.
3.1 Gaussian density and closed-form solution for axx)
A simple analytic solution for this optimal condition can be obtained for two homoscedastic Gaussian density functions. The density functions have two different means, μ4l E R.' and μ22 E RR,, but share single covariance matrix Σ E IRDx
Pi xx = (Nxx p1, E); p2(x) = Nxx M2,
12, Σ).
(17)
One solution for this homoscedastic setting can be obtained as
a<(x) =cx
cxp ( (x (x- n¹) A(x - (1)
(18)
with p' t22 and A b (I- (I- Σー --11-p11-2000 ( - - using an arbitrary constant b. 11 -2)
Σ-1
Due to the choice of b, the solution is not unique. All the solutions produce zero bias. Its detailed derivation can be found in the Appendix C. The reduction of the bias using Eq. (18) with estimated
parameters is shown in Fig. 2
3.2
Implementation
In this work, propose model-free method and mode-based method. The model-free approach uses the information of V log (x) and V 12gp2(x) (2(x) estimated by a score matching neural network 225. We obtain the second derivative, ? Log p. by the automatic differentiation of the neural network for the scores. We then obtain
V using
V2p = 6 logp
logpp logp. With the outputs of
P
the neural networks for V log and V we train a new network for the function (xx; 0) with neural network parameters 0.
On the other hand, the model- approach uses coarse Gaussian model for class-conditional densities. The Gaussian functions for each class have their estimated parameters 74, 72g E RDD and
E, Eg E RR Dxx D) We use the score information from these parameters: V log (xx = Σj (x P1) and Vogg function within the RKHS with basis kernels Ka (:,-) with kernel parameter a. We let log ax; 0)
1x 722). In the model-based approach,
let the log of a(x) be
EN+,, (x, x,) with parameters 0 (0....., ONI+Na) } for optimization.
The weight function (xx) is obtained by optimizing the following objective function with N1 number of data from (x) and N2 number of data from p2(x):
N1 N1+N2 Σ
L(
( T log (xga ()gxxxx))((xx)))
2 +
V log (x;; 0))
(19)
with the substitutions h(x) log A (x) and ğ(x) =
(2pi(x) (x)
pxp
p2(x)
In the model-based method, an addition of -egullarr,,, A ΣE No 02, with a small positive constant À makes the optimization (9) quadratic. When there are fewer than 3,000 samples, use
all of them basis points. Otherwise, randomly sample 3,000 points from {x, /=1
Ni+
A brief summary of the implementation process is shown in Algorithms and
!Code is available at https://giitub..
estimattioo
5

---

Algorithm Model-free
Algorithm 2 Model-based
Input: x, {x, PI, |xs NNN
Input: x. {x ~p1. {x+) +N2
i P2
: P2
Output:
Ratio R(xx (=
Output: Ratio R(x) (= (x))
Procedure:
Procedure:
Estimate ui,, Eg using {xi [i] Ppp..
1. Estimate V lo using (x M ~ p-p
2. Estimate fi2, Dg using {x, [xi]i=N,+
Ni+ ~22.
2. Estimate log P2 using (x, Jii ~22 No
Use 4 log Pelx Σ- (x - Pa) to obtain
3. Obtain axx) that minimizes Eq. (19
a(x) that minimizes Eq. 19
DM (xx)))
3. R(x) ENENE (x.x.)
81 axx, (x.x.)
R(x) En M (xxxx
Return R(x)
Return R(x)
4
Interpretation of (xx) for Bias Reduction
The process of finding a(x) that minimizes the square of Ba (x) in Eq. (5) can be understood from various perspectives through reformulation.
4.1 Cancellation of the bias
The second term 1/2 V2 yPp2 in Eq. 5 repeatedly appears in the bias of nonparametric processes using discrete labels [26]. In our derivation, the term is achieved with a constant a(x) with no weight function. The role of the weight function is control the first term V based on the gradient information in order to let the first term cancel the second.
or
Yei Vpa P1
4.2 Cancellation of flow in mechanical system
The equation for each class can be compared with the mechanical convection- diffussion equation, 0t - Vu D' which is known as the equation for Brownian motion under gravity (27 or the advective diffusion equation of the incompressible fluid (28). In the equation, is the mass of the fluid, is the time, v is the direction of convection, and D' is the diffusion constant. The amount of change the sum of the convective movement of mass along the direction opposite Vu and the diffusion from the neighborhood. We reorganize Eq. (5) into the following equation establishing the difference between the convection-diifusiion equations of fluids:
BeepI.2. =
V log a + logp1
V llggp
+
2 log P1
2
2
1 logg2 log p2 + 2 log P2
log a +
(20)
According to the equation, we can consider the two different fluid mass functions, M (x) log (x) and u2 log p2 (x), and the original convection movement along the directions v = / log P1 and v/2 = - v log P2. If we make an axx) that modifies the convection directions v and v to
V1 = v Va and V2 = v22 Va, and mass change in one fluid is compensated by the change of the other, in other words, if dun
thg Bt of leading term of the bias for ratio estimation.
then the (x) is the weight function that minimizes the
4.3 Prototype modification in reproducing kernel Hilbert space (RKHS)
A positive definite kernel function has its associated RKHS. A classification using the ratio of KDEs corresponds prototype classification RKHS that determines which of the two classes has closer mean than the other (29 Section 1.2]. The application of a weight function corresponds to finding a different prototype from the mean 30 The relationship between the new-found prototype
and the KDEs has been previously discussed 31

---

0.8
True LPDR LPDR VWXDE
KDE LPDR Blas'
VWKDE LPOR Bias
KDF Var
0.6
log
VWKDE LPDR Var
5.0
10 Ham
0.4
2.5
0.0
0.1
0.2
KDE KLtP:lles)
True KiPpiPP
VWKDE Kl [in]]
0.0 0.4
0.8
0.0
0.5
0.6
Figure 3: Estimation results of the LPDR log(p1 p2) and the K-L divergence. (a) Estimation of LPDR at each point. The estimation bias from the true LPDR is reduced by using VWKDE. (b) Squared bias and variance of the estimation with respect to the bandwidth h. Bias has been significantly reduced without increasing variance. (c) Mean and standard deviation of K-L divergence estimates with respect to the bandwidth h.
Isotropic
20D Isotrop
50D Isetropic
1.5
3.5
-4-
NNG
WhGardb
0.0
500
1000
1500
2000
2500
3000
500 1000 1500
2005
2500 3000
0.0
500
1009
1500
2000 2500 3000
of 22 moles per class
2 of es pet dass
of samples perclss
2.0 100 Non-isotrapic Heteroscedastic
20D Non-isotrapic Heetersccedastic
50D Varying Mean Diff
2.0
RL AND 20
CDE
10
WWKDE-MF
1900
2000
3000
4000
1000
2000
3000
40000
0.0
0.5
3.0
3.5
2.0
samples
Distance between
Figure 4: K-L divergence estimation results for synthetic distributions; (NN) Nearest-neighbor estimator; (NNG) NN estimator with metric learning (32]: (KLIEP) Direct importance estimation
21: (NNGarcia) Risk-based f-divergence estimator [33 (NNWang) Bias-reduced NN estimator
34 (MINE) Mutual Information Neural Estimation [35]; (Ensemble) Weighted ensemble KDE
estimator [36]: (vonMises) KDE estimator with von Mises expansion bias correction [37]: (KDE)
KDE estimator; (VWKDE-MB, VWKDE-MF) Model-based and model-free approach of the proposed
estimator in this paper.
Experiments
5.1 Estimation of log probability density ratio and K-L divergence 1D
We first demonstrate in Fig. how the use of VWKDE alters log probability density ratio (LPDR) and K-L divergence toward a better estimation. We use two 1-dimensional Gaussians, pi (x) and P2(x), with means 0 and 1 and variances 1.1 and 0.92, respectively. We draw 1,000 samples from each density and construct KDEs and model-based VWKDEs for both LPDR and K-L divergence. For LPDR evaluation, we draw a separate 1,000 samples from each density, and the average square of biases and the variances at those points are calculated and presented in Fig. 3bb). The K-L divergence estimation result shown in Fig. 3j),, where the true K-L. divergence can calculated analytically as KJ (p: (lp2) 2 0.664, and the estimated values are compared with this true K-L. divergence.
KDE-based LPDR estimation exhibits severe bias, but this is effectively reduced by using VWKDE as an alternative plug-in. Although VWKDE slightly increases the variance of estimation, the reduction of bias is substantial in comparison. Note that since the bias is small over wide range of h, VWKDE yields a K-L divergence estimate which is relatively insensitive to the choice of h.
5.2 Synthetic distributions
We perform the VWKDE-bbased K-L. divergence estimator along with other state-of-the-an estimators to estimate the K-I. divergence K Tpp |P2) between two synthetic Gaussian distributions P1 and p2 having μn and 42 as their mean vectors and Σn and Eg their covariance matrices, respectively.
7

---

2D Non-Gaussia
Non-Gaussian
10
MINE
10
Ensembre
102,
1p
KDE
vonNises
10-2
WIKDE-FF
1000
4000
2000
2000
4000
of samples dasss
of samples per clss
Figure 5: Estimation of K-L divergence between two non- Gaussin densities, P1 (x) and p2(xx. Each density is the Gaussian mixture of the three Gaussians, as shown in the 2-dimensional density contour in the figure on the left. They the true densities but are very dissimilar the single Gaussian model. The figure in the middle shows the estimation with 2-dimensional data, and the figure on the right shows the estimation with 20-dimensional data. With 20-dimensional data, the remaining 18 dimensionalities have the same mean isotropic Gaussians without correlation to the first two dimensionalities.
Unseen Defect
VWKDE
NN-1
CNN
15
1.0
30
0.8
10
0.6
20
0.4
5
10
0.2
0.0
Figure 6: Detection of an artificially injected defect (MNIST digit "3"). The first panel shows the image with an injected defect. The remaining three panels show the detection scores of different methods.
We use three different settings for P1 and p2: Isotropic (Iso), Non-isotropic Heteroscedastic (NH), and Varying Mean Diff (VMD). In Iso, Σ = Eg for an identity matrix I. μ4 = 0 for a zero vector 0. The first element of μ2 is v2. while the other elements are uniformly zero, resulting in LL lP2)) 1. In NH, μ1 #l = O, and E I. Mi is matrix having a pair of off-diagonal element (22)1.2 = (,2) = 0.1, and other off-diagonal elements are zero. The diagonal elements have a constant value w. which determined to yield K L(pi IlP2) Rs 1.0 with w = 0.750 (10D) and K L(pi Rv 0.5 with 0.863 (20D). In VMD, the first element of μ22 has various values between 0 and 2, while μl and the other elements jlg remain zero. Σ₁ = Eg = T. In Iso and NH, we vary the sample size, and in VMD, we use 2,000 samples per distribution. We repeat each experiment 30 times and display the mean and the standard deviation in Figure 4
In the upper left panel of Figure we observe that almost all algorithms estimate the wwwdimmnsional K-L divergence reliably, but the results deteriorate dramatically as shown in the upper middle and right panels with -ighndimensionality.. As shown in the lower left and lower middle panels, most of the baseline methods fail to produce the estimates near the true value when data are correlated. The model-based VWKDE-based estimator is the only estimator that recovers the true value in the 20D NH case. Figure shows the K-L. divergence estimation for non-Gaussian densities. In this example, the model for the score function in -oaeel-based VWKDE is different from the data-generating density, in which the estimator still shows very reliable estimates.
5.3 Unsupervised optical surface inspection
We apply the proposed K-L divergence estimation using VWKDE for the inspection of the surface integrity based on the optical images. Most of the previous works have formulated the inspection using the supervised setting [38] 39 40, 41] however, often the defect patterns are diverse, and training data do not include possible defect patterns. In real applications, identification and localization of unseen defect patterns are important. In this example, we apply the model-based VWKDE.
Detection of defective surface Following the representations of previous works on image classifi- we extract random small patches from each image and assume that those patches are the independently generated data. We use the probability density pr(x), for the patch E RD from I. Given the N number of normal surface images D = [I:)-1 and a query image I", we determine whether I" is defective surface according to the following decision function f(')) and predefined
cation 42 43
8

---

Table 1: Performances for defect surface detection (left) and defect localization (right), mAUC and mAP are averaged over six surface types DAGM. The DAGM dataset provided with labels, and only CNN used the labels for training. The unseen defect is the artificially injected MNIST digit "3."
MAUC
DAGM Defect
Unseen Defect
mAP
DAGM Defect
Unseen Defect
VWKDE
0.785 + 0.002
0.967 + 0.003
VWKDE
0.369 + 0.005
0.903 : 0.007
KDE
0.734 + 0.005
0.926 + 0.003
KDE
0.294 + 0.004
0.849 + 0.006
NN-1
0.628 + 0.002
0.813 + 0.001
NN-1
0.095 + 0.008
0.488 + 0.002
NN-10
0,540 + 0.003
0.614 = 0.002
NN-10
0.081 + 0.004
0.254 = 0.002
NNWang
0.605 1 0.002
0.657 1 0.004
NNWang
0.029 1 0.005
0.024 1 0.000
MMD
0.618 1 0.003
0.615 + 0.008
MMD
0.151 1: 0.006
0.032 + 0.001
OSVM
0.579 + 0.001
0.538 + 0.000
OSVM
0.249 1 0.012
0.444 + 0.009
CNN
0.901 = 0.011
0.809 = 0.029
CNN
0.699 + 0.037
0.564 + 0.060
threshold:
S(T) = min KT(pr. lIPI.)-
(21)
LED
Defect localization Once the defective surface is detected, the spot of the defect can be localized by inspecting the LPDR log(pr- (x)/PI.. (x),.. (x)) score between the query image I' and the I, with I,- arg gD Lpp- I!PL.). The location of the patch x with the large LPDR score is considered to be the defect location. Note that a similar approach has been used for the witness function in statistical model criticism 45
For the evaluation of the algorithm, we use a publicly available dataset for surface inspection: DAGM The dataset contains six distinct types of normal and defective surfaces. The defective samples are not used in training, but they are used in searching the decision thresholds. We extract 900 patches per image, and each patch is transformed into four-dimensional feature vector. Then, the detection is performed and compared with many well-known criteria: diverse K-L divergences estimators as well as the maximum mean discrepancy (MMD) [46 and the one-class support vector machines (OSVM) (47.. In addition, the Convolutional Neural Networks (CNNs) training result is presented for comparison with supervised method.
In DAGM, the testing data have defect patterns similar to those in the training data. To demonstrate unseen defect patterns, we artificially generate defective images by superimposing a randomly selected 15% of the normal testing images with a small image of the MNIST digit "3' at a random location (see Figure (). Table [ presents the area under curve (AUC) of the receiver operating characteristic curve for the detection as well as the mean average precision (mAP) for the localization.
CNNs which use labels for training show good performances only in detecting and localizing DAGM defects. The K-L divergence estimation with VWKDE show the best performance over many unsupervised methods, and it provides significantly better performances both at identifying unseen defects and at localizing them. Figure 6 shows one example of how well the proposed method localizes the position of the unseen defects.
6 Conclusion
In this paper, we have shown how a weighted kernel formulation for the plug-in densities could be optimized to mitigate the bias in consideration of the geometry of densities. The underlying mechanism uses the information from the first derivatives to alleviate the bias due to the second derivatives.
In our experiments, a simple choice of Gaussian density model for obtaining the first and second derivatives led to reliable reduction of bias. This insensitivity to the exactness due a coarse model is nonintuitive considering the traditional dilemma prevalent in many conventional methods; a coarse and inexact model enjoys small variance but at the cost of large bias. In our work, the usage the coarse model had no effect on the flexibility of the plug-in estimator, while the high dimensional bias was tackled precisely.
DDeutsche Arbeitsgemeinschaft für Mustererkennung (The German Association for Pattern Recognition).

---

Limitations of this study include the computational overhead for score learning using parametric or neural network methods and no benefit for the asymptotic convergence rate because it depends on the convergence rate of KDE. Using a non-flexible parametric model rather than flexible one provides consistent benefit to improve the KDE.
Acknowledgments and Disclosure of Funding
SY was supported by KIAS Individual Grant (AP095701) via the Center for Al and Natural Sciences at Korea Institute for Advanced Study. SY and FCP were supported in part by IITP-MSIT (2021-0- 02068, 2022-0-00480), ATC+ (20008547), SRRC NRF (RS-2023-0020880))
and SNU Institute for Engineering Research. GY was partly supported by IITP-MSIT (2019-0-01906), and IK and YKN was supported by NRE/MSIT (No. 201
202 222IMEE52A00119555 ITPM/SIT (ITP-2201-0--02068, 2020-0-01373, RS-2023--00220628) YKN was supported by Samsung Research Funding & Incubation Center for Future Technology (SRFC-IT1901-13) partly in the derivation of the weight dependency on the bias and its mechanical interpretation.
References
[1] Jaewoong Cho, Gyeongjo Hwang, and Changho Suh. A fair classifier using kernel density estimation, In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15088-15099. Curran Associates, Inc., 2020.
[2] Edward Gan and Peter Bailis. Scalable kemel density classification via threshold bassd pruning. In SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data, pages 945-959, 2017.
Longin Jan Latecki, Aleksandar Lazarevic, and Dragoljub Pokrajac. Outlier detcetion with kernel density functions. In Petra Perner, editor, Machine Learning and Data Mining in Pattern Recognition, pages 61 61-75, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg.
[4] Dirceu Scaldelai, Luiz Carlos Matioli, Solange Regina dos Santos, and Mariana Kleina. MulticlusterKDE: new al algorithm for clustering based on multivariate kernel density estimation. Journal of Applied Statistics, 901-119111 2020.
[5] Dirk Ormoneit and Šaunak Sen. Kernel-based reinforcement learning. Machine Learning, 49:161-178, 2002.
[6] Haanvid Loe, Jongmin Lec, Yunscon Choi, Wonscok Jeon, Byung-Jun Lee, Yung-Kyun Noh, and Kee-Eung Kim. Local metric learning for off-policy evaluation in contextual bandits with continuous actions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Procexsing Systems, volume 35, pages 3913-3925. Curran Associates, Inc., 2022.
[7] Akshay Krishnamurthy, Kirthevasan Kandasamy, Barnabas Poczos, and Larry Wasserman. Nonparametric estimation of renyi divergence and friends. Eric P. Xing and Tony Jebara. editors, Proceedings of the 31st International Conference on Machine Learning. volume 32 of Proceedings of Machine Learning Research, pages 997, Bejing, China, 22-24 Jun 2014, PMLR.
[8] Dongxin Xu and Deniz Erdogmuns. Renyi's entropy, divergence and their nonparametric estimators. In: Information Theoretic Learning. Information Science and Statistics, 2010.
[9] Jose C. Principe, Dongxin Xu, Qun Zhao, and John W. Fisher. Learning from examples with information thcoretic criteria. Journal of VLSI Signal Processing Systems, 26(1-2):75-1 1135, 13, 2000.
[10] Kumar Sricharan, Raviv Raich, and Alfred O. Hero III. Estimation of nonlinear functionals of densities with confidence. IEEE Transactions on Information Theory, 58(7):4135 41554 2012.
[11] Ziv Goldfeld, Kristjan H. Greenewald, Jonathan Niles Weed, and Yury Polyanskiy. Convergence of smoothed empirical measures with applications to entropy estimation. IEEE Transactions on Information Theory, 66(7):4368 4391, 2020.
[12] Larry Wasserman. All of nonparametric statistics: a concise course in nonparametric statistical startistt inference. Springer, 2005.
[13] Peter Hall. Effect of bias estimation on coverage accuracy of bootstrap confidence intervals for probability density. The Annals of Statistics, 20(2):675 694, 1992.
10

---

[14] Peter Hall and Byeong U. Park. New methods for bias correction at endpoints and boundaries, The Annals of Statisticx, 30(5):1460 1479, 2002.
[15] M. C. Jones, David F. Signorini, and Nils Lid Hjort. On multiplicative bias correction in kernel density estimation. Sankhyā: The Indian Journal of Statistics, Series A (1961-2002). (1(3):422-430 1999.
[16] David Ruppert and Daren H. Cline. Bias reduction kernel density estimation by smoothed empirical transformations. The Annals of Statistics,
1994.
[17] Yung-Kyun Noh, Masashi Sugiyama, Kee-Eung Kim, Frank Park, and Daniel D Lee. Generative local metric learning for kernel regression. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 2452-2462. Curran Associates, Inc.. 2017.
[18] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh, editors. Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR, 06 11 Aug 2017.
[19] Aditya Menon and Cheng Soon Ong. Linking losses for density ratio and class-probability estimation. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 304-313, New Jun 2016. PMLR.
York, New York, USA, 20 22
[20] XuanLong Nguyen, Martin J Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 1089-1096. Curran Associates, Inc., 2008
[21] Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60(4):699-746) 2008
[22] Vladimir N Vapnik and Sayan Mukherjee. Support vector method for multivariate density estimation. In Advances in Neural Information Processing Systems, pages 659-665, 2000.
[23] Mark Girolami and Chao He. Probability density estimation from optimally condensed data samples. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(10):1253 1264, 2003.
[24] Le Song, Xinhua Zhang, Alex Smola, Arthur Gretton, and Berharrd Schölkopt. Tailoring density estimation via reproducing kemel moment matching. In Proceedings of the 25th international conference on Machine learning, pages 992 999. ACM, 2008
[25] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, PMLR, volume 115, pages 574-584, 2020.
[26] Kung-Kyun Noh, Byoung-Tae Tak. Zhang, and Daniel D. Lee. Generative local metric learning for nearest neighbor classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(1):106- 18, 2018.
[27] S. Chandrasekhar. Stochastic problems in physics and astronomy. Reviews of Modern Physiscs, 15:1-89, 1943.
[28] Scott A. Socolofsky and Gerhard H. Jirka. Environmental fluid mechanics. Part Mass transfer and diffusion. Engineering-Lectin lectures [online]. 2004. Karlsruhe Inst. für Hydromechanik. 2nd ed. 2002.
[29] Bernhard.
Schölkopf and Alex J. Smola. Learning with kernels: support vector machines, regularization. optimization, and beyond. Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, USA, December 2002.
[30] Krikamol Muandet and Bernhard Schölkopf. A unifying view of support measure machines, support vector machines, and parzen window classificrs. Unpublished, 2020.
[31] Thomas Hofmann. Bernhard Schölkopf, and Alexander J. Smola. Kernel methods in machine learning. The Annals of Statistics, 36(3):1111-1220) 2008
[32] Yung-Ke Kyun Noh, Masashi Sugiyama. Song Liu, Marthinus C. du Plessis, Frank C. Park, and Daniel D. Lee. Bias reduction and metric learning for nearest-neighbo estimation of Kullback-Ledd Leibler divergence, Neural computation, page 1, 2018

---

[33] Dario Garcia-Garcia, Ulrike von Luxburg. and Raúl Santos-Rodriuus Risk-based generalizations of I-divergences, In Proceedings of the 28th International Conference on Machine Learning, ICML, pages 417-424, 2011.
[34] Qing Wang, Sanjeev R Kulkamni, and Sergio Verdú. Divergence estimation for multidimensional densities via n--nabrst-neighhoor distances. IEEE Transactions on Information Theory, 55(5):2392-2445 2009.
[35] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Devon Hjelm, and Aaron Courville. Mutual information neural estimation. International Conference on Machine Learning, pages 530 539, 2018.
[36] Kevin Moon, Kumar Sricharan, Kristjan Greenewald, and Alfred Hero. Ensemble estimation of information divergence. Entrupy, 2068):560, 2018.
[37] Kirthevasan Kandasamy, Akshay Krishnamurthy, Barnabas Poczos, Larry Wasserman, and James M Robins. Influence functiinn for machine learning: Nonparametric estimators for entropies, divergences and mutual informations. arXiv preprint arXiv:1411 1344,434, 2014.
[38] Franz Pernkoff and Paul O'Leary. Visual inspection of machined metallic high-precision surfaces. EURASIP Journal on Advances in Signal Processing, 2002(7):650750, 2002.
[39] Fabian Timm and Erhardt Barth. Non- con-parametric texture defect detection using weibull features. In Image Processing: Machine Vision Applications IV, volume 7877, page 78770J. International Society for Optics and Photonics, 2011.
[40] Daniel Weimer, Berdd Scholz Reiter, and Moshe Shpitalni. Design of deep convolutional neural network architectures for automated feature extraction in industrial inspection. CIRP Annals, (5(1):417-420 2016.
[41] Seunghyeon Kim, Wooyoung Kim, Yung-Kyun Noh, and Frank C Park. Transfer learning for automated optical inspection. In Neural Networks (ICCN)), 2017 International Joint Conference on, pages 2517-2211 -522.. JEEE, 2017.
[42] Barnabás Póczos, Liang Xiong, Dougal Sutherland, and Jeff Schneider. Nonparametric kernel estimators image classification. 2012 IEEE Conference on Computer Vision and Puttern Recognition, pages 2989-22 2996. IEEE, 2012.
[43] GA Kaminka et Randomized distribution feature image classification. In ECAI 2016: 22nd European Conference on Artificial Intelligence 29 August-2 September 2016, The Hague, The Prestigious Applications of Artificial Intelligence (PAIS 2016), volume 285, page 426. IOS Press, 2016.
[44] James R Lloyd and Zoubin Ghahramani. Statistical model criticism using kernel two sample tests. In Advances in Neural Information Processing Systems, pages 829 837, 2015.
[45] Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. Examples are not enough, learn to criticize! criticism for interprctability. In Advances in Neural Information Processing Systems, pages 2280-2288, 2016.
[46] Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bemhard Schölkopf, and Alexander Smola. A kernel swo-sample test. Journal of Machine Learning Research, 13(Mar):723 (0MM)r))723-73,,,, 2012.
[47] Bernhard Schölkopf, John C Platt, John Shawe- Tayawe,Tayyor, Alex J Smola, and Robert C Williamson. Estimating the support of niihh-immeniion distribution. Neural computation, (3(7):1443-147) 2001.
[48] John. C. Platt. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In Advances in Large Margin Classifiers, 2000.
[49] ABA. Graf, O. Bousquet, G. Rätsch, and B. Schölkopf. Prototype classification: Insights from machine learning. Neural Computation, 21(1):272-30 January 2009.
[50] Adam Kowalezyk. Maximal margin perceptron. In Smola, Bartlett Schölkopf, and Schuurmans, editors, Advancex in Large Margin Classifiers, pages 75, 2000.
12

---

Appendix
A Bias Derivation of the Posterior Estimator
The expectation of the weighted KDE is obtained from the following equation:
Ed, [x)]
Extr (a)) (x,,
a(x (pr (x (xx))xx (x,
(22)
f a(x yr (x') 10 K
xx h
dx' a(x hu)p (x + (2)dd2
(23)
with the substitution z , or x'xx== = hz + x to produce dx' h dz, and K ( ho" Pk (x, (x,xx) x') with normalized and isotropic K (z)). We apply Taylor expansion on the term ax(x hz)p1 (x + hz) around x, and with the assumption that [hz] is small,
axx + hz)p1 (x + hz) = (x))pi(x) + hz V[x(x)px + (V(o(x)p((x)))=+ + O(h"),
(24)
using the Hessian operator VV. Now the integration yields the expectation with respect to K ((zz) that satisfies f K(2)dz 1. [22K(2 (z) = 0, and f 'zz" K())dz I:
h²
Ep, Ep, (x)) (x)
2 +
(25)
with the Laplacian operator V²
Along with the expansion for Ep, [P2(x), the following plug-in posterior can be perturbed by à assuming small h:
Bo po (x)
Ep,,A [f(x) + ED, [P] (x)] + tE (x)]
(26)
f(x) + h² (x)
V2 (xx(
(-(0(()pa(x11)
a(x)p=(x) + 0(h))
2 (x) XP2(
(x))
122
f(x) 2 Ply 1|x)
BoPI.Pz O(h²):
(27)
giving the point-wise leading-order bias with respect to
2 2
Bias x) = Ply - 1[x) = 2 x) ) pa (x).
(28)
Here, the Basps.pe p2 (x) is as follows:
Baip (x) (x((())x(xx)
V (x((())22(x]]
(29)
(x)
(())p2(x)
which includes the second derivative of a(x)p) (x) and axx)pz (x). Because two classes use the same weight function (x)) Eq. (29h can be decomposed into terms without the second derivative of cx(x).
VTalz axx)
Vpilx (x))
Vpalz)
)
1 V'pilz P1(x) P2(x)
V'ralx
Bu.pi pz (x)
+
(30)
Pa(x)
B Solution of the Calculus of Variation
For the optimization of Eq. (15) with respect a(x), we first make substitution log and apply
calculus of variation technique for optimal (x)). We express the objective functional with / m((:; 3, V dx using
m(x; B, VB) (v 3lxh(x) + =(x)) r((x).
(31)
3 for notational abbreviation, we apply the Euler-Lagrange equation for the mxxx; A, 3) containing both 8 B::
With the substitution 3
Om x; 8
Vx V m(x; (, 0.
(32)
03
13

---

2 Sz, B where the divergence is Vx V = =E D.
with the i-th component of 3., 8r and the dimensionality is
The first term can be calculated as After we substitute ((x) with log a(x) back, we obtain the equation for the optimal (xx)
0m(x..8.)) 0n
The first derivative of the second term is V # m B, B*)
(3 Shhhh 9)
function:
V ((((logg)) h s)h} O.
(33)
which is Eq. 16
C Analytic Solution for Two Homoscedastic Gaussians
We consider the following two homoscedastic Gaussians
pi (x) (N(x (1, E), p2(x) N N p2, 2):
(34)
with common covariance matrix Σ.
In order to obtain the zero divergence in Eq. 16.. the divergence-free vector field can be obtained using
log Σ- x f(xx
(35)
because the inner product of V loga with h = V Vogg! V log = E-1(Ex -12) should yield a neg-
ative value of g(x), which -g(x) x " ( - nM) [ Σ- - 22 E-22p Pp2). The equation
(V log -9(x) gives
[xx -ー-ーーー1 (11 + (2).
(36)
Therefore, one possible solution for v log a(x) is
v Vlog (x(x = Σー (x 14)
(37)
with the mean of the two class-conditions means, / 22 Therefore, particular solution for axx) is
(xx exp
T 10)
(38)
This solution is not unique, and any log a that has form of
log (x()
n) xx 11) + 1(x).
(39)
with ((x) satisfying VTIE- - w2) = is also the solution. One technique for finding such [(x) that we pick up any differentiable seed function lu (x) and consider its derivative Vl, with the ā Σ (W1
μzl
com-
ponent subtracted: (I (1-
az ||]
Vla. The ((x) is function that its derivative satisfies VI (1
V T..
For example, if we choose lx) x, then (xx) is function that satisfies V/ (1
Vlo I
a
[
we can get ((x) (1-
a
x. If choose lo(x) 111x|1 we get ((x)
f (1
|||ाe
||
after similar calculations. Now the choice of
b (x 11)2
lxxx)
(40)
gives 1(x) = -x (x- 11) (1-
ag |||
(x - 11). which produces our analytic weight function in Eq.
18
(x n'''² A(x v1)).
a(x) = exp
(41)
with p² t1 and (I (1-x--ia (ي - - --1
(1-
with an arbitrary constant
|| (un -للحس
D Posterior Prediction for Various Algorithms
Fig. shows posterior prediction results of various algorithms. the estimation with support vector machines, (48) is used. For neural network estimation, 2-layer fully connected networks with 100 nodes for each layer were used minimizing the mean square error of the sigmoid output. Both VWKDE MB and VWKDE-MF show superior results to other methods.
14

---

1.0
True Posterior
VWKDE
WWKDE MF
SVMs
Neural Networks
Posterior
0.6
0.4
0.2
0.0
-4
-1
3
Figure Posterior predictions with various algorithms for 20-dimensional Gaussians.
D.1 Kernel density estimation high dimensions
We also note the difficulty of density estimation with KDE in high dimensional space. Fig. shows a one- dimensional slice of two 20-dimensional Guassians. The maximum density in this slice is on the order of 10- Meanwhile, the KDE with 5,000 data points per class shows densities on the order of about 10 with bandwidth 0.8.
1e-8
1e-10
1.4
1.2
True p:
True Pz
1.2
KDE p:
1.0
KDE pz
1.0
0.8
0.8
True
KDE
0.6
0.4
0.4
0.2
0.2
0.0
0.0
-3
-2
Figure 8: Underlying density functions and their KDE predictions.
The bandwidth should be chosen to be sufficiently large because no pairs are nearby a high-dimensional space. Although the estimated density with h = 0.8 is rcasonably smooth, the KDE differs from the true density by several orders of magnitude with 5,000 data points. However, the patterns of relative overestimation and underestimation depicted in Fig. aree evident, and the proposed method be applied even with inexact KDEs.
E
Fluid Flow Interpretation of Making Bias
The change of concentration u(t) at time f due the convection and diffusion can be written
Vu D
(42)
0t
15

---

with the direction of convection v and the diffusion constant D', The first termn represents the convection, and the second term represents the diffusion.
The bias Eq. 30
can be reformulated
Eq.
30
α
Vpi
Vp2
V2p p1
V2p2
(43)
pt
p2
p1
p2
log a
lo p1
Vlog pi +
v2 log
(44)
2
2
1 v log
log a +
log p2
V loog pa
+
(45)
2
Ou
Ouz
(46)
Öt
at
Eq. 44) and Eq. (45) can be understood as the two flows with concentrations un log Pt. and 02 log Pa. respectively, and the convection direction for in is Vı = Vilog a + 2 log p1). and the direction for u2 is V2 V(log a + 2 pa). Without the weight, the convection directions are V1 V log p1 and V2 2 log P2 but with weight, they change vı V(log + log and v2 Vllog or + _/2 log
The role of is modify the directions of convection toward V log a together, and in the example shown Fig. D the change of Wa and ua due diffusion are negative and positive, respectively. The direction of convection can control the change of un and u2 and the makes the difference between Qu1 and Ou2 as small possible.
F Prototype Classification Interpretation of the Weighted Kernel Methods in Reproducing Kernel Hilbert Space (RKHS)
The kernel algorithm for classification can be viewed as prototype algorithms in the RKHS due the decompo- sition of positive definite kernel functions [3 30 29] Section 1.2]:
k(x, x') = (o(x), oxx)), ((x), o(x')= E RKHS,
(47)
with the inner product operator (...) defined RKHS.
Given a dataset (xi, ys 1 x, E R.D. Vs E 10, the classification using two prototypes w; Nj Σ =1) (x,(x,) and Wo No E(n,,=)) (xp(x,) in RKHS determines which of the prototypes has smaller distance to the (xx) than the other. Here, Ni. .No are the numbers of data of classes I and O, respectively. The classification using KDEs with the pointwise weights can be compared with the prototype classification in RKHS using the following derivation:
y
Σ
xxk(x;,
Σ axx((x,)) x) >
(48)
N1
No (NU;=0)
(6x=11)
(o(x))) o(x))
Σ as(((x;); o(x)) >
(49)
N1
No
(1
)
Σ aud(x)), o(x)
No Σ (ni((x)) 0(x))
>
(50)
N₂ (1
(H:==0)
([(W1, ((x)) (Wo, o(x))) > 8)
(51)
I [[]w) o(x)]1 ||wo, ((x)) 0)
(52)
with a predetermined threshold 0. Eq. 52
- (l/wi wwl w 2))
With uniform weight a; for all I..... N. the classification is simply the comparison of two KDEs pi
Ni. (im.=1) k(x;; x) and Po No. Σ (A k(x,, x). which correspond to the prototype classification using two empirical means w No
Σ (1:g=1) ((x.) and wo No Σ (4:ps muj Φ(x,) in RKHS. Originating from this correspondence, one suggestion of the unification for the KDE and RKHS is presented [30] The explanations about the prototypes for various kernelized algorithms can be found in [49]. The prototypes of
SVMs are known to be the closest two points within the convex hull of different classes [50].
Despite all these discussions, it is clear that the modification of the densities using weight function will not improve the density estimation performance from the perspective of KDE. Despite the poor density estimation
16

---

performance, the modification improves the classification or informatioo-tthorrtie measure estimation for the KDE plug-in algorithms, but not necessarily the KDE itself. The improvement is partly supported by the prototype models RKHS.
G Least Square Approach for Binary Classification
Reducing the bias of the posterior equation in Eq. corresponds to the least square the prediction error. The optimal square error is achieved with the Bayes classifier, which classifies a datum according to the posterior probability. The posterior probability of xa being generated from pa (x) can he written as:
(xo))
Ply 1/xo) =
(53)
Po (x)))p( Px
P1 (xo)
(54)
YPu + PP (Xo)
where y p(y= 0)/p(y 1). The least square error with
L = / [f(x) v)xp((x,)))pp__x)(( v)dyax
(55)
is achieved with the following prediction function
f(x) Ely 1/x P(v 1/x).
(56)
An accurate estimation of posterior is essential for successful classification. We construct classifier based on the KDE density estimates (x). Pr (x).
Pi (x)
f(x)
(57)
(x) + (x)
(58)
11+(66(x)/
and consider the deviation of f(x) from the true Fly 1|x]
H Details on Optical Surface Inspection Experiments
We use a widely used public surface inspection dataset provided by DAGM for experiments. The dataset contains six distinct textile surface types and associated defect types. There are 1,150 images per class, half of which is for training and the remaining is for testing. Approximately 13% of total images are defective, and for each defective image, masking image which roughly encloses the defective region are provided. The datasct originally proposed for supervised setting.
We extract 900 patches of size 32x 32 from each image, using a sliding window with step size 16. In cach patch, we apply Gaussian smoothing and Scharr kernel to obtain gradient distribution which can capture the texture information. To encode the gradient distribution skewness, and kurtosis. As result, surface image is transformed into set of 900 four-dimensional vectors, a 900x4 matrix. Feature vectors are standardized whitened by aggregating the patches from the same surface type.
feature vector, we compute its mean, standard deviation,
VWKDE can be consumionumingg as a large number of KL divergences need to be computed. Therefore, we take a two-pass approach when applying VWKDE. Given query image. we first apply KDE-based KL divergence estimator to obtain rough estimates of KL divergences. Then, take & images with the lowest KL divergences and apply VWKDE-basee KI. divergence estimator the images to finally select the image with the lowest KL. divergence. This method enables us to have the best of both worlds, the speed of KDE and the accuracy of VWKDE.
The optimal bandwidth for bias reduction methods such as Ensemble 36). vonMises 37. and VWKDE is usually larger than other methods. We use the bandwidth with maximum leave-one-out lokeliieldoood of KDE for other methods but in these three methods, used the heuristic rule of bandwidth selection using the maximum ligelikeldoodd bandwidth for only 25% of randomly selected data.
convolutional neural network (CNN) which takes a 32x32 patch as an input and predicts whether the patch defective trained. For training. we label patches with 75% overlap to the defect mask as defective and patches
Deutsche Arbeitsgemeinschat für Mustererkennung (The German Association for Pattern Recognition).
Data access: httpp:://hci.iiie... www.:eiwwwww...iiieieeieeeee heeideelbegg.///ooe////////
17

---

without any overlap to the defect mask as normal. Patches do not belong either class are discarded. Due to class imbalance, normal patches undersamples to yield defect to normal ratio of 1:4. CNN is trained for each surface type separately. The structure our CNN is MaxPool-FCC (eooo--C(20)-DDD 100//--rrouu-FCCI1 where Conv is x convolution layer, MaxPool is x max pooling layer, FC is a fully connected layer, and DropOut is drop out operation with probability 0.5, We use binary cross entropy loss for objective function and ADAM for optimization.
In unsupervised defect localization, we threshold log probability density ratio (LPDR) estimate to obtain detection results. We threshold LPDR estimates dynamically 90% of maximum LPDR observed in the image. Then, we use KI. divergence estimate as a confidence score for the detection. For a CNN, we use output probability for a patch as a detection score, and set a threshold to 0.9, and the maximum probability of defect among the patches in an image is used as confidence score. Note that, in this experiment, we generate one detection per image because DAGM dataset constrained to have most one defect per image. However, this condition can be relaxed future work with other dataset.
Interseetion-over-umioo (IOU) computed between detection and defect mask. A detection with IOU larger than 0.1 considered as correct detection. This threshold lower than typical threshold in object detection (0.5), because the defect mask is weakly labelled and usually larger than a precise defect region, Using a confidence score assigned for detection, we compute average precision as in PASCAL VOC challenge, then take average surface types.
18


Reviews:
[{"review_id": "FEioJTaAv1", "reviewer": "Reviewer_wV5E", "summary": "The paper aims at improving the KDE estimator for the estimation of density ratios or KL divergences. To do so, they introduce a weighting function $\\alpha$ whose role is to reduce the bias of the estimates. A variational problem is obtained using standard derivations, allowing to learn $\\alpha$ in some model class (neural nets or RKHSs) using optimization principles.\n\nThe benefits of the approach are illustrated on synthetic and real benchmarks.", "questions": "Here are a mix of questions/comments:\n\n- [Major] The paper's structure could be greatly improved. In text derivations make the paper hard to read, and consume a lot of space for little gain. I would suggest to adopt an approach based on writing propositions highlighting the contributions, and making it easier to follow the logic behind the approach. Derivations without immediate interest could be put in the supplementary material. Moreover, what is the point of appendix H ? Either include it in the paper or drop it. Reorganizing the paper would free some space for these results.\n\n- It would have been nice to have a reference for the paragraph at line 19-27 that explains the phenomenon (concave -> underestimation, convex -> overestimation).\n\n- In general, please define symbols before you use them, e.g. for the plug-in posterior estimator there is no mention of labels or random variable $y$ before.\n- Formally, $E_{D_1}$ makes little sense, $E_{x \\sim p_1}$ would be better. The same holds for $E_{D_2}$ and $E_{D_1, D_2}$.\n- The integration of $\\mathcal{O}(h^3)$ at equation (7) is done without care. The local constants depend on $x$ so that there is an argument missing to just integrate.\n- At equation (8), what does the right arrow mean ?\n- At equation (9) $p_0$, $p_1$ should be $p_1, p_2$\n- Argument for going from (8) to (9) is missing\n- The assumptions regarding the kernel are not clearly stated (e.g. line 80). It would be good to have a \\begin{assumptions} dedicated to the kernel from the beginning, so that the readers knows what kind of mathematical object is dealt with.\n- At line 93: when you calculate the KL, why do you remove x_i ?\n- At equation (15), it would be good to have some notion of the hypothesis space of $\\alpha$\n- In general, I find it weird to write an equality depending on $x$ on the left side and dropping it on the right side (e.g. equation 12).\n- It is hard to understand equation (16) as it is. Is it for all $x, \\alpha(x)$ should satisfy this ? Or is it from a functional point of view ? The answer can be found in the supplementary material, but the link is not made whith section B. Having a clearly stated proposition/proof in the appendix would solve this problem.\n- At line 125: in general in the kernel literature, $\\phi$ is the feature map and not the kernel, it would be good to respect this convention.\n- I struggle to understand the difference between model free and model based ? To me there are two models: one is a NN, the other in a RKHS.\n- Algorithm 1 could be cut in two parts, model free and model based, fitting the same vertical amount of space, allowing to detail the process for the model based optimization process.\n- At line 135: one tries to minimize $B^2$ not $B$.\n- Please use vectorial images allowing to zoom without blur, in general the plots could be greatly improved.\n- Please use a standard font for the RKHS acronym.\n", "limitations": "Limitations have been addressed.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The problem is of interest to the machine learning community\n- The approach is novel\n- The experiments section shows improvements over existing techniques", "weaknesses": "- The paper is not well written and lacks mathematical rigor (see questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe acknowledge the relevance of all the comments and have outlined our responses below, explaining how we intend to revise the manuscript in accordance with the feedback.\n\n> Major comment on the structure and the illustration of the main equations\n\nWe will highlight the main equations and make sure their significant contributions come to the forefront. Part of the detailed derivation will be relocated to the Appendix, and we will let the sketch of the derivation techniques remain, so interested readers can derive the result themselves. We will get rid of Section H in the Appendix.\n\n\n> Reference for the phenomenon: concave -> underestimation and convex -> overestimation\n\nWe will include chapter 6 of Larry Wasserman’s book (All of nonparametric statistics) as a reference. The standard derivation of the KDE bias includes the second derivative of the underlying density function. Though the association of the under- and overestimation and the concave and convex local density, respectively, is obvious from the literature and Fig. 1, the role of the weighting function to mitigate the bias proposed at line 28-35 stems from our original intuition and, to the best of our knowledge, have no prior reference. \n\n\n> Define symbols before you use them.\n\nWe will check every symbol whether it is used before definition. It is correct that the variable y is used before definition.\n\n\n> Formally, E_{D_1} makes little sense, E_{x\\sim p_1} would be better. The same holds for E_{D_2} and E_{D_1, D_2}.\n\n$D_1$ and $D_2$ are used to represent sets. For example in Eq. (4), the first and the second equation, \n$\\mathbb{E}_{\\mathcal{D}_1}[\\cdot]$\n\n$=\\mathbb{E}_{\\mathbf{x}’\\sim p_1(\\mathbf{x})}[\\cdot]$,\nis used to represent that the expectation in terms of the set $\\mathcal{D}_1$ is equal to the expectation with respect to one variable $\\mathbf{x}’$. Note that $\\mathbf{x}$ is a fixed point, and $\\mathbf{x}’$ is a random variable..\n\n\n> The integration of O(h^3) at equation (7) is done without care.\n\nWe appreciate this correction! The comment is correct!! We will change O(h^4) to O(h^3) and change the writings accordingly including the definition of the kernels that we use.\n\n\n> The meaning of right arrow in Equation (8)\n\nRight arrow means the convergence. Asymptotically, each estimator in the numerator and denominator convergences.\n\n\n> At equation (9), p_0, p_1 should be p_1, p_2\n\nThank you! The indexes are corrected.\n\n\n> Argument for going from (8) to (9) is missing\n\nThe explanation will be included!\n\n\n> Assumption of the kernel\n\nThis comment is completely relevant. We will include the mathematical assumption for the kernel.\n\n\n> At line 93: when you calculate the KL, why do you remove x_i ?\n\nThe derived expectation $\\mathbb{E}_{\\mathbf{x}’\\sim p}$ does not hold for $\\mathbf{x}’ = \\mathbf{x}_i$ because $\\mathbf{x}_i$ is a fixed point for Monte-Carlo summation. We have to use $\\mathbf{x}_i$ for $\\mathbf{x} = \\mathbf{x}_i$.\n\n\n> Hypothesis space for \\alpha at equation (15)\n\nThank you. $\\alpha$ is a positive differentiable function. We will include the hypothesis space.\n\n\n> Argument x in the right hand side of equation (12)\n\nWe will include x in every derivative and function on the right hand side.\n\n\n> Explanation for equation (16) and the link to Section B.\n\nThe alpha function should satisfy Eq. (16) for all x. We will include this statement and write down a detailed derivation from Eq. (23) to Eq. (24) in Section B of the supplementary material and write down in the main text to refer to Section B.\n\n> Do not use \\phi for kernel function and respect the convention\n\nBecause $k$ is reserved for the kernel in KDE, we will use $\\kappa$ to represent the RKHS basis. \n\n\n> Model-based and model-based: are they simply a function in RKHS and a NN?\n\nThe name model-based does not come from the construction of a function in RKHS. For model-based function, we used a strong assumption that p_1 and p_2 are both Gaussians (L121). Model-free does not assume such models.\n\n> Algorithm 1\n\nWe will make room to elaborate the model-based method in Algorithm 1.\n\n\n> B -> B^2\n\nYes, corrected.\n\n\n> Vectorial image\n\nFigs 1-3 are not vectorized. We will make vectorial images for those figures.\n\n\n> RKHS acronym font\n\nWe will use the standard font.\n"}, {"review_id": "5LRgH9amzd", "reviewer": "Reviewer_QwPp", "summary": "The authors propose an optimal weighting scheme to reduce bias in kernel density ratio estimation that alleviates the problem of under (or over) estimation due to the geometry of the distribution. They show that employing this weighting scheme leads to better estimates of prediction posteriors. ", "questions": "See above.", "limitations": "Limitations have been discussed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is clearly motivated, and is theoretically sound (as far as I could tell).\n- I am not an expert in the area and found the paper clear and easily accessible. \n- I found the problem well motivated and the idea of adjusting the weight function variationally after estimating the first and second order properties of the underlying distributions to account for the under (or over) estimation interesting and well explored.\n- The empirical analysis is detailed and shows the proposed approach leads to better estimation.", "weaknesses": "The proposed method is computationally more expensive. I am curious how the performance compares between different methods given the same amount of compute resources to make the comparison fair. Also, in the experimental results that are reported, what quantity has been kept constant across methods for the comparison? The weighting function does not improve convergence rates for KDE.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive questions.\n\n> The proposed method is computationally expensive. Fair comparison\n\nThe following is a table for the computation time with the 10D non-isotropic experiment in Fig. 4 for 4.2k samples per density:\n\n| Algorithm  | Time (s) |\n|------------|----------|\n| KDE         | 0.02       |\n| Ensemble |  1.68     |\n| Von          |   1.09    |\n| MINE       |   1.83    |\n| VWKDE_MB                                    |  2.22    |\n| VWKDE_MF Score model training  |  12.46  |\n| VWKDE_MF Weight model training |  73.7   |\n| VWKDE inference (with learned $\\alpha(\\mathbf{x})$) |  0.52   |\n\n\nThe learning of weight function ($\\alpha(\\mathbf{x})$) is computationally expensive as pointed out by the reviewer. We will include the table of computation time in the Appendix for the readers who are interested in the fair comparison.\n\nFrom the perspective of practical usage, we note that VWKDE consists of two separable operations: the weight function learning and the nonparametric estimation with the learned weight. The weight function learning is a one-time operation, and the learned function can be reused. For instance, after training the $\\alpha(\\mathbf{x})$ function with a specific number of samples, we have the option to reuse the learned $\\alpha(\\mathbf{x})$ when we are capable of performing VWKDE with a larger dataset. While the results might not match those achieved with a retrained $\\alpha(\\mathbf{x})$, this approach offers a practical trade-off between time efficiency and increased accuracy. The VWKDE calculation time with a given $\\alpha(\\mathbf{x})$ is short as shown in the [VWKDE inference] in the above table.\nWe will also include this discussion in the Appendix.\n\n\n\n> The weighting function does not improve convergence rates for KDE.\n\nThe convergence rate has not improved. Only the asymptotic bias is improved. \nOur algorithm also provides many empirical factors on top of the derivation for bias reduction. We report that the proposed model-based and model-free methods empirically yield favorable outcomes when the direct information about the underlying densities and their derivatives are unavailable.\n"}, {"review_id": "ngM2Mj6eDH", "reviewer": "Reviewer_rQ6e", "summary": "The paper studies the weighted kernel density estimation (wKDE) and proposes a method to learn the weights.", "questions": "Instead of introducing the weights, why not learning a new kernel?", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors illustrate biases in the standard KDEs and propose a mitigation by introducing weights in the KDE.\n2. An estimator for the weight function is provided based on minimizing a quadratic loss.\n3. Improved results are shown for the proposed method on the polluted MNIST data compared to several other approaches.", "weaknesses": "1. It seems that the bias occurs only when the data x is not sampled from the true distribution p(x). In this setting, the proposed method could reduce the bias. For x sampled from p(x), the standard KDE is a kernel mean embedding which is an unbiased estimator of the density.\n2. The weighted KDE is essentially equivalent to the standard KDE with a kernel specified by the integral (Eq. 2). Thus, the weight learning may not outperform the standard KDE with an appropriate kernel.\n3. The learning objective (Eq. 15) is based on a quadratic function, there could be better suited losses for densities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for taking the time to review our paper. Below, we rebut the claims that the reviewer provided. We hope the concerns of the reviewer to be resolved.\n\n\n> The bias occurs only when the data x is not sampled from the true distribution p(x). \n\nThis is not true. KDE should produce asymptotic bias with “nonzero bandwidth” even when x is from the true density.\n\n\n> For x sampled from p(x), the standard KDE is a kernel mean embedding which is an unbiased estimator of the density.\n\nNo! The KDE is a “biased” estimator of the density. The nonzero bias exists even with infinite samples due to the nonzero bandwidth. Are you mentioning that the KDE equation is also used as the estimator of kernel mean embedding (KME)? In this case, it is an unbiased estimator of KME, not the unbiased estimator of the density. Could you clarify? The unbiasedness of KME estimator is found in the Eq.(3.12) in arXiv:1605.09522.\n\n\n> The weighted KDE is essentially equivalent to the standard KDE with a kernel specified by the integral (Eq. 2). Thus, the weight learning may not outperform the standard KDE with an appropriate kernel.\n\nRegarding the asymptotic convergence rate, the reviewer's assertion is true. Thus, the standard convergence proof can be applied interchangeably to both the conventional and the weighted KDE. However the asymptotic biases to which they converge are different with nonzero bandwidth. According to our derivation, the weighted KDE converges with a bias in Eq. (12), and the standard KDE converges with a bias which equals the second term in Eq. (12). Therefore, the first term in Eq. (12) is the difference that can make the weighted KDE outperform.\n\n\n> Quadratic objective function in (Eq. 15)\n\nThe proposed quadratic objective produces the optimal condition for $\\alpha(\\mathbf{x})$ shown in Eq. (16) after applying the calculus of variation. Because of the special form in Eq. (16) regarding r(x), we could make a practical algorithm without considering r(x). If we use other objective functions, we expect many other objectives will produce an equation where r(x) is entangled.\n\n\n> Instead of introducing the weights, why not learning a new kernel?\n\nIn Eq. (12), the first term can be controlled by weight, and we tried to make the whole bias in Eq. (12) disappear by controlling the first term. Without weight, the first term is a constant (zero). However as the reviewer suggested, learning the shape of a kernel that directly tackles the second term looks also valid. This motivation is similar to those tried in [25] though the research is performed in a different setting with nearest neighbor information. However, we expect the kernel learning to directly tackle the second term will produce similar results to those from [25] which we presented in Fig. 4.\n"}, {"review_id": "VGL2mw7UDw", "reviewer": "Reviewer_oTo7", "summary": "The paper presents an optimized weight function for kernel density estimations, which reduces bias and improves estimates of prediction posteriors and information-theoretic measures. The weight function is derived using tools from multidimensional calculus of variations and utilizes information from the first derivatives to mitigate bias caused by the second derivatives.\n\nExperiments demonstrate the reliable bias reduction achieved by using a simple choice of a Gaussian density model for the derivatives. Interestingly, this bias reduction is independent of the accuracy of the model, addressing a common dilemma in conventional methods. Typically, using a coarse and inaccurate model results in low bias but high variance. In this case, the use of a coarse model does not affect the flexibility of the estimation procedure, while effectively addressing high-dimensional bias.\n\nHowever, there are some limitations to consider, such as the computational overhead of score learning using parametric or neural network methods, and the lack of impact on the asymptotic convergence rate, which depends on the convergence rate of the kernel density estimation. Nevertheless, the use of a non-flexible parametric model consistently improves the kernel density estimation.\n", "questions": "- How does VWKDE react to very small bandwidth for multimodal datasets really many observations? I am interested in KDE applications with such data.", "limitations": "No limitations are being stated. I do not see any limitation myself.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Content: The paper presents a novel approach to kernel density estimation using an optimized weight function. This method reduces bias and variance, addressing the traditional trade-off dilemma between bias and variance. The proposed method is independent of the model's accuracy and effectively handles dimension-dependent bias. Importantly, it does not compromise the flexibility of the estimation procedure. Overall, the paper stands out for its innovative approach, solid theoretical foundation, effective bias reduction, flexibility, and superior performance.\n\nFormalism: The paper is well-written (up to minor details, see below), full of details, perhaps slightly too formalistic. It makes a non-trivial contribution seem easy to follow.\n\nExperiments: Several experiments in various complex ways are being presented. \n", "weaknesses": "I have no major problems with the paper. While regularly using KDEs and all tools employed in the paper, I am not sure that I know all of the recent literature. \n\nThere are some minor things, that can easily be fixed for the camera ready version:\n - l. 23-27: Please mention here the cause of underestimation and overestimation in convex and concave regions (Why does it happen? Perhaps refer to Figure 1?). \n - Figure 1: In this figure, only the difference at a single point x_0 is depicted. Would it be possible to include the complete representation of $\\hat{p_1}$ and $\\hat{p_2}$ (dotted lines)? Here or elsewhere?\n - l. 30-35 or 36: Perhaps mention in the introduction that $\\alpha$ is used as the product in the kernel density estimator.\n - l. 72-73: Explain $f$ in Equation 3. The symbol was not mentioned before.\n - Fig. 2: Write out the abbreviations (VWKDE) when they are used for the first time. Perhaps even state the name much earlier, like in the introduction or in the abstract?\n - Inclusion of code would have been nice. (However, the method does not seem to be difficult to implemented and does not strongly depend on hyperparameter choices. So this is not a major drawback.)\n\nIt is typical that the supplementary material does not reduplicate the paper. ;-)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for taking the time to review our paper. We deeply value your feedback.\n\n\n> l. 23-27: Mention the cause of underestimation and overestimation in convex and concave regions\n\nFirst, we will include Larry Wasserman (2005) *All of nonparametric statistics*, Springer, in our revised version as a reference for providing a standard derivation of the KDE bias. In the proof of Theorem 6.28 in this book, the following equation for the KDE bias can be found:\n\\begin{eqnarray}\n\\mathbb{E}[K_{h_n}(x, X)]  - f(x) = \\frac{1}{2}\\sigma_K^2 h_n^2 f’’(x) + O(h_n^4),\n\\end{eqnarray}\nwhere $f(x)$ is the true density and $f’’(x)$ is its second derivative. \n\nThe first two figures in Fig. 1 is an illustration of this equation. For explanation, we additionally assume a new density $p_3$ that has an exactly equivalent value to $p_1$ at $\\mathbf{x}_0$, or $p_3(\\mathbf{x}_0) = p_1(\\mathbf{x}_0)$, but is uniform in the neighborhood of $\\mathbf{x}_0$. Then the number of samples from $p_1$ should be smaller than the number of samples from $p_3$ around $\\mathbf{x}_0$ because the integration of the density is the probability. The KDE counts the effective number of the neighborhood samples and is unbiased only for the samples from a uniform density throughout the domain near $\\mathbf{x}_0$. Therefore, the KDE should underestimate $p_1(\\mathbf{x}_0)$ while the KDE for $p_3(\\mathbf{x}_0)$ is unbiased. The overestimation of $p_2(\\mathbf{x}_0)$ can be explained similarly. The association of the second derivative and the bias is inevitable due to the nonzero bandwidth in the kernel. \n\n\n> Figure 1: not only at a single point x_0, include the complete representation of $\\widehat{p}_1$ and $\\widehat{p}_2$.\n\nIn addition to Fig.2(a) and Fig3(a), we will include a figure in the Appendix that shows the true p_1, p_2, the estimated $\\widehat{p}_1$, $\\widehat{p}_2$, and the true and the estimated ratios for all points along a one-dimension span.\n\n\n> l. 30-36: Mention that \\alpha is used as the product in the kernel density estimator\n\nWe will update the introduction to highlight that the alpha is the weight being multiplied to kernels.\n\n\n> l. 72-73: Explain f in Equation 3. The symbol was not mentioned before.\n\nWe will mention f(x) and its meaning before use. \n\n\n> State the abbreviation VWKDE early.\n\nWe will write out the abbreviation VWKDE at the title of Section 2 and the abstract.\n\n\n> Inclusion of code\n\nThank you for your suggestion. We will make the implementation of VWKDE publicly available through an online repository.\n\n\n\n> How does VWKDE react to very small bandwidth for multimodal datasets really many observations? I am interested in KDE applications with such data.\n\nAs in conventional KDE, the bandwidth of VWKDE can be small if a large number of training data are provided. In theory, VWKDE can allow a larger bandwidth without sacrificing the mean square error because the weight function yields a bias reduction, which otherwise should be achieved by using a small bandwidth.\n"}]

Author rebuttal:

Meta-review: Reviewers agree about the novelty, structure, and the results of the paper.
Therefore the paper is recommended for acceptance with the changes discussed in the rebuttal.
Decision: Accept (poster)