SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The equation for each class can be compared with the mechanical convection-diffusion equation,  $\frac{\partial u}{\partial t} = -\mathbf{v}^{\top}\nabla u + D^{\prime}\nabla^{2}u$ , which is known as the equation for Brownian motion under gravity [27] or the advective diffusion equation of the incompressible fluid [28]. In the equation,  $u$  is the mass of the fluid,  $t$  is the time,  $\mathbf{v}$  is the direction of convection, and  $D^{\prime}$  is the diffusion constant. The amount of mass change is the sum of the convective movement of mass along the direction opposite to  $\nabla u$  and the diffusion from the neighborhood. We reorganize Eq. (5) into the following equation establishing the difference between the convection-diffusion equations of two fluids:
$$
\begin{array}{l} B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}) = \left[ \nabla^ {\top} \left(\log \alpha + \frac {1}{2} \log p _ {1}\right) \nabla \log p _ {1} + \frac {1}{2} \nabla^ {2} \log p _ {1} \right] \\ - \left[ \nabla^ {\top} \left(\log \alpha + \frac {1}{2} \log p _ {2}\right) \nabla \log p _ {2} + \frac {1}{2} \nabla^ {2} \log p _ {2} \right]. \tag {20} \\ \end{array}
$$
According to the equation, we can consider the two different fluid mass functions,  $u_{1}(\mathbf{x}) = \log p_{1}(\mathbf{x})$  and  $u_{2} = \log p_{2}(\mathbf{x})$ , and the original convection movement along the directions  $\mathbf{v}_1' = -\frac{1}{2}\nabla \log p_1$  and  $\mathbf{v}_2' = -\frac{1}{2}\nabla \log p_2$ . If we make an  $\alpha (\mathbf{x})$  that modifies the convection directions  $\mathbf{v}_1'$  and  $\mathbf{v}_2'$  to  $\mathbf{v}_1 = \mathbf{v}_1' - \nabla \alpha$  and  $\mathbf{v}_2 = \mathbf{v}_2' - \nabla \alpha$ , and a mass change in one fluid is compensated by the change of the other, in other words, if  $\frac{\partial u_1}{\partial t} = \frac{\partial u_2}{\partial t}$ , then the  $\alpha (\mathbf{x})$  is the weight function that minimizes the leading term of the bias for ratio estimation.
4.3 Prototype modification in reproducing kernel Hilbert space (RKHS)
A positive definite kernel function has its associated RKHS. A classification using the ratio of KDEs corresponds to a prototype classification in RKHS that determines which of the two classes has a closer mean than the other [29, Section 1.2]. The application of a weight function corresponds to finding a different prototype from the mean [30]. The relationship between the new-found prototype and the KDEs has been previously discussed [31].
Figure 3: Estimation results of the LPDR  $\log (p_1 / p_2)$  and the K-L divergence. (a) Estimation of LPDR at each point. The estimation bias from the true LPDR is reduced by using VWKDE. (b) Squared bias and variance of the estimation with respect to the bandwidth  $h$ . Bias has been significantly reduced without increasing variance. (c) Mean and standard deviation of K-L divergence estimates with respect to the bandwidth  $h$ .
Figure 4: K-L divergence estimation results for synthetic distributions; (NN) Nearest-neighbor estimator; (NNG) NN estimator with metric learning [32]; (KLIEP) Direct importance estimation [21]; (NNGarcia) Risk-based  $f$ -divergence estimator [33]; (NNWang) Bias-reduced NN estimator [34]; (MINE) Mutual Information Neural Estimation [35]; (Ensemble) Weighted ensemble KDE estimator [36]; (vonMises) KDE estimator with von Mises expansion bias correction [37]; (KDE) KDE estimator; (VWKDE-MB, VWKDE-MF) Model-based and model-free approach of the proposed estimator in this paper.
5.1 Estimation of log probability density ratio and K-L divergence in 1D
We first demonstrate in Fig. 3 how the use of VWKDE alters log probability density ratio (LPDR) and K-L divergence toward a better estimation. We use two 1-dimensional Gaussians,  $p_1(x)$  and  $p_2(x)$ , with means 0 and 1 and variances  $1.1^2$  and  $0.9^2$ , respectively. We draw 1,000 samples from each density and construct KDEs and model-based VWKDEs for both LPDR and K-L divergence. For LPDR evaluation, we draw a separate 1,000 samples from each density, and the average square of biases and the variances at those points are calculated and presented in Fig. 3(b). The K-L divergence estimation result is shown in Fig. 3(c), where the true K-L divergence can be calculated analytically as  $KL(p_1||p_2) \approx 0.664$ , and the estimated values are compared with this true K-L divergence.
KDE-based LPDR estimation exhibits a severe bias, but this is effectively reduced by using VWKDE as an alternative plug-in. Although VWKDE slightly increases the variance of estimation, the reduction of bias is substantial in comparison. Note that since the bias is small over a wide range of  $h$ , VWKDE yields a K-L divergence estimate which is relatively insensitive to the choice of  $h$ .
5.2 Synthetic distributions
We perform the VWKDE-based K-L divergence estimator along with other state-of-the-art estimators to estimate the K-L divergence  $KL(p_{1}||p_{2})$  between two synthetic Gaussian distributions  $p_1$  and  $p_2$  having  $\mu_{1}$  and  $\mu_{2}$  as their mean vectors and  $\Sigma_{1}$  and  $\Sigma_{2}$  as their covariance matrices, respectively.
Figure 5: Estimation of K-L divergence between two non-Gaussian densities,  $p_1(\mathbf{x})$  and  $p_2(\mathbf{x})$ . Each density is the Gaussian mixture of the three Gaussians, as shown in the 2-dimensional density contour in the figure on the left. They are the true densities but are very dissimilar to the single Gaussian model. The figure in the middle shows the estimation with 2-dimensional data, and the figure on the right shows the estimation with 20-dimensional data. With 20-dimensional data, the remaining 18 dimensionalities have the same mean isotropic Gaussians without correlation to the first two dimensionalities.
Figure 6: Detection of an artificially injected defect (MNIST digit "3"). The first panel shows the image with an injected defect. The remaining three panels show the detection scores of different methods.
We use three different settings for  $p_1$  and  $p_2$ : Isotropic (Iso), Non-isotropic Heteroscedastic (NH), and Varying Mean Diff (VMD). In Iso,  $\Sigma_1 = \Sigma_2 = I$  for an identity matrix  $I$ .  $\mu_1 = 0$  for a zero vector  $\mathbf{0}$ . The first element of  $\mu_2$  is  $\sqrt{2}$ , while the other elements are uniformly zero, resulting in  $KL(p_1||p_2) = 1$ . In NH,  $\mu_1 = \mu_2 = 0$ , and  $\Sigma_1 = \mathbf{I}$ .  $\Sigma_2$  is a matrix having a pair of off-diagonal elements  $(\Sigma_2)_{1,2} = (\Sigma_2)_{2,1} = 0.1$ , and other off-diagonal elements are zero. The diagonal elements have a constant value  $\omega$ , which is determined to yield  $KL(p_1||p_2) \approx 1.0$  with  $\omega = 0.750^2$  (10D) and  $KL(p_1||p_2) \approx 0.5$  with  $\omega = 0.863^2$  (20D). In VMD, the first element of  $\mu_2$  has various values between 0 and 2, while  $\mu_1 = 0$  and the other elements in  $\mu_2$  remain zero.  $\Sigma_1 = \Sigma_2 = I$ . In Iso and NH, we vary the sample size, and in VMD, we use 2,000 samples per distribution. We repeat each experiment 30 times and display the mean and the standard deviation in Figure 4.
In the upper left panel of Figure 4, we observe that almost all algorithms estimate the low-dimensional K-L divergence reliably, but the results deteriorate dramatically as shown in the upper middle and right panels with high-dimensionality. As shown in the lower left and lower middle panels, most of the baseline methods fail to produce the estimates near the true value when data are correlated. The model-based VWKDE-based estimator is the only estimator that recovers the true value in the 20D NH case. Figure 5 shows the K-L divergence estimation for non-Gaussian densities. In this example, the model for the score function in model-based VWKDE is different from the data-generating density, in which the estimator still shows very reliable estimates.
5.3 Unsupervised optical surface inspection
We apply the proposed K-L divergence estimation using VWKDE for the inspection of the surface integrity based on the optical images. Most of the previous works have formulated the inspection using the supervised setting [38, 39, 40, 41]; however, often the defect patterns are diverse, and training data do not include all possible defect patterns. In real applications, identification and localization of unseen defect patterns are important. In this example, we apply the model-based VWKDE.
Detection of defective surface Following the representations of previous works on image classification [42, 43], we extract random small patches from each image  $\mathbf{I}$  and assume that those patches are the independently generated data. We use the probability density  $p_{\mathbf{I}}(\mathbf{x})$ , for the patch  $\mathbf{x} \in \mathbb{R}^D$  from  $\mathbf{I}$ . Given the  $N$  number of normal surface images  $\mathcal{D} = \{\mathbf{I}_i\}_{i=1}^N$  and a query image  $\mathbf{I}^*$ , we determine whether  $\mathbf{I}^*$  is a defective surface according to the following decision function  $f(\mathbf{I}^*)$  and a predefined
Table 1: Performances for defect surface detection (left) and defect localization (right). mAUC and mAP are averaged over six surface types of DAGM. The DAGM dataset is provided with labels, and only CNN used the labels for training. The unseen defect is the artificially injected MNIST digit "3."
mAUC	DAGM Defect	Unseen Defect	mAP	DAGM Defect	Unseen Defect
VWKDE	0.785 ± 0.002	0.967 ± 0.003	VWKDE	0.369 ± 0.005	0.903 ± 0.007
KDE	0.734 ± 0.005	0.926 ± 0.003	KDE	0.294 ± 0.004	0.849 ± 0.006
NN-1	0.628 ± 0.002	0.813 ± 0.001	NN-1	0.095 ± 0.008	0.488 ± 0.002
NN-10	0.540 ± 0.003	0.614 ± 0.002	NN-10	0.081 ± 0.004	0.254 ± 0.002
NNWang	0.605 ± 0.002	0.657 ± 0.004	NNWang	0.029 ± 0.005	0.024 ± 0.000
MMD	0.618 ± 0.003	0.615 ± 0.008	MMD	0.151 ± 0.006	0.032 ± 0.001
OSVM	0.579 ± 0.001	0.538 ± 0.000	OSVM	0.249 ± 0.012	0.444 ± 0.009
CNN	0.901 ± 0.011	0.809 ± 0.029	CNN	0.699 ± 0.037	0.564 ± 0.060
threshold:
$$
f \left(\mathbf {I} ^ {*}\right) = \min  _ {\mathbf {I} _ {i} \in \mathcal {D}} \widehat {K L} \left(p _ {\mathbf {I} ^ {*}} \| p _ {\mathbf {I} _ {i}}\right). \tag {21}
$$
Defect localization Once the defective surface is detected, the spot of the defect can be localized by inspecting the LPDR  $\log (p_{\mathbf{I}^*}(\mathbf{x}) / p_{\mathbf{I}_m}(\mathbf{x}))$  score between the query image  $\mathbf{I}^*$  and the  $\mathbf{I}_m$  with  $\mathbf{I}_m = \arg \min_{\mathbf{I}_i\in \mathcal{D}}\widehat{KL}(p_{\mathbf{I}^*}\| |p_{\mathbf{I}_i})$ . The location of the patch  $\mathbf{x}$  with the large LPDR score is considered to be the defect location. Note that a similar approach has been used for the witness function in statistical model criticism [44, 45].
For the evaluation of the algorithm, we use a publicly available dataset for surface inspection: DAGM $^2$ . The dataset contains six distinct types of normal and defective surfaces. The defective samples are not used in training, but they are used in searching the decision thresholds. We extract 900 patches per image, and each patch is transformed into a four-dimensional feature vector. Then, the detection is performed and compared with many well-known criteria: diverse K-L divergences estimators as well as the maximum mean discrepancy (MMD) [46] and the one-class support vector machines (OSVM) [47]. In addition, the Convolutional Neural Networks (CNNs) training result is presented for comparison with a supervised method.
In DAGM, the testing data have defect patterns similar to those in the training data. To demonstrate unseen defect patterns, we artificially generate defective images by superimposing a randomly selected  $15\%$  of the normal testing images with a small image of the MNIST digit '3' at a random location (see Figure 6). Table 1 presents the area under curve (AUC) of the receiver operating characteristic curve for the detection as well as the mean average precision (mAP) for the localization.
CNNs which use labels for training show good performances only in detecting and localizing DAGM defects. The K-L divergence estimation with VWKDE show the best performance over many unsupervised methods, and it provides significantly better performances both at identifying unseen defects and at localizing them. Figure 6 shows one example of how well the proposed method localizes the position of the unseen defects.
In this paper, we have shown how a weighted kernel formulation for the plug-in densities could be optimized to mitigate the bias in consideration of the geometry of densities. The underlying mechanism uses the information from the first derivatives to alleviate the bias due to the second derivatives.
In our experiments, a simple choice of Gaussian density model for obtaining the first and second derivatives led to a reliable reduction of bias. This insensitivity to the exactness due to a coarse model is nonintuitive considering the traditional dilemma prevalent in many conventional methods; a coarse and inexact model enjoys a small variance but at the cost of large bias. In our work, the usage of the coarse model had no effect on the flexibility of the plug-in estimator, while the high dimensional bias was tackled precisely.
2Deutsche Arbeitsgemeinschaft fur Mustererkennung (The German Association for Pattern Recognition).
Limitations of this study include the computational overhead for score learning using parametric or neural network methods and no benefit for the asymptotic convergence rate because it depends on the convergence rate of KDE. Using a non-flexible parametric model rather than a flexible one provides a consistent benefit to improve the KDE.
SY was supported by a KIAS Individual Grant (AP095701) via the Center for AI and Natural Sciences at Korea Institute for Advanced Study. SY and FCP were supported in part by IITP-MSIT (2021-0-02068, 2022-0-00480), ATC+ (20008547), SRRC NRF (RS-2023-00208052), and SNU Institute for Engineering Research. GY was partly supported by IITP-MSIT (2019-0-01906), and IK and YKN was supported by NRF/MSIT (No. 2018R1A5A7059549, 2021M3E5D2A01019545), IITP/MSIT (IITP-2021-0-02068, 2020-0-01373, RS-2023-00220628). YKN was supported by Samsung Research Funding & Incubation Center for Future Technology (SRFC-IT1901-13) partly in the derivation of the weight dependency on the bias and its mechanical interpretation.
[1] Jaewoong Cho, Gyeongjo Hwang, and Changho Suh. A fair classifier using kernel density estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15088-15099. Curran Associates, Inc., 2020.
[2] Edward Gan and Peter Bailis. Scalable kernel density classification via threshold-based pruning. In SIGMOD '17: Proceedings of the 2017 ACM International Conference on Management of Data, pages 945-959, 2017.
[3] Longin Jan Latecki, Aleksandar Lazarevic, and Dragoljub Pokrajac. Outlier detection with kernel density functions. In Petra Perner, editor, Machine Learning and Data Mining in Pattern Recognition, pages 61-75, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg.
[4] Dirceu Scaldelai, Luiz Carlos Matioli, Solange Regina dos Santos, and Mariana Kleina. MulticlusterKDE: a new algorithm for clustering based on multivariate kernel density estimation. Journal of Applied Statistics, 49(1):98-121, 2020.
[5] Dirk Ormoneit and Saunak Sen. Kernel-based reinforcement learning. Machine Learning, 49:161-178, 2002.
[6] Haanvid Lee, Jongmin Lee, Yunseon Choi, Wonseok Jeon, Byung-Jun Lee, Yung-Kyun Noh, and Kee-Eung Kim. Local metric learning for off-policy evaluation in contextual bandits with continuous actions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 3913-3925. Curran Associates, Inc., 2022.
[7] Akshay Krishnamurthy, Kirthevasan Kandasamy, Barnabas Poczos, and Larry Wasserman. Nonparametric estimation of renyi divergence and friends. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 919–927, Beijing, China, 22–24 Jun 2014. PMLR.
[8] Dongxin Xu and Deniz Erdogmuns. Renyi's entropy, divergence and their nonparametric estimators. In: Information Theoretic Learning. Information Science and Statistics, 2010.
[9] Jose C. Principe, Dongxin Xu, Qun Zhao, and John W. Fisher. Learning from examples with information theoretic criteria. Journal of VLSI Signal Processing Systems, 26(1-2):75-113, 2000.
[10] Kumar Sricharan, Raviv Raich, and Alfred O. Hero III. Estimation of nonlinear functionals of densities with confidence. IEEE Transactions on Information Theory, 58(7):4135-4159, 2012.
[11] Ziv Goldfeld, Kristjan H. Greenewald, Jonathan Niles-Weed, and Yury Polyanskiy. Convergence of smoothed empirical measures with applications to entropy estimation. IEEE Transactions on Information Theory, 66(7):4368-4391, 2020.
[12] Larry Wasserman. All of nonparametric statistics: a concise course in nonparametric statistical inference. Springer, 2005.
[13] Peter Hall. Effect of bias estimation on coverage accuracy of bootstrap confidence intervals for a probability density. The Annals of Statistics, 20(2):675 - 694, 1992.
A Bias Derivation of the Posterior Estimator
The expectation of the weighted KDE is obtained from the following equation:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {1}} \left[ \widehat {p} _ {1} (\mathbf {x}) \right] = \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p _ {1} (\mathbf {x})} \left[ \alpha \left(\mathbf {x} ^ {\prime}\right) k _ {h} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) \right] = \int \alpha \left(\mathbf {x} ^ {\prime}\right) p _ {1} \left(\mathbf {x} ^ {\prime}\right) k _ {h} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) d \mathbf {x} ^ {\prime} (22) \\ = \int \alpha \left(\mathbf {x} ^ {\prime}\right) p _ {1} \left(\mathbf {x} ^ {\prime}\right) \frac {1}{h ^ {D}} K \left(\frac {\mathbf {x} ^ {\prime} - \mathbf {x}}{h}\right) d \mathbf {x} ^ {\prime} = \int \alpha (\mathbf {x} + h \mathbf {z}) p _ {1} (\mathbf {x} + h \mathbf {z}) K (\mathbf {z}) d \mathbf {z}, (23) \\ \end{array}
$$
with the substitution  $\mathbf{z} = \frac{\mathbf{x}' - \mathbf{x}}{h}$ , or  $\mathbf{x}' = h\mathbf{z} + \mathbf{x}$  to produce  $\mathrm{d}\mathbf{x}' = h^{D}\mathrm{d}\mathbf{z}$ , and  $K\left(\frac{\mathbf{x}' - \mathbf{x}}{h}\right) = h^{D}k_{h}(\mathbf{x},\mathbf{x}')$  with normalized and isotropic  $K(\mathbf{z})$ . We apply Taylor expansion on the term  $\alpha (\mathbf{x} + h\mathbf{z})p_1(\mathbf{x} + h\mathbf{z})$  around  $\mathbf{x}$ , and with the assumption that  $|h\mathbf{z}|$  is small,
$$
\alpha (\mathbf {x} + h \mathbf {z}) p _ {1} (\mathbf {x} + h \mathbf {z}) = \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) + h \mathbf {z} ^ {\top} \nabla [ \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) ] + \frac {h ^ {2}}{2} \mathbf {z} ^ {\top} \nabla \nabla [ \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) ] \mathbf {z} + O \left(h ^ {3}\right), \tag {24}
$$
using the Hessian operator  $\nabla \nabla$ . Now the integration yields the expectation with respect to  $K(\mathbf{z})$  that satisfies  $\int K(\mathbf{z}) d\mathbf{z} = 1$ ,  $\int \mathbf{z} K(\mathbf{z}) d\mathbf{z} = 0$ , and  $\int \mathbf{z} \mathbf{z}^{\top} K(\mathbf{z}) d\mathbf{z} = I$ :
$$
\mathbb {E} _ {\mathcal {D} _ {1}} \left[ \widehat {p} _ {1} (\mathbf {x}) \right] = \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) + \frac {h ^ {2}}{2} \nabla^ {2} [ \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) ] + O \left(h ^ {3}\right), \tag {25}
$$
with the Laplacian operator  $\nabla^2$ .
Along with the expansion for  $\mathbb{E}_{\mathcal{D}_2}[\widehat{p}_2(\mathbf{x})]$ , the following plug-in posterior can be perturbed by  $h$  assuming a small  $h$ :
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} [ f (\mathbf {x}) ] \rightarrow \frac {\mathbb {E} _ {\mathcal {D} _ {1}} [ \widehat {p} _ {1} (\mathbf {x}) ]}{\mathbb {E} _ {\mathcal {D} _ {1}} [ \widehat {p} _ {1} (\mathbf {x}) ] + \gamma \mathbb {E} _ {\mathcal {D} _ {2}} [ \widehat {p} _ {2} (\mathbf {x}) ]} (26) \\ = f (\mathbf {x}) + \frac {h ^ {2}}{2} \frac {\gamma p _ {1} (\mathbf {x}) p _ {2} (\mathbf {x})}{(p _ {1} (\mathbf {x}) + \gamma p _ {2} (\mathbf {x})) ^ {2}} \left(\frac {\nabla^ {2} [ \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) ]}{\alpha (\mathbf {x}) p _ {1} (\mathbf {x})} - \frac {\nabla^ {2} [ \alpha (\mathbf {x}) p _ {2} (\mathbf {x}) ]}{\alpha (\mathbf {x}) p _ {2} (\mathbf {x})}\right) + \mathcal {O} (h ^ {3}) \\ = f (\mathbf {x}) + \frac {h ^ {2}}{2} P (y = 1 | \mathbf {x}) P (y = 2 | \mathbf {x}) B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}) + \mathcal {O} \left(h ^ {3}\right), (27) \\ \end{array}
$$
giving the point-wise leading-order bias with respect to  $h$ :
$$
\operatorname {B i a s} (\mathbf {x}) = \frac {h ^ {2}}{2} P (y = 1 | \mathbf {x}) P (y = 2 | \mathbf {x}) B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}). \tag {28}
$$
Here, the  $B_{\alpha ;p_1,p_2}(\mathbf{x})$  is as follows:
$$
B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}) \equiv \frac {\nabla^ {2} [ \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) ]}{\alpha (\mathbf {x}) p _ {1} (\mathbf {x})} - \frac {\nabla^ {2} [ \alpha (\mathbf {x}) p _ {2} (\mathbf {x}) ]}{\alpha (\mathbf {x}) p _ {2} (\mathbf {x})}, \tag {29}
$$
which includes the second derivative of  $\alpha (\mathbf{x})p_1(\mathbf{x})$  and  $\alpha (\mathbf{x})p_{2}(\mathbf{x})$ . Because two classes use the same weight function  $\alpha (\mathbf{x})$ , Eq. (29) can be decomposed into two terms without the second derivative of  $\alpha (\mathbf{x})$ .
$$
B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}) = \frac {\left. \nabla^ {\top} \alpha \right| _ {\mathbf {x}}}{\alpha (\mathbf {x})} \left(\frac {\left. \nabla p _ {1} \right| _ {\mathbf {x}}}{p _ {1} (\mathbf {x})} - \frac {\left. \nabla p _ {2} \right| _ {\mathbf {x}}}{p _ {2} (\mathbf {x})}\right) + \frac {1}{2} \left(\frac {\left. \nabla^ {2} p _ {1} \right| _ {\mathbf {x}}}{p _ {1} (\mathbf {x})} - \frac {\left. \nabla^ {2} p _ {2} \right| _ {\mathbf {x}}}{p _ {2} (\mathbf {x})}\right). \tag {30}
$$
B Solution of the Calculus of Variation
For the optimization of Eq. (15) with respect to  $\alpha (\mathbf{x})$ , we first make a substitution  $\beta = \log \alpha$  and apply a calculus of variation technique for optimal  $\beta (\mathbf{x})$ . We express the objective functional with  $\int m(\mathbf{x};\beta ,\nabla \beta)d\mathbf{x}$  using
$$
m (\mathbf {x}; \beta , \nabla \beta) = \left(\nabla^ {\top} \beta | _ {\mathbf {x}} \mathbf {h} (\mathbf {x}) + g (\mathbf {x})\right) ^ {2} r (\mathbf {x}). \tag {31}
$$
With the substitution  $\vec{\beta}^{\prime} = \nabla \beta$  for notational abbreviation, we apply the Euler-Lagrange equation for the  $m(\mathbf{x};\beta ,\vec{\beta}^{\prime})$  containing both  $\beta$  and  $\vec{\beta}^{\prime}$ :
$$
\frac {\partial m (\mathbf {x} ; \beta , \vec {\beta^ {\prime}})}{\partial \beta} - \nabla_ {\mathbf {x}} \cdot \nabla_ {\vec {\beta^ {\prime}}} m (\mathbf {x}; \beta , \vec {\beta^ {\prime}}) = 0, \tag {32}
$$
where the divergence is  $\nabla_{\mathbf{x}}\cdot \nabla_{\vec{\beta}^{\prime}} = \sum_{i = 1}^{D}\frac{\partial}{\partial x_{i}}\frac{\partial}{\partial\beta_{i}^{\prime}}$  with the  $i$ -th component of  $\vec{\beta}^{\prime},\beta_{i}^{\prime}$ , and the dimensionality is  $D$ .
The first term can be calculated as  $\frac{\partial m(\mathbf{x};\beta,\vec{\beta'})}{\partial\beta} = 0$ . The first derivative of the second term is  $\nabla_{\vec{\beta}'} m(\mathbf{x};\beta,\vec{\beta'}) = r(\vec{\beta}'^{\top}\mathbf{h} + g)\mathbf{h}$ . After we substitute  $\beta(\mathbf{x})$  with  $\log \alpha(\mathbf{x})$  back, we obtain the equation for the optimal  $\alpha(\mathbf{x})$  function:
$$
\nabla \cdot \left[ r \left(\left(\nabla \log \alpha\right) ^ {\top} \mathbf {h} + g\right) \mathbf {h} \right] = 0. \tag {33}
$$
which is Eq. (16).
C Analytic Solution for Two Homoscedastic Gaussians
We consider the following two homoscedastic Gaussians
$$
p _ {1} (\mathbf {x}) = \mathcal {N} (\mathbf {x}; \mu_ {1}, \Sigma), \quad p _ {2} (\mathbf {x}) = \mathcal {N} (\mathbf {x}; \mu_ {2}, \Sigma), \tag {34}
$$
with a common covariance matrix  $\Sigma$ .