SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Korea Institute for Advanced Study swyoon@kias.re.kr
Frank C. Park
Seoul National University / Saige Research fcp@snu.ac.kr
Gunsu Yun
gunsu@postech.ac.kr
Iljung Kim
iljung0810@hanyang.ac.kr
Yung-Kyun Noh
Hanyang University / Korea Institute for Advanced Study
Kernel density estimation (KDE) is integral to a range of generative and discriminative tasks in machine learning. Drawing upon tools from the multidimensional calculus of variations, we derive an optimal weight function that reduces bias in standard kernel density estimates for density ratios, leading to improved estimates of prediction posteriors and information-theoretic measures. In the process, we shed light on some fundamental aspects of density estimation, particularly from the perspective of algorithms that employ KDEs as their main building blocks.
One fundamental component for building many applications in machine learning is a correctly estimated density for prediction and estimation tasks, with examples ranging from classification [1, 2], anomaly detection [3], and clustering [4] to the generalization of value functions [5], policy evaluation [6], and estimation of various information-theoretic measures [7, 8, 9]. Nonparametric density estimators, such as the nearest neighbor density estimator or kernel density estimators (KDEs), have been used as substitutes for the probability density component within the equation of the posterior probability, or the density-ratio equation, with theoretical guarantees derived in part from the properties of the density estimators used [10, 11].
Given a specific task which uses the ratio between two densities,  $p_1(\mathbf{x})$  and  $p_2(\mathbf{x})$  at a point  $\mathbf{x} \in \mathbb{R}^D$ , we consider the ratio handled by the ratio of their corresponding two KDEs,  $\widehat{p}_1(\mathbf{x})$  and  $\widehat{p}_2(\mathbf{x})$ :
$$
\frac {\widehat {p} _ {1} (\mathbf {x})}{\widehat {p} _ {2} (\mathbf {x})} \xrightarrow [ \text {E s t i m a t e} ]{} \frac {p _ {1} (\mathbf {x})}{p _ {2} (\mathbf {x})}. \tag {1}
$$
Each estimator is a KDE which counts the effective number of data within a small neighborhood of  $\mathbf{x}$  by averaging the kernels. The biases produced by the KDEs in the nominator and denominator [12, Theorem 6.28] are combined to produce a single bias of the ratio, as demonstrated in Fig. 1. For example, the ratio  $\frac{p_1(\mathbf{x})}{p_2(\mathbf{x})}$  at  $\mathbf{x}_0$  in Fig. 1(a) is clearly expected to be underestimated because of the dual effects in Fig. 1(b): the underestimation of the nominator  $p_1(\mathbf{x}_0)$  and the overestimation of the denominator  $p_2(\mathbf{x}_0)$ . The underestimation is attributed to the concavity of  $p_1(\mathbf{x})$  around  $\mathbf{x}_0$  which leads to a reduced number of data being generated compared to a uniform density. The underestimation of  $p_2(\mathbf{x})$  can be explained similarly. The second derivative—Laplacian—that creates
(b)
Figure 1: Estimation of the density ratio  $\frac{p_1(\mathbf{x}_0)}{p_2(\mathbf{x}_0)}$  and bias correction using the weight function  $\alpha(\mathbf{x})$ . (a) Two density functions,  $p_1(\mathbf{x})$  and  $p_2(\mathbf{x})$ , and the point of interest  $\mathbf{x}_0$  for ratio estimation. Two regions delineated by dashed lines are magnified in (b). (b) The concavity and convexity of the density functions around  $\mathbf{x}_0$  and their KDEs. Concave density  $p_1(\mathbf{x})$  generates less data than the uniform density of  $p_1(\mathbf{x}_0)$  around  $\mathbf{x}_0$  resulting in an underestimation. For a similar reason, convex density  $p_2(\mathbf{x})$  results in an overestimation. The two biases are combined into an underestimation of the ratio. (c) KDE augmented with a nonsymmetric weight function  $\alpha(\mathbf{x})$  can alleviate this bias by transforming the bias of  $\widehat{p_2}(\mathbf{x})$  to an appropriate underestimation from an overestimation.
the concavity or convexity of the underlying density is a dominant factor that causes the bias in this example.
The Laplacian of density has been used to produce equations in various bias reduction methods, such as bias correction [13, 14, 15] and smoothing of the data space [16, 17]. However, the example in Fig. 1 motivates a novel, position-dependent weight function  $\alpha(\mathbf{x})$  to be multiplied with kernels in order to alleviate the bias. For example, to alleviate the overestimation of  $p_2(\mathbf{x}_0)$ , we can consider the  $\alpha(\mathbf{x})$  shown in Fig. 1(c) that assigns more weight on the kernels associated with data located to the left of  $\mathbf{x}_0$ , which is a low-density region. When the weighted kernels are averaged, the overestimation of  $\widehat{p_2}(\mathbf{x}_0)$  can be mitigated or potentially even underestimated. Meanwhile, the bias of  $\widehat{p_1}(\mathbf{x}_0)$  remains unchanged after applying the weights since  $p_1(\mathbf{x})$  is symmetric around  $\mathbf{x}_0$ . This allows the reversed underestimation of  $p_2(\mathbf{x}_0)$  from the initial overestimation to effectively offset or counterbalance the underestimation of  $p_1(\mathbf{x}_0)$  within the ratio.
We derive the  $\alpha (\mathbf{x})$  function that performs this alleviation over the entire data space. The appropriate information for  $\alpha (\mathbf{x})$  comes from the geometry of the underlying densities. The aforementioned principle of bias correction leads to novel, model-based and model-free approaches. Based on the assumption of the underlying densities, we learn the parameters for the densities' first and second derivatives and then variationally adjust  $\alpha (\mathbf{x})$  for the estimator to create the variationally weighted KDE (VWKDE). We note that the model for those densities and their derivatives need not be exact because the goal is not to achieve precise density estimation but rather to accurately capture the well-behaved  $\alpha (\mathbf{x})$  for the KDE ratios.
Applications include classification with posterior information and information-theoretic measure estimates using density-ratio estimation. Calibration of posteriors [18] has been of interest to many researchers, in part, to provide a ratio of correctness of the prediction. Plug-in estimators of information-theoretic measures, such as the Kullback-Leibler (K-L) divergence, can also be advantageous. For K-L divergence estimation, similar previous formulations for the variational approach have included optimizing a functional bound with respect to the function constrained within the reproducing kernel Hilber space (RKHS) [19, 20, 21]. These and other methods that use weighted kernels (e.g., [22, 23, 24]) take advantage of the flexibility offered by universal approximator functions in the form of linear combinations of kernels. These methods, however, do not adequately explain why the weight optimization leads to an improved performance. Based on a derivation of how bias is produced, we provide an explicit modification of weight for standard kernel density estimates, with details of how the estimation is improved.
The remainder of the paper is organized as follows. In Section 2, we introduce the variational formulation for the posterior estimator and explain how to minimize the bias. Section 3 shows how a weight function can be derived using the calculus of variations, which is then extended to general density-ratio and K-L divergence estimation in Section 4. Experimental results are presented in Section 5. Finally, we conclude with discussion in Section 6.
2 Variationally Weighted KDE for Ratio Estimation
KDE  $\widehat{p}(\mathbf{x}) = \frac{1}{N}\sum_{j=1}^{N}k_h(\mathbf{x},\mathbf{x}_j)$  is conventionally the average of kernels. The average roughly represents the count of data within a small region around  $\mathbf{x}$ , the size of which is determined by a bandwidth parameter  $h$ . The amount of convexity and concavity inside the region determines the bias of estimation, as depicted in Fig. 1(a),(b).
2.1 Plug-in estimator of posterior with weight
We consider a weighted KDE as a plug-in component adjusted for reliable ratio estimation using a positive and twice differentiable weight function:  $\alpha (\mathbf{x})\in \mathcal{A}$  with  $\mathcal{A} = \{\alpha :\mathbb{R}^D\to \mathbb{R}^+ | \alpha \in C^2 (\mathbb{R}^D)\}$ . For two given sets of i.i.d. samples,  $\mathcal{D}_1 = \{\mathbf{x}_i\}_{i = 1}^{N_1}\sim p_1(\mathbf{x})$  for class  $y = 1$  and  $\mathcal{D}_2 = \{\mathbf{x}_i\}_{i = N_1 + 1}^{N_1 + N_2}\sim p_2(\mathbf{x})$  for class  $y = 2$ , we use the following weighted KDE formulation:
$$
\widehat {p _ {1}} (\mathbf {x}) = \frac {1}{N _ {1}} \sum_ {j = 1} ^ {N _ {1}} \alpha \left(\mathbf {x} _ {j}\right) k _ {h} \left(\mathbf {x}, \mathbf {x} _ {j}\right), \quad \widehat {p _ {2}} (\mathbf {x}) = \frac {1}{N _ {2}} \sum_ {j = N _ {1} + 1} ^ {N _ {1} + N _ {2}} \alpha \left(\mathbf {x} _ {j}\right) k _ {h} \left(\mathbf {x}, \mathbf {x} _ {j}\right). \tag {2}
$$
Here, the two estimators use a single  $\alpha (\mathbf{x})$ . The kernel function  $k_{h}(\mathbf{x},\mathbf{x}^{\prime})$  is a positive, symmetric, normalized, isotropic, and translation invariant function with bandwidth  $h$ . The weight function  $\alpha (\mathbf{x})$  informs the region that should be emphasized, and a constant function  $\alpha (\mathbf{x}) = c$  reproduces the ratios from the conventional KDE. We let their plug-in posterior estimator be  $f(\mathbf{x})$ , and the function can be calculated using
$$
f (\mathbf {x}) = \widehat {P} (y = 1 | \mathbf {x}) = \frac {\widehat {p _ {1}} (\mathbf {x})}{\widehat {p _ {1}} (\mathbf {x}) + \gamma \widehat {p _ {2}} (\mathbf {x})}. \tag {3}
$$
with a constant  $\gamma \in \mathbb{R}$  determined by the class-priors.
2.2 Bias of the posterior estimator
We are interested in reducing the expectation of the bias square:
$$
\mathbb {E} [ \operatorname {B i a s} (\mathbf {x}) ^ {2} ] = \int (f (\mathbf {x}) - \mathbb {E} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} [ f (\mathbf {x}) ]) ^ {2} p (\mathbf {x}) d \mathbf {x}. \tag {4}
$$
The problem of finding the optimal weight function can be reformulated as the following equation in Proposition 1.
Proposition 1. With small  $h$ , the expectation of the bias square in Eq. (4) is minimized by any  $\alpha(\mathbf{x})$  that eliminates the following function
$$
B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}) = \left(\nabla \log \alpha | _ {\mathbf {x}}\right) ^ {\top} \mathbf {h} (\mathbf {x}) + g (\mathbf {x}), \tag {5}
$$
at every point  $\mathbf{x}$ . Here,  $\mathbf{h}(\mathbf{x}) = \left(\frac{\nabla p_1}{p_1} -\frac{\nabla p_2}{p_2}\right)$  and  $g(\mathbf{x}) = \frac{1}{2}\left(\frac{\nabla^2p_1}{p_1} -\frac{\nabla^2p_2}{p_2}\right)$  with gradient and Laplacian operators,  $\nabla$  and  $\nabla^2$ , respectively. All derivatives are with respect to  $\mathbf{x}$ .
The derivation of Eq. (5) begins with the expectation of the weighted KDE:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {1}} \left[ \widehat {p} _ {1} (\mathbf {x}) \right] = \mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p _ {1} (\mathbf {x})} \left[ \alpha \left(\mathbf {x} ^ {\prime}\right) k _ {h} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) \right] = \int \alpha \left(\mathbf {x} ^ {\prime}\right) p _ {1} \left(\mathbf {x} ^ {\prime}\right) k _ {h} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) d \mathbf {x} ^ {\prime} (6) \\ = \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) + \frac {h ^ {2}}{2} \nabla^ {2} [ \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) ] + O \left(h ^ {3}\right). (7) \\ \end{array}
$$
Along with the similar expansion for  $\mathbb{E}_{\mathcal{D}_2}[\widehat{p}_2(\mathbf{x})]$ , the following plug-in posterior can be perturbed with small  $h$ :
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {1}, \mathcal {D} _ {2}} [ f (\mathbf {x}) ] \rightarrow \frac {\mathbb {E} _ {\mathcal {D} _ {1}} [ \widehat {p} _ {1} (\mathbf {x}) ]}{\mathbb {E} _ {\mathcal {D} _ {1}} [ \widehat {p} _ {1} (\mathbf {x}) ] + \gamma \mathbb {E} _ {\mathcal {D} _ {2}} [ \widehat {p} _ {2} (\mathbf {x}) ]} (8) \\ = f (\mathbf {x}) + \frac {h ^ {2}}{2} \frac {\gamma p _ {1} (\mathbf {x}) p _ {2} (\mathbf {x})}{(p _ {1} (\mathbf {x}) + \gamma p _ {2} (\mathbf {x})) ^ {2}} \left(\frac {\nabla^ {2} [ \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) ]}{\alpha (\mathbf {x}) p _ {1} (\mathbf {x})} - \frac {\nabla^ {2} [ \alpha (\mathbf {x}) p _ {2} (\mathbf {x}) ]}{\alpha (\mathbf {x}) p _ {2} (\mathbf {x})}\right) + \mathcal {O} (h ^ {3}) \\ = f (\mathbf {x}) + \frac {h ^ {2}}{2} P (y = 1 | \mathbf {x}) P (y = 2 | \mathbf {x}) B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}) + \mathcal {O} \left(h ^ {3}\right), (9) \\ \end{array}
$$
(b)
Figure 2: Posterior estimates with KDEs and VEKDEs for two 20-dimensional homoscedastic Gaussian densities. (a) Bias corrected by VWKDE. (b) Bias and variance of posterior estimates depending on the bandwidth for standard KDE and for (c) VWKDE.
with the substitution
$$
B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}) \equiv \frac {\nabla^ {2} [ \alpha (\mathbf {x}) p _ {1} (\mathbf {x}) ]}{\alpha (\mathbf {x}) p _ {1} (\mathbf {x})} - \frac {\nabla^ {2} [ \alpha (\mathbf {x}) p _ {2} (\mathbf {x}) ]}{\alpha (\mathbf {x}) p _ {2} (\mathbf {x})}. \tag {10}
$$
The point-wise leading-order bias can be written as
$$
\operatorname {B i a s} (\mathbf {x}) = \frac {h ^ {2}}{2} P (y = 1 | \mathbf {x}) P (y = 2 | \mathbf {x}) B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}). \tag {11}
$$
Here, the  $B_{\alpha ;p_1,p_2}(\mathbf{x})$  includes the second derivative of  $\alpha (\mathbf{x})p_{1}(\mathbf{x})$  and  $\alpha (\mathbf{x})p_2(\mathbf{x})$ . Because two classes share the weight function  $\alpha (\mathbf{x})$ , Eq. (10) can be simplified into two terms without the second derivative of  $\alpha (\mathbf{x})$  as
$$
B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}) = \frac {\nabla^ {\top} \alpha | _ {\mathbf {x}}}{\alpha (\mathbf {x})} \left(\frac {\nabla p _ {1} | _ {\mathbf {x}}}{p _ {1} (\mathbf {x})} - \frac {\nabla p _ {2} | _ {\mathbf {x}}}{p _ {2} (\mathbf {x})}\right) + \frac {1}{2} \left(\frac {\nabla^ {2} p _ {1} | _ {\mathbf {x}}}{p _ {1} (\mathbf {x})} - \frac {\nabla^ {2} p _ {2} | _ {\mathbf {x}}}{p _ {2} (\mathbf {x})}\right), \tag {12}
$$
which leads to Eq. (5) in the Proposition. The detailed derivation in this section can be found in Appendix A.
2.3 Plug-in estimator of K-L divergence with weight
In order to estimate  $KL(p_{1}||p_{2}) = \mathbb{E}_{\mathbf{x}\sim p_{1}}\left[\log \frac{p_{1}(\mathbf{x})}{p_{2}(\mathbf{x})}\right]$ , we consider the following plug-in estimator:
$$
\widehat {K L} \left(p _ {1} \| p _ {2}\right) = \frac {1}{N _ {1}} \sum_ {i = 1} ^ {N _ {1}} \log \frac {\widehat {p} _ {1} \left(\mathbf {x} _ {i}\right)}{\widehat {p} _ {2} \left(\mathbf {x} _ {i}\right)}, \tag {13}
$$
using  $\mathbf{x}_i$  in  $\mathbf{x}_i\in \mathcal{D}_1$  for Monte Carlo averaging. When we calculate  $\widehat{p_1}$  at  $\mathbf{x}_i$ , we exclude  $\mathbf{x}_i$  from the KDE samples. We use  $\widehat{p_1} (\mathbf{x}_i) = \frac{1}{N_1 - 1}\sum_{j = 1}^{N_1}\alpha (\mathbf{x}_j)k_h(\mathbf{x}_i,\mathbf{x}_j)\mathbf{I}_{(i\neq j)}$  with the indicator function  $\mathbf{I}_{(\mathcal{I})}$ , which is 1 if  $\mathcal{I}$  is true and 0 otherwise.
Proposition 2. With small  $h$ , the expectation of the bias square is minimized by finding any  $\alpha(\mathbf{x})$  that eliminates the same function as  $B_{\alpha; p_1, p_2}(\mathbf{x})$  in Eq. (5) in Proposition 1 at each point  $\mathbf{x}$ .
In the task of estimating the K-L divergence, Proposition 2 claims that we obtain the equivalent  $B_{\alpha ;p_1,p_2}(\mathbf{x})$  to Eq. (5) during the derivation of bias. The pointwise bias in the K-L divergence estimator can be written as
$$
\operatorname {B i a s} (\mathbf {x}) = \frac {h ^ {2}}{2} \frac {p _ {1} (\mathbf {x})}{p _ {2} (\mathbf {x})} B _ {\alpha ; p _ {1}, p _ {2}} (\mathbf {x}), \tag {14}
$$
with  $B_{\alpha ;p_1,p_2}(\mathbf{x})$  equivalent to Eq. (5).
3 Variational Formulation and Implementation
Now we consider the  $\alpha (\mathbf{x})$  that minimizes the mean square error for the estimation:
$$
\min  _ {\alpha (\mathbf {x}) \in \mathcal {A}} \int \left(\left(\nabla \log \alpha | _ {\mathbf {x}}\right) ^ {\top} \mathbf {h} (\mathbf {x}) + g (\mathbf {x})\right) ^ {2} r (\mathbf {x}) d \mathbf {x}, \tag {15}
$$
with  $\mathbf{h}(\mathbf{x}) = \left(\frac{\nabla p_1}{p_1} -\frac{\nabla p_2}{p_2}\right),g(\mathbf{x}) = \frac{1}{2}\left(\frac{\nabla^2p_1}{p_1} -\frac{\nabla^2p_2}{p_2}\right)$ . The  $r(\mathbf{x})$  function depends on the problem:  $r(\mathbf{x}) = P(y = 1|\mathbf{x})^{2}P(y = 2|\mathbf{x})^{2}p(\mathbf{x})$  for posterior estimation and  $r(\mathbf{x}) = \left(\frac{p_1(\mathbf{x})}{p_2(\mathbf{x})}\right)^2 p(\mathbf{x})$  for K-L divergence estimation, with the total density,  $p(\mathbf{x}) = (p_{1}(\mathbf{x}) + \gamma p_{2}(\mathbf{x}))P(y = 1)$ . The calculus of variation for optimizing the functional in Eq. (15) provides an equation that the optimal  $\alpha (\mathbf{x})$  should satisfy:
$$
\nabla \cdot \left[ r \left(\left(\nabla \log \alpha\right) ^ {\top} \mathbf {h} + g\right) \mathbf {h} \right] = 0. \tag {16}
$$
The detailed derivation of this equation can be found in Appendix B.
3.1 Gaussian density and closed-form solution for  $\alpha (\mathbf{x})$
A simple analytic solution for this optimal condition can be obtained for two homoscedastic Gaussian density functions. The density functions have two different means,  $\mu_1 \in \mathbb{R}^D$  and  $\mu_2 \in \mathbb{R}^D$ , but share a single covariance matrix  $\Sigma \in \mathbb{R}^{D \times D}$ :
$$
p _ {1} (\mathbf {x}) = \mathcal {N} (\mathbf {x}; \mu_ {1}, \Sigma), \quad p _ {2} (\mathbf {x}) = \mathcal {N} (\mathbf {x}; \mu_ {2}, \Sigma). \tag {17}
$$
One solution for this homoscedastic setting can be obtained as
$$
\alpha (\mathbf {x}) = \exp \left(- \frac {1}{2} \left(\mathbf {x} - \mu^ {\prime}\right) ^ {\top} A \left(\mathbf {x} - \mu^ {\prime}\right)\right), \tag {18}
$$
with  $\mu' = \frac{\mu_1 + \mu_2}{2}$  and  $A = b\left(I - \frac{\Sigma^{-1}(\mu_1 - \mu_2)(\mu_1 - \mu_2)^\top\Sigma^{-1}}{||\Sigma^{-1}(\mu_1 - \mu_2)||^2}\right) - \Sigma^{-1}$  using an arbitrary constant  $b$ . Due to the choice of  $b$ , the solution is not unique. All the solutions produce a zero bias. Its detailed derivation can be found in the Appendix C. The reduction of the bias using Eq. (18) with estimated parameters is shown in Fig. 2.
3.2 Implementation
In this work, we propose a model-free method and a mode-based method. The model-free approach uses the information of  $\widehat{\nabla}\log p_1(\mathbf{x})$  and  $\widehat{\nabla}\log p_2(\mathbf{x})$  estimated by a score matching neural network [25]. We obtain the second derivative,  $\widehat{\nabla}^2\log p$ , by the automatic differentiation of the neural network for the scores. We then obtain  $\frac{\widehat{\nabla^2p}}{p}$  using  $\frac{\widehat{\nabla^2p}}{p} = \widehat{\nabla}^2\log p - \widehat{\nabla}^\top \log p\widehat{\nabla}\log p$ . With the outputs of the neural networks for  $\widehat{\nabla}\log p$  and  $\frac{\widehat{\nabla^2p}}{p}$ , we train a new network for the function  $\alpha (\mathbf{x};\theta)$  with neural network parameters  $\theta$ .
On the other hand, the model-based approach uses a coarse Gaussian model for class-conditional densities. The Gaussian functions for each class have their estimated parameters  $\widehat{\mu}_1,\widehat{\mu}_2\in \mathbb{R}^D$  and  $\widehat{\Sigma}_1,\widehat{\Sigma}_2\in \mathbb{R}^{D\times D}$ . We use the score information from these parameters:  $\widehat{\nabla}\log p_1(\mathbf{x}) = \widehat{\Sigma}_1^{-1}(\mathbf{x} - \widehat{\mu}_1)$  and  $\widehat{\nabla}\log p_2(\mathbf{x}) = \widehat{\Sigma}_2^{-1}(\mathbf{x} - \widehat{\mu_2})$ . In the model-based approach, we let the log of  $\alpha (\mathbf{x})$  be a function within the RKHS with basis kernels  $\kappa_{\sigma}(\cdot ,\cdot)$  with kernel parameter  $\sigma$ . We let  $\log \alpha (\mathbf{x};\theta) = \sum_{i = 1}^{N_1 + N_2}\theta_i\kappa_\sigma (\mathbf{x},\mathbf{x}_i)$  with parameters  $\theta = \{\theta_{1},\dots ,\theta_{N_{1} + N_{2}}\}$  for optimization.
The weight function  $\alpha (\mathbf{x})$  is obtained by optimizing the following objective function with  $N_{1}$  number of data from  $p_1(\mathbf{x})$  and  $N_{2}$  number of data from  $p_2(\mathbf{x})$ :
$$
L (\theta) = \sum_ {i = 1} ^ {N _ {1} + N _ {2}} \frac {1}{2} \left(\nabla^ {\top} \log \alpha \left(\mathbf {x} _ {i}; \theta\right) \widehat {\mathbf {h}} \left(\mathbf {x} _ {i}\right)\right) ^ {2} + \nabla^ {\top} \log \alpha \left(\mathbf {x} _ {i}; \theta\right) \widehat {\mathbf {h}} \left(\mathbf {x} _ {i}\right) \widehat {g} \left(\mathbf {x} _ {i}\right), \tag {19}
$$
with the substitutions  $\widehat{\mathbf{h}}(\mathbf{x}) = \widehat{\nabla} \log p_1(\mathbf{x}) - \widehat{\nabla} \log p_2(\mathbf{x})$  and  $\widehat{g}(\mathbf{x}) = \frac{1}{2} \left( \frac{\widehat{\nabla^2 p_1}(\mathbf{x})}{p_1(\mathbf{x})} - \frac{\widehat{\nabla^2 p_2}(\mathbf{x})}{p_2(\mathbf{x})} \right)$ .
In the model-based method, an addition of  $\ell_2$ -regularizer,  $\lambda \sum_{i=1}^{N_1 + N_2} \theta_i^2$ , with a small positive constant  $\lambda$  makes the optimization (19) quadratic. When there are fewer than 3,000 samples, we use all of them as basis points. Otherwise, we randomly sample 3,000 points from  $\{\mathbf{x}_i\}_{i=1}^{N_1 + N_2}$ .
A brief summary of the implementation process is shown in Algorithms 1 and 2.
$^{1}$ Code is available at https://github.com/swyoon/variationally-weighted-kernel-density-estimation
Algorithm 1 Model-free
Input:  $\mathbf{x}$ ,  $\{\mathbf{x}_i\}_{i=1}^{N_1} \sim p_1$ ,  $\{\mathbf{x}_i\}_{i=N_1+1}^{N_1+N_2} \sim p_2$
Output: Ratio  $\widehat{R} (\mathbf{x})$ $(= \widehat{p}_1 / \widehat{p}_2(\mathbf{x}))$
1. Estimate  $\widehat{\nabla}\log p_1$  using  $\{\mathbf{x}_i\}_{i = 1}^{N_1}\sim p_1$
2. Estimate  $\widehat{\nabla}\log p_2$  using  $\{\mathbf{x}_i\}_{i = N_1 + 1}^{N_1 + N_2}\sim p_2$
3. Obtain  $\alpha (\mathbf{x})$  that minimizes Eq. (19)
3.  $\widehat{R} (\mathbf{x}) = \frac{\sum_{i = 1}^{N_1}\alpha(\mathbf{x}_i)k_h(\mathbf{x},\mathbf{x}_i)}{\sum_{i = N_1 + 1}^{N_1 + N_2}\alpha(\mathbf{x}_i)k_h(\mathbf{x},\mathbf{x}_i)}$
Return  $\widehat{R} (\mathbf{x})$
Algorithm 2 Model-based
1. Estimate  $\widehat{\mu}_1, \widehat{\Sigma}_1$  using  $\{\mathbf{x}_i\}_{i=1}^{N_1} \sim p_1$ .
2. Estimate  $\widehat{\mu}_2, \widehat{\Sigma}_2$  using  $\{\mathbf{x}_i\}_{i = N_1 + 1}^{N_1 + N_2} \sim p_2$ .
3. Use  $\widehat{\nabla}\log p_c|_{\mathbf{x}} = \widehat{\Sigma}_c^{-1}(\mathbf{x} - \widehat{\mu}_c)$  to obtain  $\alpha (\mathbf{x})$  that minimizes Eq. (19)
3.  $\widehat{R} (\mathbf{x}) = \frac{\sum_{i = 1}^{N_1}\alpha(\mathbf{x}_i)k_h(\mathbf{x},\mathbf{x}_i)}{\sum_{i = N_1 + 1}^{N_1 + N_2}\alpha(\mathbf{x}_i)k_h(\mathbf{x},\mathbf{x}_i)}$
4 Interpretation of  $\alpha (\mathbf{x})$  for Bias Reduction
The process of finding  $\alpha (\mathbf{x})$  that minimizes the square of  $B_{\alpha ;p_1,p_2}(\mathbf{x})$  in Eq. (5) can be understood from various perspectives through reformulation.
4.1 Cancellation of the bias
The second term  $\frac{1}{2}\left(\frac{\nabla^2p_1}{p_1} -\frac{\nabla^2p_2}{p_2}\right)$  in Eq. (5) repeatedly appears in the bias of nonparametric processes using discrete labels [26]. In our derivation, the term is achieved with a constant  $\alpha (\mathbf{x})$  or with no weight function. The role of the weight function is to control the first term  $\frac{\nabla^{\top}\alpha}{\alpha}\left(\frac{\nabla p_1}{p_1} -\frac{\nabla p_2}{p_2}\right)$  based on the gradient information in order to let the first term cancel the second.
4.2 Cancellation of flow in a mechanical system