SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In order to obtain the zero divergence in Eq. (16), the divergence-free vector field can be obtained using
$$
\nabla \log \alpha \equiv \Sigma^ {- 1} \mathbf {x} + \vec {v} (\mathbf {x}) \tag {35}
$$
because the inner product of  $\nabla \log \alpha$  with  $\mathbf{h} = \nabla \log p_1 - \nabla \log p_2 = \Sigma^{-1}(\mu_1 - \mu_2)$  should yield a negative value of  $g(\mathbf{x})$ , which is  $-g(\mathbf{x}) = -\mathbf{x}^{\top}\Sigma^{-2}(\mu_2 - \mu_1) - \frac{1}{2} (\mu_1^{\top}\Sigma^{-2}\mu_1 - \mu_2^{\top}\Sigma^{-2}\mu_2)$ . The equation  $(\nabla \log \alpha)^{\top}\mathbf{h} = -g(\mathbf{x})$  gives
$$
\vec {v} (\mathbf {x}) = - \frac {1}{2} \Sigma^ {- 1} \left(\mu_ {1} + \mu_ {2}\right). \tag {36}
$$
Therefore, one possible solution for  $\nabla \log \alpha (\mathbf{x})$  is
$$
\nabla \log \alpha (\mathbf {x}) = \Sigma^ {- 1} (\mathbf {x} - \mu), \tag {37}
$$
with the mean of the two class-conditional means,  $\mu = \frac{\mu_1 + \mu_2}{2}$ . Therefore, one particular solution for  $\alpha(\mathbf{x})$  is
$$
\alpha (\mathbf {x}) = \exp \left[ \frac {1}{2} (\mathbf {x} - \mu) ^ {\top} \Sigma^ {- 1} (\mathbf {x} - \mu) \right]. \tag {38}
$$
This solution is not unique, and any  $\log \alpha$  that has a form of
$$
\log \alpha (\mathbf {x}) = \frac {1}{2} \left(\mathbf {x} - \mu\right) ^ {\top} \Sigma^ {- 1} (\mathbf {x} - \mu) + l (\mathbf {x}), \tag {39}
$$
with  $l(\mathbf{x})$  satisfying  $\nabla^{\top}l\Sigma^{-1}(\mu_1 - \mu_2) = 0$  is also the solution. One technique for finding such  $l(\mathbf{x})$  is that we pick up any differentiable seed function  $l_{0}(\mathbf{x})$  and consider its derivative  $\nabla l_{0}$  with the  $\vec{a} = \Sigma^{-1}(\mu_1 - \mu_2)$  component subtracted:  $\left(I - \frac{\vec{a}\vec{a}^{\top}}{||\vec{a}||^{2}}\right)\nabla l_{0}$ . The  $l(\mathbf{x})$  is a function that its derivative satisfies  $\nabla l = \left(I - \frac{\vec{a}\vec{a}^{\top}}{||\vec{a}||^{2}}\right)\nabla l_{0}$ .
For example, if we choose  $l_0(\mathbf{x}) = \mathbf{x}$ , then  $l(\mathbf{x})$  is a function that satisfies  $\nabla l = \left(I - \frac{\vec{a}\vec{a}^{\top}}{||\vec{a}||^2}\right)\nabla l_0 = \mathbf{1} - \frac{\vec{a}^{\top}\mathbf{I}}{||\vec{a}||^2\vec{a}}$ , and we can get  $l(\mathbf{x}) = \left(\mathbf{I} - \frac{\vec{a}^{\top}\mathbf{I}}{||\vec{a}||^2}\right)^{\top}\mathbf{x}$ . If we choose  $l_0(\mathbf{x}) = \frac{1}{2} ||\mathbf{x}||^2$ , we get  $l(\mathbf{x}) = \frac{1}{2}\mathbf{x}^{\top}\left(I - \frac{\vec{a}\vec{a}^{\top}}{||\vec{a}||^2}\right)\mathbf{x}$  after similar calculations. Now the choice of
$$
l _ {0} (\mathbf {x}) = - \frac {b}{2} \left(\mathbf {x} - \mu\right) ^ {2}, \tag {40}
$$
gives us  $l(\mathbf{x}) = -\frac{b}{2} (\mathbf{x} - \mu)^{\top}\left(I - \frac{\vec{a}\vec{a}^{\top}}{||\vec{a}||^{2}}\right)(\mathbf{x} - \mu)$ , which produces our analytic weight function in Eq. (18):
$$
\alpha (\mathbf {x}) = \exp \left(- \frac {1}{2} \left(\mathbf {x} - \mu^ {\prime}\right) ^ {\top} A \left(\mathbf {x} - \mu^ {\prime}\right)\right), \tag {41}
$$
with  $\mu' = \frac{\mu_1 + \mu_2}{2}$  and  $A = b\left(I - \frac{\Sigma^{-1}(\mu_1 - \mu_2)(\mu_1 - \mu_2)^\top\Sigma^{-1}}{||\Sigma^{-1}(\mu_1 - \mu_2)||^2}\right) - \Sigma^{-1}$ , with an arbitrary constant  $b$ .
D Posterior Prediction for Various Algorithms
Fig. 7 shows the posterior prediction results of various algorithms. For the estimation with support vector machines, [48] is used. For neural network estimation, 2-layer fully connected networks with 100 nodes for each layer were used minimizing the mean square error of the sigmoid output. Both VWKDE-MB and VWKDE-MF show superior results to other methods.
Figure 7: Posterior predictions with various algorithms for 20-dimensional Gaussians.
D.1 Kernel density estimation in high dimensions
We also note the difficulty of density estimation with KDE in high dimensional space. Fig. 8 shows a one-dimensional slice of two 20-dimensional Gaussians. The maximum density in this slice is on the order of  $10^{-8}$ . Meanwhile, the KDE with 5,000 data points per class shows densities on the order of about  $10^{-10}$  with a bandwidth of  $h = 0.8$ .
Figure 8: Underlying density functions and their KDE predictions.
The bandwidth should be chosen to be sufficiently large because no pairs are nearby in a high-dimensional space. Although the estimated density with  $h = 0.8$  is reasonably smooth, the KDE differs from the true density by several orders of magnitude with 5,000 data points. However, the patterns of relative overestimation and underestimation depicted in Fig. 1 are evident, and the proposed method can be applied even with inexact KDEs.
E Fluid Flow Interpretation of Making Bias
The change of concentration  $u(t)$  at time  $t$  due to the convection and diffusion can be written as
$$
\frac {\partial u}{\partial t} = - \mathbf {v} ^ {\top} \nabla u + D ^ {\prime} \nabla^ {2} u, \tag {42}
$$
with the direction of convection  $\mathbf{v}$  and the diffusion constant  $D^{\prime}$ . The first term represents the convection, and the second term represents the diffusion.
The bias Eq. (30) can be reformulated as
$$
\begin{array}{l} \text {E q .} (3 0) = \frac {\nabla^ {\top} \alpha}{\alpha} \left(\frac {\nabla p _ {1}}{p _ {1}} - \frac {\nabla p _ {2}}{p _ {2}}\right) + \frac {1}{2} \left(\frac {\nabla^ {2} p _ {1}}{p _ {1}} - \frac {\nabla^ {2} p _ {2}}{p _ {2}}\right). (43) \\ = \left[ \nabla^ {\top} \left(\log \alpha + \frac {1}{2} \log p _ {1}\right) \nabla \log p _ {1} + \frac {1}{2} \nabla^ {2} \log p _ {1} \right] (44) \\ - \left[ \nabla^ {\top} \left(\log \alpha + \frac {1}{2} \log p _ {2}\right) \nabla \log p _ {2} + \frac {1}{2} \nabla^ {2} \log p _ {2} \right]. (45) \\ = \frac {\partial u _ {1}}{\partial t} - \frac {\partial u _ {2}}{\partial t} (46) \\ \end{array}
$$
Eq. (44) and Eq. (45) can be understood as the two flows with concentrations  $u_{1} = \log p_{1}$  and  $u_{2} = \log p_{2}$ , respectively, and the convection direction for  $u_{1}$  is  $\mathbf{v}_1 = \nabla \left( \log \alpha + \frac{1}{2} \log p_1 \right)$ , and the direction for  $u_{2}$  is  $\mathbf{v}_2 = \nabla \left( \log \alpha + \frac{1}{2} \log p_2 \right)$ . Without the weight, the convection directions are  $\mathbf{v}_1 = \frac{1}{2} \nabla \log p_1$  and  $\mathbf{v}_2 = \frac{1}{2} \nabla \log p_2$  but with weight, they change to  $\mathbf{v}_1 = \nabla \left( \log \alpha + \frac{1}{2} \log p_1 \right)$  and  $\mathbf{v}_2 = \nabla \left( \log \alpha + \frac{1}{2} \log p_2 \right)$ .
The role of  $\alpha$  is to modify the directions of convection toward  $\nabla \log \alpha$  together, and in the example shown in Fig. 1, the change of  $u_{1}$  and  $u_{2}$  due to diffusion are negative and positive, respectively. The direction of convection can control the change of  $u_{1}$  and  $u_{2}$ , and the  $\alpha$  makes the difference between  $\frac{\partial u_1}{\partial t}$  and  $\frac{\partial u_2}{\partial t}$  as small as possible.
F Prototype Classification Interpretation of the Weighted Kernel Methods in Reproducing Kernel Hilbert Space (RKHS)
The kernel algorithm for classification can be viewed as prototype algorithms in the RKHS due to the decomposition of positive definite kernel functions [31, 30, 29, Section 1.2]:
$$
k (\mathbf {x}, \mathbf {x} ^ {\prime}) = \langle \phi (\mathbf {x}), \phi \left(\mathbf {x} ^ {\prime}\right) \rangle , \quad \phi (\mathbf {x}), \phi \left(\mathbf {x} ^ {\prime}\right) \in \mathrm {R K H S}, \tag {47}
$$
with the inner product operator  $\langle .,.\rangle$  defined in RKHS.
Given a dataset  $\{\mathbf{x}_i, y_i\}_{i=1}^N$ ,  $\mathbf{x}_i \in \mathbb{R}^D$ ,  $y_i \in \{0,1\}$ , the classification using two prototypes  $\mathbf{w}_1 = \frac{1}{N_1} \sum_{\{i:y_i=1\}} \alpha_i \phi(\mathbf{x}_i)$  and  $\mathbf{w}_0 = \frac{1}{N_0} \sum_{\{i:y_i=0\}} \alpha_i \phi(\mathbf{x}_i)$  in RKHS determines which of the prototypes has a smaller distance to the  $\phi(\mathbf{x})$  than the other. Here,  $N_1, N_0$  are the numbers of data of classes 1 and 0, respectively. The classification using KDEs with the pointwise weights can be compared with the prototype classification in RKHS using the following derivation:
$$
\begin{array}{l} y = \mathbf {1} \left(\left[ \frac {1}{N _ {1}} \sum_ {\{i; y _ {i} = 1 \}} \alpha_ {i} k (\mathbf {x} _ {i}, \mathbf {x}) - \frac {1}{N _ {0}} \sum_ {\{i; y _ {i} = 0 \}} \alpha_ {i} k (\mathbf {x} _ {i}, \mathbf {x}) \right] > \theta\right) (48) \\ = \mathbf {1} \left(\left[ \frac {1}{N _ {1}} \sum_ {\{i; y _ {i} = 1 \}} \alpha_ {i} \langle \phi (\mathbf {x} _ {i}), \phi (\mathbf {x}) \rangle - \frac {1}{N _ {0}} \sum_ {\{i; y _ {i} = 0 \}} \alpha_ {i} \langle \phi (\mathbf {x} _ {i}), \phi (\mathbf {x}) \rangle \right] > \theta\right) (49) \\ = \mathbf {I} \left(\left[ \left\langle \frac {1}{N _ {1}} \sum_ {\{i: y _ {i} = 1 \}} \alpha_ {i} \phi (\mathbf {x} _ {i}), \quad \phi (\mathbf {x}) \right\rangle - \left\langle \frac {1}{N _ {0}} \sum_ {\{i: y _ {i} = 0 \}} \alpha_ {i} \phi (\mathbf {x} _ {i}), \quad \phi (\mathbf {x}) \right\rangle \right] > \theta\right) (50) \\ = \mathbf {I} ([ \langle \mathbf {w} _ {1}, \phi (\mathbf {x}) \rangle - \langle \mathbf {w} _ {0}, \phi (\mathbf {x}) \rangle ] > \theta) (51) \\ = \mathbf {I} \left(\left[ \left\| \mathbf {w} _ {1} - \phi (\mathbf {x}) \right\| ^ {2} - \left\| \mathbf {w} _ {0}, \phi (\mathbf {x}) \right\| ^ {2} \right] > \theta^ {\prime}\right) (52) \\ \end{array}
$$
with a predetermined threshold  $\theta$ . In Eq. (52),  $\theta' = \theta - (||\mathbf{w}_1||^2 - ||\mathbf{w}_0||^2)$ .
With uniform weight  $\alpha_{i} = 1$  for all  $i = 1,\dots ,N$ , the classification is simply the comparison of two KDEs  $\widehat{p}_1 = \frac{1}{N_1}\sum_{\{i;y_i = 1\}}k(\mathbf{x}_i,\mathbf{x})$  and  $\widehat{p}_0 = \frac{1}{N_0}\sum_{\{i;y_i = 0\}}k(\mathbf{x}_i,\mathbf{x})$ , which correspond to the prototype classification using two empirical means  $\mathbf{w}_1 = \frac{1}{N_1}\sum_{\{i;y_i = 1\}}\phi (\mathbf{x}_i)$  and  $\mathbf{w}_0 = \frac{1}{N_0}\sum_{\{i;y_i = 0\}}\phi (\mathbf{x}_i)$  in RKHS. Originating from this correspondence, one suggestion of the unification for the KDE and RKHS is presented in [30]. The explanations about the prototypes for various kernelized algorithms can be found in [49]. The prototypes of SVMs are known to be the closest two points within the convex hull of different classes [50].
Despite all these discussions, it is clear that the modification of the densities using a weight function will not improve the density estimation performance from the perspective of KDE. Despite the poor density estimation
performance, the modification improves the classification or information-theoretic measure estimation for the KDE plug-in algorithms, but not necessarily the KDE itself. The improvement is partly supported by the prototype models in RKHS.
G Least Square Approach for Binary Classification
Reducing the bias of the posterior equation in Eq. (3) corresponds to the least square of the prediction error. The optimal square error is achieved with the Bayes classifier, which classifies a datum according to the posterior probability. The posterior probability of  $\mathbf{x}_0$  being generated from  $p_1(\mathbf{x})$  can be written as:
$$
\begin{array}{l} P (y = 1 \mid \mathbf {x} _ {0}) = \frac {p _ {1} (\mathbf {x} _ {0}) p (y = 1)}{p _ {0} (\mathbf {x} _ {0}) p (y = 0) + p _ {1} (\mathbf {x} _ {0}) p (y = 1)} (53) \\ = \frac {p _ {1} \left(\mathbf {x} _ {0}\right)}{\gamma p _ {0} \left(\mathbf {x} _ {0}\right) + p _ {1} \left(\mathbf {x} _ {0}\right)}, (54) \\ \end{array}
$$
where  $\gamma = p(y = 0) / p(y = 1)$ . The least square error with
$$
L = \int (f (\mathbf {x}) - y) ^ {2} p (\mathbf {x}, y) d y d \mathbf {x}, \tag {55}
$$
is achieved with the following prediction function
$$
f (\mathbf {x}) = \mathbb {E} [ y = 1 | \mathbf {x} ] = P (y = 1 | \mathbf {x}). \tag {56}
$$
An accurate estimation of posterior is essential for successful classification. We construct a classifier based on the KDE density estimates  $\widehat{p}_0(\mathbf{x}),\widehat{p}_1(\mathbf{x})$
$$
\begin{array}{l} f (\mathbf {x}) = \frac {\widehat {p} _ {1} (\mathbf {x})}{\gamma \widehat {p} _ {0} (\mathbf {x}) + \widehat {p} _ {1} (\mathbf {x})} (57) \\ = \frac {1}{1 + \gamma \left(\widehat {p} _ {0} (\mathbf {x}) / \widehat {p} _ {1} (\mathbf {x})\right)}, (58) \\ \end{array}
$$
and consider the deviation of  $f(\mathbf{x})$  from the true  $\mathbb{E}[y = 1|\mathbf{x}]$ .
H Details on Optical Surface Inspection Experiments
We use a widely used public surface inspection dataset provided by DAGM<sup>3</sup> for experiments. The dataset contains six distinct textile surface types and associated defect types. There are 1,150 images per class, half of which is for training and the remaining is for testing. Approximately  $13\%$  of total images are defective, and for each defective image, a masking image which roughly encloses the defective region are provided. The dataset is originally proposed for a supervised setting.
We extract 900 patches of size  $32 \times 32$  from each image, using a sliding window with step size 16. In each patch, we apply Gaussian smoothing and Scharr kernel to obtain a gradient distribution which can capture the texture information. To encode the gradient distribution as a feature vector, we compute its mean, standard deviation, skewness, and kurtosis. As a result, a surface image is transformed into a set of 900 four-dimensional vectors, or a  $900 \times 4$  matrix. Feature vectors are standardized and whitened by aggregating all the patches from the same surface type.
VWKDE can be time-consuming as a large number of KL divergences need to be computed. Therefore, we take a two-pass approach when applying VWKDE. Given a query image, we first apply KDE-based KL divergence estimator to obtain rough estimates of KL divergences. Then, we take  $k$  images with the lowest KL divergences and apply VWKDE-based KL divergence estimator to the  $k$  images to finally select the image with the lowest KL divergence. This method enables us to have the best of both worlds, the speed of KDE and the accuracy of VWKDE.
The optimal bandwidth for bias reduction methods such as Ensemble [36], vonMises [37], and VWKDE is usually larger than other methods. We use the bandwidth with maximum leave-one-out log-likelihood of KDE for other methods but in these three methods, we used the heuristic rule of bandwidth selection using the maximum log-likelihood bandwidth for only  $25\%$  of randomly selected data.
A convolutional neural network (CNN) which takes a  $32 \times 32$  patch as an input and predicts whether the patch is defective is trained. For training, we label patches with  $75\%$  overlap to the defect mask as defective and patches
<sup>3</sup>Deutsche Arbeitsgemeinschaft für Mustererkennung (The German Association for Pattern Recognition). Data access: https://hci.iwr.uni-heidelberg.de/node/3616
without any overlap to the defect mask as normal. Patches do not belong to either class are discarded. Due to class imbalance, normal patches are undersamples to yield defect to normal ratio of 1:4. CNN is trained for each surface type separately. The structure of our CNN is Conv(20)-Conv(20)-MaxPool-Conv(20)-Conv(20)-MaxPool-FC(20)-DropOut-FC(1), where Conv is a  $3 \times 3$  convolution layer, MaxPool is a  $2 \times$  max pooling layer, FC is a fully connected layer, and DropOut is a drop out operation with probability 0.5. We use binary cross entropy loss for objective function and ADAM for optimization.
In unsupervised defect localization, we threshold log probability density ratio (LPDR) estimate to obtain detection results. We threshold LPDR estimates dynamically at  $90\%$  of maximum LPDR observed in the image. Then, we use its KL divergence estimate as a confidence score for the detection. For a CNN, we use the output probability for a patch as a detection score, and set a threshold to 0.9, and the maximum probability of defect among the patches in an image is used as a confidence score. Note that, in this experiment, we generate one detection per an image because DAGM dataset is constrained to have as most one defect per an image. However, this condition can be relaxed in future work with other dataset.
Intersection-over-union (IOU) is computed between a detection and a true defect mask. A detection with IOU larger than 0.1 considered as a correct detection. This threshold is lower than a typical threshold in object detection (0.5), because the defect mask is weakly labelled and usually larger than a precise defect region. Using a confidence score assigned for a detection, we compute average precision as in PASCAL VOC challenge, then take average over surface types.