SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pum Jun Kim<sup>1</sup> Yoojin Jang<sup>1</sup> Jisu Kim<sup>2,3,4</sup> Jaejun Yoo<sup>1</sup>
<sup>1</sup> Ulsan National Institute of Science and Technology
$^{2}$ Seoul National University  $^{3}$ Inria  $^{4}$ Paris-Saclay University
{pumjun.kim,softjin,jaejun.yoo}@unist.ac.kr
jkim82133@snu.ac.kr
We propose a robust and reliable evaluation metric for generative models called Topological Precision and Recall (TopP&R, pronounced "topper"), which systematically estimates supports by retaining only topologically and statistically significant features with a certain level of confidence. Existing metrics, such as Inception Score (IS), Fréchet Inception Distance (FID), and various Precision and Recall (P&R) variants, rely heavily on support estimates derived from sample features. However, the reliability of these estimates has been overlooked, even though the quality of the evaluation hinges entirely on their accuracy. In this paper, we demonstrate that current methods not only fail to accurately assess sample quality when support estimation is unreliable, but also yield inconsistent results. In contrast, TopP&R reliably evaluates the sample quality and ensures statistical consistency in its results. Our theoretical and experimental findings reveal that TopP&R provides a robust evaluation, accurately capturing the true trend of change in samples, even in the presence of outliers and non-independent and identically distributed (Non-IID) perturbations where other methods result in inaccurate support estimations. To our knowledge, TopP&R is the first evaluation metric specifically focused on the robust estimation of supports, offering statistical consistency under noise conditions.
In keeping with the remarkable improvements of deep generative models [1, 2, 3, 4, 5, 6, 7, 8, 9], evaluation metrics that can well measure the performance of generative models have also been continuously developed [10, 11, 12, 13, 14]. For instance, Inception Score (IS) [10] measures the Kullback-Leibler divergence between the real and fake sample distributions. Fréchet Inception Score (FID) [11] calculates the distance between the real and fake supports using the estimated mean and variance under the multi-Gaussian assumption. The original Precision and Recall [12] and its variants [13, 14] measure fidelity and diversity separately, where fidelity is about how closely the generated samples resemble real samples, while diversity is about whether a generative model can generate samples that are as diverse as real samples.
Considering the eminent progress of deep generative models based on these existing metrics, some may question why we need another evaluation study. In this paper, we argue that we need more reliable evaluation metrics now precisely, because deep generative models have reached sufficient maturity and evaluation metrics are saturated (Table 8 in [15]). Even more, it has been recently reported that even the most widely used evaluation metric, FID, sometimes doesn't match with the expected perceptual quality, fidelity, and diversity, which means the metrics are not always working properly [16].
Kernel Density Estimator for features Bootstrap Sampling
(b) Given confidence level  $1 - \alpha$ , bootstrap confidence band  $c_{\alpha}$  computed from  $\theta_1, \dots, \theta_k$  satisfies  $\lim \inf_{n \to \infty} \mathbb{P}(|\hat{p}_h - p_h|_{\infty} < c_{\alpha}) \geq 1 - \alpha$
(c) Given  $\widehat{supp}(P)$  for real features  $\mathcal{X}$  and  $\widehat{supp}(Q)$  for generated features  $\mathcal{Y}$ ,
Figure 1: Illustration of the proposed evaluation pipeline. (a) Confidence band estimation in Section 2, (b) Robust support estimation, and (c) Evaluation via TopP&R in Section 3.
$$
T o p P _ {X} (\mathcal {Y}) := \frac {\sum_ {j = 1} ^ {m} 1 (Y _ {j} \in \widehat {s u p p} (P) \cap \widehat {s u p p} (Q))}{\sum_ {j = 1} ^ {m} 1 (Y _ {j} \in s u p p (Q))}
$$
$$
T o p R _ {y} (\mathcal {X}) := \frac {\sum_ {i = 1} ^ {n} 1 (X _ {i} \in \widehat {s u p p} (Q) \cap \widehat {s u p p} (P))}{\sum_ {i = 1} ^ {n} 1 (X _ {i} \in \widehat {s u p p} (P))}
$$
In addition, existing metrics are vulnerable to the existence of noise because all of them rely on the assumption that real data is clean. However, in practice, real data often contain lots of artifacts, such as mislabeled data and adversarial examples [17, 18], which can cause the overestimation of the data distribution in the evaluation pipeline. This error seriously perturbs the scores, leading to a false impression of improvement when developing generative models. See Appendix G.2 for possible scenarios. Thus, to provide more comprehensive ideas for improvements and to illuminate a new direction in the generative field, we need a more robust and reliable evaluation metric.
An ideal evaluation metric should effectively capture significant patterns (signal) within the data, while being robust against insignificant or accidental patterns (noise) that may arise from factors such as imperfect embedding functions, mislabeled data in the real samples, and other sources of error. Note that there is an inherent tension in developing metrics that meet these goals. On one hand, the metric should be sensitive enough so that it can capture real signals lurking in data. On the other hand, it must ignore noises that hide the signal. However, sensitive metrics are inevitably susceptible to noise to some extent. To address this, one needs a systematic way to answer the following three questions: 1) What is signal and what is noise? 2) How do we draw a line between them? 3) How confident are we on the result?
One solution can be to use the idea of Topological Data Analysis (TDA) [19] and statistical inference. TDA is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data. A key object in TDA is persistent homology, which observes how long each topological feature would survive over varying resolutions and provides a measure to quantify its significance; i.e., if some features persist longer than others over varying resolutions, we consider them as topological signals and vice versa as noise.
In this paper, we combine these ideas to estimate supports more robustly and overcome various issues of previous metrics such as unboundedness, inconsistency, etc. Our main contributions are as follows: we establish 1) a systematic approach to estimate supports via Kernel Density Estimator (KDE) derived under topological conditions; 2) a new metric that is robust to outliers while reliably detecting the change of distributions on various scenarios; 3) a consistency guarantee with robustness under very weak assumptions that are suitable for high dimensional data; 4) a combination of a noise framework and a statistical inference in TDA. Our code is available at TopP&R-NeurIPS 2023.
To lay the foundation for our method and theoretical analysis, we first explain the previous evaluation method Precision and Recall (P&R). Then, we introduce the main idea of persistent homology and its confidence estimation techniques that bring the benefit of using topological and statistical tools for addressing uncertainty in samples. For the sake of space constraints and to streamline the discussion of our main idea, we only provide a brief overview of the concepts that are relevant to this work and refer the reader to Appendix A or [20, 21, 22, 23] for further details on TDA.
Notation. For any  $x$  and  $r > 0$ , we use the notation  $\mathcal{B}_d(x,r) = \{y:d(y,x) < r\}$  be the open ball in distance  $d$  of radius  $r$ . We also write  $\mathcal{B}(x,r)$  when  $d$  is understood from the context. For a distribution  $P$  on  $\mathbb{R}^d$ , we let  $\operatorname{supp}(P) := \{x \in \mathbb{R}^d : P(\mathcal{B}(x,r)) > 0 \text{ for all } r > 0\}$  be the support of  $P$ . Throughout the paper, we refer to  $\operatorname{supp}(\mathrm{P})$  as support of  $P$ , or simply support, or manifold, but we don't necessarily require the (geometrical) manifold structure on  $\operatorname{supp}(\mathrm{P})$ . Note that, when the support is high dimensional, what we can recover at most through estimation is the partial support. For a kernel function  $K : \mathbb{R}^d \to \mathbb{R}$ , a dataset  $\mathcal{X} = \{X_1, \ldots, X_n\} \subset \mathbb{R}^d$  and bandwidth  $h > 0$ , we let the kernel density estimator (KDE) as  $\hat{p}_h(x) := \frac{1}{nh^d}\sum_{i=1}^{n}K\left(\frac{x - X_i}{h}\right)$ , and we let the average KDE as  $p_h := \mathbb{E}[\hat{p}_h]$ . We denote by  $P, Q$  the probability distributions in  $\mathbb{R}^d$  of real data and generated samples, respectively. And we use  $\mathcal{X} = \{X_1, \ldots, X_n\} \subset \mathbb{R}^d$  and  $\mathcal{Y} = \{Y_1, \ldots, Y_m\} \subset \mathbb{R}^d$  for real data and generated samples possibly with noise, respectively.
Precision and Recall. There exist two aspects of generative samples' quality; fidelity and diversity. Fidelity refers to the degree to which the generated samples resemble the real ones. Diversity, on the other hand, measures whether the generated samples cover the full variability of the real samples. Sajjadi et al. [12] was the first to propose assessing these two aspects separately via Precision and Recall1 (P&R). In the ideal case where we have full access to the probability distributions  $P$  and  $Q$ , precision  $_{P}(Q)\coloneqq Q$  ( $\mathrm{supp}(P)$ ), recall  $_{Q}(P)\coloneqq P$  ( $\mathrm{supp}(Q)$ ), which correspond to the max Precision and max Recall in [12], respectively.
Persistent homology and diagram. Persistent homology is a tool in computational topology that measures the topological invariants (homological features) of data that persist across multiple scales, and is represented in the persistence diagram. Formally, let filtration be a collection of subspaces  $\mathcal{F} = \{\mathcal{F}_{\delta}\subset \mathbb{R}^{d}\}_{\delta \in \mathbb{R}}$  with  $\delta_1\leq \delta_2$  implying  $\mathcal{F}_{\delta_1}\subset \mathcal{F}_{\delta_2}$ . Typically, filtration is defined through a function  $f$  related to data. Given a function  $f\colon \mathbb{R}^d\to \mathbb{R}$ , we consider its sublevel filtration  $\{f^{-1}(-\infty ,\delta ]\}_{\delta \in \mathbb{R}}$  or superlevel filtration  $\{f^{-1}[\delta ,\infty)\}_{\delta \in \mathbb{R}}$ . For a filtration  $\mathcal{F}$  and for each nonnegative  $k$ , we track when  $k$ -dimensional homological features (e.g., 0-dimension: connected component, 1-dimension: loop, 2-dimension: cavity, ...) appear and disappear. As  $\delta$  increases or decreases in the filtration  $\{\mathcal{F}_{\delta}\}$ , if a homological feature appears at  $\mathcal{F}_b$  and disappears at  $\mathcal{F}_d$ , we say that it is born at  $b$  and dies at  $d$ . By considering these pairs  $\{(b,d)\}$  as points in the plane  $(\mathbb{R}\cup \{\pm \infty \})^2$ , we obtain a persistence diagram. From this, a homological feature with a longer life length,  $d - b$ , can be treated as a significant feature, and a homological feature with a shorter life length as a topological noise, which lies near the diagonal line  $\{(\delta ,\delta):\delta \in \mathbb{R}\}$  (Figure 1 (b)).
Confidence band estimation. Statistical inference has recently been developed for TDA [24, 25, 26]. TDA consists of features reflecting topological characteristics of data, and it is of question to systematically distinguish features that are indeed from geometrical structures and features that are insignificant or due to noise. To statistically separate topologically significant features from topological noise, we employ confidence band estimation. Given the significance level  $\alpha$ , let confidence band  $c_{\alpha}$  be the bootstrap bandwidth of  $\| \hat{p}_h - p_h\|_{\infty}$ , computed as in Algorithm 1 (see Appendix H.2). Then it satisfies  $\lim \inf_{n\to \infty}\mathbb{P}\left(\| \hat{p}_h - p_h\|_{\infty} < c_\alpha\right)\geq 1 - \alpha$ , as in Proposition D.2 (see Appendix D). This confidence band can simultaneously determine significant topological features while filtering out noise features. We use  $c_{\mathcal{X}}$  and  $c_{\mathcal{Y}}$  to denote the confidence band defined under significance level  $\alpha$  according to the datasets  $\mathcal{X}$  and  $\mathcal{Y}$ . In later sections, we use these tools to provide a more rigorous way of scoring samples based on the confidence level we set.
3 Robust support estimation for reliable evaluation
Current evaluation metrics for generative models typically rely on strong regularity conditions. For example, they assume samples are well-curated without outliers or adversarial perturbation, real or generative models have bounded densities, etc. However, practical scenarios are wild: both real and generated samples can be corrupted with noise from various sources, and the real data can be very sparsely distributed without density. In this work, we consider more general and practical situations, wherein both real and generated samples can have noises coming from the sampling procedure, remained uncertainty due to data or model, etc. See Appendix G.2 for more on practical scenarios.
Overview of our metric We design our metric to evaluate the performance very conservatively. Our metric is based on topologically significant data structures with statistical confidence above a certain level. Toward this, we apply KDE as a function  $f$  to define a filtration, which allows
us to approximate the support with data through  $\{f^{-1}[\delta, \infty)\}_{\delta \in \mathbb{R}}$ . Since the significance of data comprising the support is determined by the life length of homological features, we calculate  $c_{\alpha}$  that enables us to systematically separate short/long lifetimes of homological features. We then estimate the supports with topologically significant data structure via superlevel set  $f^{-1}[c_{\alpha}, \infty)$  and finally, we evaluate fidelity and diversity with the estimated supports. We have collectively named this process TopP&R. By its nature, TopP&R is bounded and yields consistent performance under various conditions such as noisy data points, outliers, and even with long-tailed data distribution.
3.1 Topological precision and recall
To facilitate our discussion, we rewrite the precision in Section 2 as  $\mathrm{precision}_P(\mathcal{Y}) = Q(\mathrm{supp}(P)\cap \mathrm{supp}(Q)) / Q(\mathrm{supp}(Q))$  and define the precision of data points as
$$
\operatorname {p r e c i s i o n} _ {P} (\mathcal {Y}) := \frac {\sum_ {j = 1} ^ {m} 1 \left(Y _ {j} \in \operatorname {s u p p} (P) \cap \operatorname {s u p p} (Q)\right)}{\sum_ {j = 1} ^ {m} 1 \left(Y _ {j} \in \operatorname {s u p p} (Q)\right)}, \tag {1}
$$
which is just replacing the distribution  $Q$  with the empirical distribution  $\frac{1}{m}\sum_{j=1}^{m}\delta_{Y_j}$  of  $Y$  in the precision. Similarly,
$$
\operatorname {r e c a l l} _ {Q} (\mathcal {X}) := \frac {\sum_ {i = 1} ^ {n} 1 \left(X _ {i} \in \operatorname {s u p p} (Q) \cap \operatorname {s u p p} (P)\right)}{\sum_ {i = 1} ^ {n} 1 \left(X _ {i} \in \operatorname {s u p p} (P)\right)}. \tag {2}
$$
In practice,  $\mathrm{supp}(P)$  and  $\mathrm{supp}(Q)$  are not known a priori and need to be estimated, and these estimates should be robust to noise since we allow it now. For this, we use the KDE  $\hat{p}_{h_n}(x) \coloneqq \frac{1}{nh_n^d}\sum_{i=1}^{n}K\left(\frac{x-X_i}{h_n}\right)$  of  $\mathcal{X}$  and the bootstrap bandwidth  $c_{\mathcal{X}}$  of  $\| \hat{p}_{h_n} - p_{h_n}\|_{\infty}$ , where  $h_n > 0$  and a significance level  $\alpha \in (0,1)$  (Section 2). Then, we estimate the support of  $P$  by the superlevel set at  $c_{\mathcal{X}}^1$  as  $\mathrm{supp}(P) = \hat{p}_{h_n}^{-1}[c_{\mathcal{X}},\infty)$ , which allows to filter out noise whose KDE values are likely to be small. Similarly, the support of  $Q$  is estimated:  $\mathrm{supp}(Q) = \hat{q}_{h_m}^{-1}[c_{\mathcal{Y}},\infty)$ , where  $\hat{q}_{h_m}(x) \coloneqq \frac{1}{mh_m^d}\sum_{j=1}^{m}K\left(\frac{x-Y_j}{h_m}\right)$  is the KDE of  $\mathcal{Y}$  and  $c_{\mathcal{Y}}$  is the bootstrap bandwidth of  $\| \hat{q}_{h_m} - q_{h_m}\|_{\infty}$
For the robust estimates of the precision, we apply the support estimates to precision  $_P(\mathcal{V})$  and recall  $_Q(\mathcal{X})$  and define the topological precision and recall (TopP&R) as
$$
\begin{array}{l} \operatorname {T o p P} _ {\mathcal {X}} (\mathcal {Y}) := \frac {\sum_ {j = 1} ^ {m} 1 \left(Y _ {j} \in \operatorname {s u p p} (P) \cap \operatorname {s u p p} (Q)\right)}{\sum_ {j = 1} ^ {m} 1 \left(Y _ {j} \in \operatorname {s u p p} (Q)\right)} \\ = \frac {\sum_ {j = 1} ^ {m} 1 \left(\hat {p} _ {h _ {n}} \left(Y _ {j}\right) > c \chi , \hat {q} _ {h _ {m}} \left(Y _ {j}\right) > c \mathcal {Y}\right)}{\sum_ {j = 1} ^ {m} 1 \left(\hat {q} _ {h _ {m}} \left(Y _ {j}\right) > c \mathcal {Y}\right)}, \tag {3} \\ \end{array}
$$
$$
\operatorname {T o p R} _ {\mathcal {Y}} (\mathcal {X}) := \frac {\sum_ {i = 1} ^ {n} 1 \left(\hat {q} _ {h _ {m}} \left(X _ {i}\right) > c _ {\mathcal {Y}} , \hat {p} _ {h _ {n}} \left(X _ {i}\right) > c _ {\mathcal {X}}\right)}{\sum_ {i = 1} ^ {n} 1 \left(\hat {p} _ {h _ {n}} \left(X _ {i}\right) > c _ {\mathcal {X}}\right)}. \tag {4}
$$
The kernel bandwidths  $h_n$  and  $h_m$  are hyperparameters, and we provide guidelines to select the optimal bandwidths  $h_n$  and  $h_m$  in practice (See Appendix H.4).
3.2 Bandwidth estimation using bootstrapping
Using the bootstrap bandwidth  $c_{\mathcal{X}}$  as the threshold is the key part of our estimator (TopP&R) for robustly estimating  $\mathrm{supp}(P)$ . As we have seen in Section 2, the bootstrap bandwidth  $c_{\mathcal{X}}$  filters out the topological noise in topological data analysis. Analogously, using  $c_{\mathcal{X}}$  allows to robustly estimate  $\mathrm{supp}(P)$ . When  $X_{i}$  is an outlier, its KDE value  $\hat{p}_h(X_i)$  is likely to be small as well as the values at the connected component generated by  $X_{i}$ . So those components from outliers are likely to be removed in the estimated support  $\hat{p}_h^{-1}[c_{\mathcal{X}},\infty)$ . Higher dimensional homological noises are also removed. Hence, the estimated support denoises topological noise and robustly estimates  $\mathrm{supp}(P)$ . See Appendix C for a more detailed explanation.
Now that we are only left with topological features of high confidence, this allows us to draw analogies to confidence intervals in statistical analysis, where the uncertainty of the samples is treated
<sup>1</sup>The computation of  $c_{\alpha}$  and its practical interpretation is described in Algorithm 1.
by setting the level of confidence. In the next section, we show that TopP&R not only gives a more reliable evaluation score for generated samples but also has good theoretical properties.
3.3 Addressing the curse of dimensionality
As discussed in Section 3.2, getting the bootstrap bandwidth  $c_{\mathcal{X}}$  with a theoretical guarantee plays a key role in our metric, and the choice of KDE as a filtration function is inevitable, as in Remark 4.4. However, computing the support of a high-dimensional feature with KDE demands significant computation, and the accuracy is low due to low density values. This hinders an efficient and correct evaluation in practice. To address this issue, we apply a random projection into a low-dimensional space by leveraging the Johnson-Lindenstrauss Lemma (Lemma B.1). This lemma posits that using a random projection effectively preserves information regarding distances and homological features, composed of high-dimensional features, in a low-dimensional representation. Furthermore, we have shown that random projection does not substantially reduce the influence of noise, nor does it affect the performance of TopP&R under various conditions with complex data (Section 5, I.7, and I.8).
4 Consistency with robustness of TopP&R
The key property of TopP&R is consistency with robustness. The consistency ensures that, the precision and the recall we compute from the data approaches the precision and the recall from the distribution as we have more samples. The consistency allows to investigate the precision and recall of full distributions only with access to finite sampled data. TopP&R achieves consistency with robustness, that is, the consistency holds with the data possibly corrupted by noise. This is due to the robust estimation of supports with KDE with confidence bands.
We demonstrate the statistical model for both data and noise. Let  $P$ ,  $Q$ ,  $\mathcal{X}$ ,  $\mathcal{Y}$  be as in Section 2, and let  $\mathcal{X}^0$ ,  $\mathcal{Y}^0$  be real data and generated data without noise.  $\mathcal{X}, \mathcal{Y}, \mathcal{X}^0, \mathcal{Y}^0$  are understood as multisets, i.e., elements can be repeated. We first assume that the uncorrupted data are IID.
Assumption 1. The data  $\mathcal{X}^0 = \{X_1^0,\ldots ,X_n^0\}$  and  $\mathcal{Y}^0 = \{Y_1^0,\dots ,Y_m^0\}$  are IID from  $P$  and  $Q$ , respectively.
In practice, the data is often corrupted with noise. We consider the adversarial noise, where some fraction of data are replaced with arbitrary point cloud data.
Assumption 2. Let  $\{\rho_k\}_{k\in \mathbb{N}}$  be a sequence of nonnegative real numbers. Then the observed data  $\mathcal{X}$  and  $\mathcal{Y}$  satisfies  $|\mathcal{X} \backslash \mathcal{X}^0| = n\rho_n$  and  $|\mathcal{Y} \backslash \mathcal{Y}^0| = m\rho_m$ .
In the adversarial model, we control the level of noise by the fraction  $\rho$ , but do not assume other conditions such as IID or boundedness, to make our noise model very general and challenging.
For distributions and kernel functions, we assume weak conditions, detailed in Assumption A1 and A2 in Appendix D. Under the data and the noise models, TopP&R achieves consistency with robustness. That is, the estimated precision and recall are asymptotically correct with high probability even if up to a portion of  $1 / \sqrt{n}$  or  $1 / \sqrt{m}$  are replaced by adversarial noise. This is due to the robust estimation of the support with the kernel density estimator with the confidence band of the persistent homology.
Proposition 4.1. Suppose Assumption 1, 2, A1, A2 hold. Suppose  $\alpha \to 0$ ,  $h_n \to 0$ ,  $nh_n^d \to \infty$ ,  $nh_n^{-d}\rho_n^2 \to 0$ , and similar relations hold for  $h_m$ ,  $\rho_m$ . Then,
$$
\left| \operatorname {T o p P} _ {\mathcal {X}} (\mathcal {Y}) - \operatorname {p r e c i s i o n} _ {P} (\mathcal {Y}) \right| = O _ {\mathbb {P}} \left(Q \left(B _ {n, m}\right) + \rho_ {m}\right),
$$
$$
\left| \operatorname {T o p R} _ {\mathcal {Y}} (\mathcal {X}) - \operatorname {r e c a l l} _ {Q} (\mathcal {X}) \right| = O _ {\mathbb {P}} \left(P \left(A _ {n, m}\right) + \rho_ {n}\right),
$$
for fixed sequences of sets  $\{A_{n,m}\}_{n,m\in \mathbb{N}},\{B_{n,m}\}_{n,m\in \mathbb{N}}$  with  $P(A_{n,m})\to 0$  and  $Q(B_{n,m})\to 0$  as  $n,m\to \infty$
Theorem 4.2. Under the same condition as in Proposition 4.1,
$$
\left| \operatorname {T o p P} _ {\mathcal {X}} (\mathcal {Y}) - \operatorname {p r e c i s i o n} _ {P} (Q) \right| = O _ {\mathbb {P}} \left(Q \left(B _ {n, m}\right) + \rho_ {m}\right),
$$
$$
\left| \operatorname {T o p R} _ {\mathcal {Y}} (\mathcal {X}) - \operatorname {r e c a l l} _ {Q} (P) \right| = O _ {\mathbb {P}} \left(P \left(A _ {n, m}\right) + \rho_ {n}\right).
$$
Since  $P(A_{n,m}) \to 0$  and  $Q(B_{n,m}) \to 0$ , these imply consistencies of TopP&R. In fact, additionally under minor probabilistic and geometrical assumptions,  $P(A_{n,m})$  and  $Q(B_{n,m})$  are of order  $h_m + h_n$ .
Lemma 4.3. Under the same condition as in Proposition 4.1 and additionally under Assumption A3, A4,  $P(A_{n,m}) = O(h_n + h_m)$  and  $Q(B_{n,m}) = O(h_n + h_m)$ .
Remark 4.4. Consistency guarantees from Proposition 4.1 and Theorem 4.2 are in principle due to the uniform convergence of KDE over varying bandwidth  $h_n$  (Proposition D.2). Once we replace estimating the support with KDE by k-NN or something else, we wouldn't have consistency guarantees. Hence, using the KDE is an essential part for the theoretical guarantees of TopP&R.