SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our theoretical results in Proposition 4.1 and Theorem 4.2 are novel and important in several perspectives. These results are among the first theoretical guarantees for evaluation metrics for generative models as far as we are aware of. Also, as in Remark D.1, assumptions are very weak and suitable for high dimensional data. Also, robustness to adversarial noise is provably guaranteed.
A good evaluation metric should not only possess desirable theoretical properties but also effectively capture the changes in the underlying data distribution. To examine the performance of evaluation metrics, we carefully select a set of experiments for sanity checks. With toy and real image data, we check 1) how well the metric captures the true trend of underlying data distributions and 2) how well the metric resists perturbations applied to samples.
We compare TopP&R with several representative evaluation metrics that include Improved Precision and Recall (P&R) [13], Density and Coverage (D&C) [14], Geometric Component Analysis (GCA) [27], and Manifold Topology Divergence (MTD) [28] (Appendix F). Both GCA and MTD are the recent evaluation metrics that utilize topological features to some extent; GCA defines precision and recall based on connected components of  $P$  and  $Q$ , and MTD measures the distance between two distributions using the sum of lifetimes of homological features. For all the experiments, linear random projection to 32 dimensions is additionally used for TopP&R, and the shaded area of all figures denotes the  $\pm 1$  standard deviation for ten trials. For a fair comparison with existing metrics, we have utilized 10k real and fake samples for all experiments. For more details, please refer to Appendix H.1.
5.1 Sanity checks with toy data
Following [14], we first examine how well the metric reflects the trend of  $\mathcal{V}$  moving away from  $\mathcal{X}$  and whether it is suitable for finding mode-drop phenomena. In addition to these, we newly design several experiments that can highlight TopP&R's favorable theoretical properties of consistency with robustness in various scenarios.
5.1.1 Shifting the generated feature manifold
We generate samples from  $\mathcal{X} \sim \mathcal{N}(\mathbf{0}, I)$  and  $\mathcal{Y} \sim \mathcal{N}(\mu \mathbf{1}, I)$  in  $\mathbb{R}^{64}$ , where  $\mathbf{1}$  is a vector of ones and  $I$  is an identity matrix. We examine how each metric responds to shifting  $\mathcal{Y}$  with  $\mu \in [-1, 1]$  while there are outliers at  $\mathbf{3} \in \mathbb{R}^{64}$  for both  $\mathcal{X}$  and  $\mathcal{Y}$  (Figure 2). We discovered that GCA struggles to detect changes using its default hyperparameter configuration, and this issue persists even after performing an exhaustive hyperparameter sweep. Since empirical tuning of the hyperparameters is required for each dataset, utilizing GCA in practical applications proves to be challenging (Appendix F). Both improved P&R and D&C behave pathologically since these methods estimate the support via the k-nearest neighbor algorithm, which inevitably overestimate the underlying support when there are outliers. For example, when  $\mu < 0.5$ , Recall returns a high-diversity score, even though the true supports of  $\mathcal{X}$  and  $\mathcal{Y}$  are actually far apart. In addition, P&R does not reach 1 in high dimensions even when  $\mathcal{X} = \mathcal{Y}$ . D&C [14] yields better results than P&R because it consistently uses  $\mathcal{X}$  (the real data distribution) as a reference point, which typically has fewer outliers than  $\mathcal{Y}$  (the fake data distribution). However, there is no guarantee that this will always be the case in practice [17, 18]. If an outlier is present in  $\mathcal{X}$ , D&C also returns an incorrect high-fidelity score at  $\mu > 0.5$ . On the other hand, TopP&R shows a stable trend unaffected by the outlier, demonstrating its robustness.
5.1.2 Dropping modes
We simulate mode-drop phenomena by gradually dropping all but one mode from the fake distribution  $\mathcal{V}$  that is initially identical to  $\mathcal{X}$  (Figure 3). Here, we consider the mixture of Gaussians with seven
Figure 2: Behaviors of evaluation metrics for outliers on real and fake distribution. For both real and fake data, the outliers are fixed at  $3 \in \mathbb{R}^{64}$ , and the parameter  $\mu$  is shifted from -1 to 1.
Figure 3: Behaviors of evaluation metrics for (a) sequential and (b) simultaneous mode-drop scenarios. The horizontal axis shows the concentration ratio on the distribution centered at  $\mu = 0$ .
Figure 4: Behaviors of evaluation metrics on Non-IID perturbations. We replace a certain percentage of real and fake data (a) with random uniform noise or (b) by switching some of real and fake data.
modes in  $\mathbb{R}^{64}$ . We keep the number of samples in  $\mathcal{X}$  constant so that the same amount of decreased samples are supplemented to the first mode which leads fidelity to be fixed to 1. We observe that the values of Precision fail to saturate, i.e., mainly smaller than 1, and the Density fluctuates to a value greater than 1, showing their instability and unboundedness. Recall and GCA do not respond to the simultaneous mode drop, and Coverage decays slowly compared to the reference line. In contrast, TopP performs well, being held at the upper bound of 1 in sequential mode drop, and TopR also decreases closest to the reference line in simultaneous mode drop.
5.1.3 Tolerance to Non-IID perturbations
Robustness to perturbations is another important aspect we should consider when designing a metric. Here, we test whether TopP&R behaves stably under two variants of noise cases (see Section G.3; 1) scatter noise: replacing  $X_{i}$  and  $Y_{j}$  with uniformly distributed noise and 2) swap noise: swapping the position between  $X_{i}$  and  $Y_{j}$ . These two cases all correspond to the adversarial noise model of Assumption 2. We set  $\mathcal{X} \sim \mathcal{N}(\mu = 0, I) \in \mathbb{R}^{64}$  and  $\mathcal{Y} \sim \mathcal{N}(\mu = 1, I) \in \mathbb{R}^{64}$  where  $\mu = 1$ , and thus an ideal evaluation metric must return zero for both fidelity and diversity. In the result, while the GCA precision is relatively robust to the scatter noise, GCA recall tends to be sensitive to the swap noise. In both cases, we find that P&R and D&C are more sensitive while TopP&R remains relatively stable until the noise ratio reaches  $15\%$  of the total data, which is a clear example of the weakness of existing metrics to perturbation (Figure 4).
Figure 5: Comparison of evaluation metrics on Non-IID perturbations using FFHQ dataset. We replaced certain ratio of  $\mathcal{X}$  and  $\mathcal{Y}$  (a) with outliers and (b) by switching some of real and fake features.
Figure 6: Verification of whether TopP&R can make an accurate quantitative assessment of noisy image features. Gaussian noise, gaussian blur, salt and pepper, and black rectangle noise are added to the FFHQ images and embedded with T4096.
5.2 Sanity check with Real data
Now that we have verified the metrics on toy data, we move on to real data. Just like in the toy experiments, we concentrate on how the metrics behave in extreme situations, such as outliers, mode-drop phenomena, perceptual distortions, etc. We also test different image embeddings, including pretrained VGG16 [29], InceptionV3 [30], and SwAV [31].
5.2.1 Dropping modes in Baby ImageNet
We have conducted an additional experiment using Baby ImageNet [15] to investigate the sensitivity of TopP&R to mode-drop in real-world data. The performance of each metric (Figure A4) is measured with the identical data while simultaneously dropping the modes of nine classes of Baby ImageNet, in total of ten classes. Since our experiment involves gradually reducing the fixed number of fake samples until nine modes of the fake distribution vanish, the ground truth diversity should decrease linearly. From the experimental results, consistent to the toy result of Figure 3, both D&C and P&R still struggle to respond to simultaneous mode dropping. In contrast, TopP&R consistently exhibit a high level of sensitivity to subtle distribution changes. This notable capability of TopP&R can be attributed to its direct approximation of the underlying distribution, distinguishing it from other metrics. In addition, we perform the experiments on a dataset with long-tailed distribution and find that TopP&R captures the trend well even when there are minority sets (Appendix I.1). This again shows the reliability of TopP&R.
5.2.2 Robustness to perturbations
To test the robustness of our metric against the adversarial noise model of Assumption 2, we test both scatter-noise and swap noise scenarios with real data (see Section G.3). In the experiment, following Kynkänniemi et al. [13], we first classify inliers and outliers that are generated by StyleGAN [1]. For scatter noise we add the outliers to the inliers and for swap noise we swap the real FFHQ images with generated images. Under these specific noise conditions, Precision shows similar or even better robustness than Density (Figure 5). On the other hand, Coverage is more robust than Recall. In both cases, TopP&R shows the best performance, resistant to noise.
5.2.3 Sensitiveness to the noise intensity
One of the advantages of FID [11] is that it is good at estimating the degrees of distortion applied to the images. Similarly, we check whether the F1-score based on TopP&R provides a reasonable evaluation according to different noise levels. As illustrated in Figure 6 and A5,  $\mathcal{X}$  and  $\mathcal{Y}$  are sets
Table 1: Generative models trained on CIFAR-10 are ranked by FID, KID, MTD, and F1-scores based on TopP&R, D&C and P&R, respectively. The  $\mathcal{X}$  and  $\mathcal{Y}$  are embedded with InceptionV3, VGG16, and SwAV. The number inside the parenthesis denotes the rank based on each metric.
Model	StyleGAN2	ReACGAN	BigGAN	PDGAN	ACGAN	WGAN-GP
InceptionV3	FID (↓)	3.78 (1)	3.87 (2)	4.16 (3)	31.54 (4)	33.39 (5)	107.68 (6)
KID (↓)	0.002 (1)	0.012 (3)	0.011 (2)	0.025 (4)	0.029 (5)	0.137 (6)
TopP&R (↑)	0.9769 (1)	0.8457 (2)	0.7751 (3)	0.7339 (4)	0.6951 (5)	0.0163 (6)
D&C (↑)	0.9626 (2)	0.9409 (3)	1.1562 (1)	0.4383 (4)	0.3883 (5)	0.1913 (6)
P&R (↑)	0.6232 (1)	0.3320 (2)	0.3278 (3)	0.1801 (4)	0.0986 (5)	0.0604 (6)
MTD (↓)	2.3380 (3)	2.2687 (2)	1.4473 (1)	7.0188 (4)	8.0728 (5)	11.498 (6)
VGG16	TopP&R (↑)	0.9754 (1)	0.5727 (3)	0.7556 (2)	0.4021 (4)	0.3463 (5)	0.0011 (6)
D&C (↑)	0.9831 (3)	1.0484 (1)	0.9701 (4)	0.9872 (2)	0.8971 (5)	0.6372 (6)
P&R (↑)	0.6861 (1)	0.1915 (3)	0.3526 (2)	0.0379 (4)	0.0195 (5)	0.0001 (6)
MTD (↓)	25.757 (4)	25.826 (3)	34.755 (5)	24.586 (2)	23.318 (1)	41.346 (6)
SwAV	TopP&R (↑)	0.9093 (1)	0.3568 (3)	0.5578 (2)	0.1592 (4)	0.1065 (5)	0.0003 (6)
D&C (↑)	1.0732 (1)	0.9492 (3)	1.0419 (2)	0.6328 (4)	0.4565 (5)	0.0721 (6)
P&R (↑)	0.5623 (1)	0.0901 (3)	0.1459 (2)	0.0025 (4)	0.0000 (6)	0.0002 (5)
MTD (↓)	1.1098 (1)	1.5512 (3)	1.3280 (2)	1.8302 (4)	2.2982 (5)	4.9378 (6)
of reference and noisy features, respectively. The experimental results show that TopP&R actually reflects well the different degrees of distortion added to the images while a similar topology-based method MTD shows inconsistent behavior to the distortions.
5.2.4 Ranking between generative models
The alignment between FID (or KID) and perceptual evaluation has been well-established in prior research, and these scores are widely used as a primary metric in the development of generative models due to its close correspondence with human perception. Consequently, generative models have evolved to align with FID's macroscopic perspective. Therefore, we believed that the order determined by FID at a high level captures to some extent the true performance hierarchy among models, even if it may not perfectly reflect it. In other words, if the development of generative models based on FID leads to genuine improvements in generative performance and if there is a meaningful correlation, similar rankings should be maintained even when the representation or embedding model changes. From this standpoint, while other metrics exhibit fluctuating rankings, TopP&R consistently provides the most stable and consistent results similar to both FID and KID. To quantitatively compare the similarity of rankings across varying embeddings by different metrics, we have computed mean Hamming Distance (MHD) (Appendix H.6) where lower value indicates more similarity. TopP&R, P&R, D&C, and MTD have MHDs of 1.33, 2.66, 3.0, and 3.33, respectively.
Many works have been proposed recently to assess the fidelity and diversity of generative models. However, none of them has focused on the accurate estimation of support even though it is one of the key components in the entire evaluation pipeline. In this paper, we proposed topological precision and recall (TopP&R) that provides a systematical fix by robustly estimating the support with both topological and statistical treatments. To the best of our knowledge, TopP&R is the first evaluation metric that offers statistical consistency under noisy conditions, which may arise in real practice. Our theoretical and experimental results showed that TopP&R serves as a robust and reliable evaluation metric under various embeddings and noise conditions, including mode drop, outliers, and Non-IID perturbations. Last but not least, TopP&R provides the most consistent ranking among different generative models across different embeddings via calculating its F1-score.
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2.220574.01), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2020-0-01336, Artificial Intelligence Graduate School Program (UNIST), No.2021-0-02068, Artificial Intelligence Innovation Hub, No.2022-0-00959, (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making, No.2022-0-00264, Comprehensive Video Understanding and Generation with Knowledge-based Deep Logic Neural Network).
[1] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[2] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110-8119, 2020.
[3] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[6] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[7] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. arXiv preprint arXiv:2202.00273, 2022.
[8] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems, 34, 2021.
[9] Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. Advances in Neural Information Processing Systems, 33:21357-21369, 2020.
[10] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[11] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[12] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in Neural Information Processing Systems, 31, 2018.
[13] Tuomas Kynkänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
[14] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjoy Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pages 7176-7185. PMLR, 2020.
[15] Minguk Kang, Joonghyuk Shin, and Jaesik Park. Studiogan: A taxonomy and benchmark of gans for image synthesis. arXiv preprint arXiv:2206.09479, 2022.
[16] Tuomas Kynkänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. arXiv preprint arXiv:2203.06026, 2022.
A More Background on Topological Data Analysis
Topological data analysis (TDA) [19] is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data. A key object in TDA is persistent homology, which quantifies salient topological features of data by observing them in multi-resolutions.
A.1 Persistent Homology
Persistent homology. Persistent homology is a multiscale approach to represent the topological features. For a filtration  $\mathcal{F}$  and for each  $k\in \mathbb{N}_0 = \mathbb{N}\cup \{0\}$ , the associated  $k$ -th persistent homology  $PH_{k}\mathcal{F}$  is a collection of  $k$ -th dimensional homologies  $\{H_k\mathcal{F}_{\delta}\}_{\delta \in \mathbb{R}}$  equipped with homomorphisms  $\{t_k^{a,b}:H_k\mathcal{F}_a\to H_k\mathcal{F}_b\}_{a\leq b}$  induced by the inclusion  $\mathcal{F}_a\subset \mathcal{F}_b$ .
Persistence diagram. For the  $k$ -th persistent homology  $PH_{k}\mathcal{F}$ , the set of filtration levels at which a specific homology appears is always an interval  $[b,d)\subset [-\infty ,\infty ]$ . The corresponding  $k$ -th persistence diagram is a multiset of points  $(\mathbb{R}\cup \{\infty \})^{2}$ , consisting of all pairs  $(b,d)$  where  $[b,d)$  is the interval of filtration values for which a specific homology appears in  $PH_{k}\mathcal{F}$ .
A.2 Statistical Inference of Persistent Homology
As discussed above, a homological feature with a long life-length is important information in topology while the homology with a short life-length can be treated as non-significant information or noise. The confidence band estimator provides the confidence set from the features that only include topologically and statistically significant (statistically considered as elements in the population set) under a certain level of confidence. And to build a confidence set, we first need to endow a metric on the space of persistence diagrams.
Bottleneck distance. The most fundamental metric to measure the distance between two persistence diagrams is the bottleneck distance.
Definition A.1. The bottleneck distance between two persistence diagrams  $D_{1}$  and  $D_{2}$  is defined by
$$
d_{B}(D_{1},D_{2}) = \inf_{\gamma \in \Gamma}\sup_{p\in D_{1}}\| p - \gamma (p)\|_{\infty},
$$
where the set  $\Gamma$  consists of all the bijections  $\gamma : D_1 \cup \text{Diag} \to D_2 \cup \text{Diag}$ , and  $\text{Diag}$  is the diagonal  $\{(x, x) : x \in \mathbb{R}\} \subset \mathbb{R}^2$  with infinite multiplicity.
One way of constructing the confidence set uses the superlevel filtration of the kernel density estimator and the bootstrap confidence band. Let  $\mathcal{X} = \{X_1, X_2, \dots, X_n\}$  as given points cloud, then the probability for the distribution of points can be estimated via KDE defined as following:  $\hat{p}_h(x) := \frac{1}{nh^d} \sum_{i=1}^{n} K\left(\frac{x - X_i}{h}\right)$  where  $h$  is the bandwidth and  $d$  as a dimension of the space. We compute  $\hat{p}_h$  and  $\hat{p}_h^*$ , which are the KDE of  $\mathcal{X}$  and the KDE of bootstrapped samples  $\mathcal{X}^*$ , respectively. Now, given the significance level  $\alpha$  and  $h > 0$ , let confidence band  $q_{\mathcal{X}}$  be bootstrap bandwidth of a Gaussian Empirical Process [32, 33],  $\sqrt{n} ||\hat{p}_h - \hat{p}_h^*||_\infty$ . Then it satisfies  $P(\sqrt{n} ||\hat{p}_h - p_h||_\infty < q_{\mathcal{X}}) \geq 1 - \alpha$ , as in Proposition D.2 in Section D. Then  $\mathcal{B}_{d_B}(\hat{\mathcal{P}}_h, c_{\mathcal{X}})$ , the ball of persistent homology centered at  $\hat{\mathcal{P}}_h$  and radius  $c_{\mathcal{X}} = q_{\mathcal{X}} / \sqrt{n}$  in the bottleneck distance  $d_B$ , is a valid confidence set as  $\lim \inf_{n \to \infty} \mathbb{P}\left(\mathcal{P} \in \mathcal{B}_{d_B}(\hat{\mathcal{P}}_h, c_{\mathcal{X}})\right) \geq 1 - \alpha$ . This confidence set has further interpretation that in the persistence diagram, homological features that are above twice the radius  $2c_{\mathcal{X}}$  from the diagonal are simultaneously statistically significant.
B Johnson-Lindenstrauss Lemma
Johnson-Lindenstrass Lemma is stated as follows:
Lemma B.1 (Johnson-Lindenstrauss Lemma). [34, Lemma 1]
Let  $0 < \epsilon < 1$  and  $\mathcal{X} \subset \mathbb{R}^D$  be a set of points with size  $n$ . Then for  $d = O\left(\min \left\{n, D, \epsilon^{-2}\log n\right\}\right)$ , there exists a linear map  $f: \mathbb{R}^D \to \mathbb{R}^d$  such that for all  $x, y \in \mathcal{X}$
$$
(1 - \epsilon) \| x - y \| ^ {2} \leq \| f (x) - f (y) \| ^ {2} \leq (1 + \epsilon) \| x - y \| ^ {2}. \tag {5}
$$
Remark B.2. This Johnson-Lindenstrass Lemma is optimal for linear maps, and nearly-optimal even if we allow non-linear maps: there exists a set of points  $\mathcal{X} \subset \mathbb{R}^D$  with size  $n$ , such that  $d$  should satisfy: (a)  $d = \Omega (\epsilon^{-2}\log n)$  for a linear map  $f:\mathbb{R}^{D}\to \mathbb{R}^{d}$  satisfying (5) to exist [35, Theorem 3], and (b)  $d = \Omega (\epsilon^{-2}\log (\epsilon^{2}n))$  for a map  $f:\mathbb{R}^{D}\rightarrow \mathbb{R}^{d}$  satisfying (5) to exist [36, Theorem 1].
C Denoising topological features from outliers
Using the bootstrap bandwidth  $c_{\mathcal{X}}$  as the threshold is the key part of our estimators TopP&R for robustly estimating  $\operatorname{supp}(P)$ . When the level set  $\hat{p}_h^{-1}[c_{\mathcal{X}},\infty)$  is used, the homology of  $\hat{p}_h^{-1}[c_{\mathcal{X}},\infty)$  consists of homological features whose (birth)  $\geq c_{\mathcal{X}}$  and (death)  $\leq c_{\mathcal{X}}$ , which are the homological features in skyblue area in Figure A1. In this example, we consider three types of homological noise, though there can be many more corresponding to different homological dimensions.
- There can be a 0-dimensional homological noise of (birth)  $< c_{\mathcal{X}}$  and (death)  $< c_{\mathcal{X}}$ , which is the red point in the persistence diagram of Figure A1. This noise corresponds to the orange connected component on the left. As in the figure, this type of homological noise usually corresponds to outliers.
- There can be a 0-dimensional homological noise of (birth)  $>c_{\mathcal{X}}$  and (death)  $>c_{\mathcal{X}}$ , which is the green point in the persistence diagram of Figure A1. This noise corresponds to the connected component surrounded by the green line on the left. As in the figure, this type of homological noise lies within the estimated support, not like the other two.
- There can be a 1-dimensional homological noise of (birth)  $< c_{\mathcal{X}}$  and (death)  $< c_{\mathcal{X}}$ , which is the purple point in the persistence diagram of Figure A1. This noise corresponds to the purple loop on the left.
These homological noises satisfy either their (birth)  $< c_{\mathcal{X}}$  and (death)  $< c_{\mathcal{X}}$  or their (birth)  $> c_{\mathcal{X}}$  and (death)  $> c_{\mathcal{X}}$  simultaneously with high probability, so those homological noises are removed in the estimated support  $\hat{p}_h^{-1}[c_{\mathcal{X}},\infty)$ , which is the blue area in the left and the skyblue area in the right in Figure A1.