TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Models
Authors: Pum Jun Kim
Yoojin Jang
Jisu Kim
Jaejun Yoo
Venue: NeurIPS 2023 poster (2023)
Keywords: GAN
Evaluation
Support Estimation
TL;DR: We introduce a topological and statistical treatments for rigorous support estimation.

Abstract: We propose a robust and reliable evaluation metric for generative models called Topological Precision and Recall (TopP&R, pronounced “topper”), which systematically estimates supports by retaining only topologically and statistically significant features with a certain level of confidence. Existing metrics, such as Inception Score (IS), Frechet Inception Distance (FID), and various Precision and Recall (P&R) variants, rely heavily on support estimates derived from sample features. However, the reliability of these estimates has been overlooked, even though the quality of the evaluation hinges entirely on their accuracy. In this paper, we demonstrate that current methods not only fail to accurately assess sample quality when support estimation is unreliable, but also yield inconsistent results. In contrast, TopP&R reliably evaluates the sample quality and ensures statistical consistency in its results. Our theoretical and experimental findings reveal that TopP&R provides a robust evaluation, accurately capturing the true trend of change in samples, even in the presence of outliers and non-independent and identically distributed (Non-IID) perturbations where other methods result in inaccurate support estimations. To our knowledge, TopP&R is the first evaluation metric specifically focused on the robust estimation of supports, offering statistical consistency under noise conditions.

TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Models
Pum Jun Kim!
Yoojin Jang'
Jisu 2.34 Jaejun Yoo
Ulsan National Institute of Science and Technology
2 Seoul National University *Inraa "Paris-Saclay University
{pumjun. kim, softjin, aao ac.kr kr
jkim821330snu.acc ac
Abstract
We propose a robust and reliable evaluation metric for generative models called Topological Precision and Recall (TopP&R, pronounced "topper"), which systematically estimates supports by retaining only topologically and statistically significant features with certain level of confidence. Existing metrics, such as Inception Score (IS), Fréchet Inception Distance (FID), and various Precision and Recall (P&R) variants, rely heavily on support estimates derived from sample features. However, the reliability of these estimates has been overlooked, even though the quality of the evaluation hinges entirely on their accuracy. this paper, we demonstrate that current methods not only fail to accurately assess sample quality when support estimation is unreliable, but also yield inconsistent results. In contrast, TopP&R reliably evaluates the sample quality and ensures statistical consistency in its results. Our theoretical and experimental findings reveal that TopP&R provides robust evaluation, accurately capturing the true trend of change in samples, even the presence of outliers and non-independent and identically distributed (Non-IID) perturbations where other methods result in inaccurate support estimations. To our knowledge, TopP&R is the first evaluation metric specifically focused the robust estimation supports, offering statistical consistency under noise conditions.
Introduction
In keeping with the remarkable improvements deep generative models 11 2 3 6
evaluation metrics that can well measure the performance of generative models have also been continuously developed [10] II
13 14]. For instance, Inception Score (IS) [00] measures the Kullback-Leiller divergence between the real and fake sample distributions. Fréchet Inception Score (FID) [T calculates the distance between the real and fake supports using the estimated mean assumption. The original Precision and Recall [22 and its variants B3 [4 measure fidelity and diversity separately, where fidelity about how closely the generated samples resemble real samples, while diversity is about whether generative model can generate samples that are as diverse as real samples.
and variance under the
Considering the eminent progress of deep generative models based these existing metrics, some may question why we need another evaluation study. this paper, we argue that we need more reliable evaluation metrics precisely, because deep generative models have reached sufficient maturity and evaluation metrics are saturated (Table in (15)). Even more, it has been recently reported that even the most widely used evaluation metric, FID, sometimes doesn't 't match with the expected perceptual quality, fidelity, and diversity, which means the metrics are not always working properly (16).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

images Generated images
Foom enbealer
Probability support ostimation vin KDE and bootstrap
Confidence hand
(c) Topl' and TopR
estimation
evaluation
P
Komel Denait Estimatoc for foorrres
(b) confidence a, bootstrap confidence hand bg computed from A,.
sutisfies Inf s-z
al1 Pallz < (2) 2
Boutstrup Sampling
Btrth
Topological
0
Ah
P
(2)
02
Pa
P
Topological signal
IL.
Pn [G m)
Pa
Death
(c) Given Givenstipp(P (P) for real features x and supp for generated features y.
TopPx(9) 2/ 10, E SPpp n sup)
(Pip(P)
E, 10; E supp (Q)
(X)) El., 1(K, E (upp(() n (P))
(upp(Q)
Mi= 1(X, E supp
Figure Section
Illustaaioon of the proposed evaluation pipeline. (a) Confidence band estimation in (b) Robust support estimation, and (c) Evaluation via TopP&R in Section
In addition, existing metrics are vulnerable to the existence of noise because all of them rely on the assumption that real data is clean. However, in practice, real data often contain lots of artifacts, such as mislabeled data and adversarial examples [17 [8], which can cause the overestimation of the data distribution in the evaluation pipeline. This error seriously perturbs the scores, leading false impression of improvement when developing generative models. See Appendix G.2 for possible scenarios. Thus, to provide more comprehensive ideas for improvements and illuminate new direction in the generative field, we need a more robust and reliable evaluation metric.
An ideal evaluation metric should effectively capture significant patterns (signal) within the data, while being robust against insignificant or accidental patterns (noise) that may arise from factors such imperfect embedding functions, mislabeled data in the real samples, and other sources of error. Note that there is an inherent tension in developing metrics that meet these goals. On one hand, the metric should be sensitive enough so that it can capture real signals lurking in data. On the other hand, it must ignore noises that hide the signal. However, sensitive metrics are inevitably susceptible to noise to some extent. To address this, one needs a systematic way answer the following three questions: 1) What is signal and what is noise? 2) How do we draw a line between them? 3) How confident are we on the result?
One solution can be to use the idea of Topological Data Analysis (TDA) 19 and statistical inference. TDA is recent and emerging field of data science that relies topological tools to infer relevant features for possibly complex data. A key object in TDA is persistent homology, which observes how long each topological feature would survive over varying resolutions and provides a measure to quantify its significance; i.e,, if some features persist longer than others over varying resolutions, we consider them as topological signals and vice versa noise.
In this paper, we combine these ideas to estimate supports more robustly and overcome various issues of previous metrics such unboundedness, inconsistency, etc. Our main contributions are as follows: we establish 1) a systematic approach to estimate supports via Kernel Density Estimator (KDE) derived under topological conditions; 2) new metric that is robust to outliers while reliably detecting the change of distributions on various scenarios; 3) a consistency guarantee with robustness under very weak assumptions that are suitable for high dimensional data; 4) a combination of noise framework and a statistical inference in TDA, Our code is available at TooP&&-NeurlPP 2023
2 Background
To lay the foundation for our method and theoretical analysis, we first explain the previous evaluation method Precision and Recall (P&R). Then, we introduce the main idea of persistent homology and its confidence estimation techniques that bring the benefit of using topological and statistical tools for addressing uncertainty in samples. For the sake of space constraints and to streamline the discussion of our main idea, we only provide a brief overview of the concepts that are relevant to this work and refer the reader to Appendix A or (20) 21 22 23 for further details on TDA.

---

Notation. For any x and > O, we use the notation (x r) {y d(y.z) < r be the open ball in distance d of radius r. We also write B(z, r) when d understood from the context. For a distribution P on I2, we let supp(P)) == {z E H² (( r)) r))
> 0 for all r > 0} be the support of P. Throughout the paper, we refer to supp(P) as support of P. or simply support, or manifold, but we don't 't necessarily require the (geometrical) manifold structure on (P)p(P)) Note that, when the support is high dimensional, what we can recover at most through estimation is the partial support. For kernel function K
Rd + R., dataset X' (XI..... Xn) R and bandwidth > O, we let the kernel density estimator (KDE) as (z) == nha Σi= K (*x and we let the average KDE as ph == E [.n]. We denote by P. Q the probability distributions in Rd of real data and generated samples, respectively. And we use X' = {X,..... X,. c R and y = (Y,...., Ym3 C F. for real data and generated samples possibly with noise, respectively.
Precision and Recall. There exist two aspects of generative samples' quality; fidelity and diversity. Fidelity refers to the degree to which the generated samples resemble the real ones. Diversity, on the other hand, measures whether the generated samples cover the full variability of the real samples. Sajjadi et al. 12 was the first to propose assessing these two aspects separately via Precision on and Recall (P&R). In the ideal case where we have full access to the probability distributions P and Q, precision p (Q) = Q (supp(P)) P)), recallo (P) == P (supp(Q)), which correspond to the max Precision on and max Recall in [12], respectively.
Persistent homology and diagram. Persistent homology is a tool in computational topology that measures the topological invariants (homological features) of data that persist across multiple scales, and is represented in the persistence diagram. Formally, let filtration be a collection of subspaces F {Fs C IR JoES with 61 S d2 implying Fs, Fbg. Typically, filtration is defined through function f related to data. Given function f: Rd - R. we consider its sublevel filtration (-xx, bEs or superlevel filtration {f [6, ]o)R For a filtration F and for each nonnegative k. track when k-dimensional homological features (e.g., O-dimension: connected component, 1-dimension: loop, 2-dimension: cavity,... appear and disappear. As 5 increases or decreases in the filtration (Fo), ). if homological feature appears F, and disappears at Fa, say that it is born at b and dies at d. By considering these pairs [(b,d) as points in the plane (IR 2 we obtain persistence diagram. From this, homological feature with longer life length, d - b, can be treated as a significant feature, and a homological feature with a shorter life length as topological noise, which lies near the diagonal line (6 8 E E (Figuree (b)).
Confidence band estimation. Statistical inference has recently been developed for TDA 25 26 TDA consists of features reflecting topological characteristies of data, and it is of question to systematically distinguish features that are indeed from geometrical structures and features that insignificant or due noise. To statistically separate topologically significant features from topological noise, we employ confidence band estimation. Given the significance level at let confidence band Ca be the bootstrap bandwidth of ||pm pull computed
in Algorithm [ (see Appendix H.2p. Then it satisfies lim = TP (1lé), Ph ll < c.) 2 1 - Ot, as in Proposition D.2 (see Appendix D). This confidence band can simultancously determine significant topological features while filtering out noise features. We use cx and ey to denote the confidence band defined under significance level a according to the datasets X' and V In later sections, we use these tools to provide a more rigorous way of scoring samples based on the confidence level we set.
3 Robust support estimation for reliable evaluation
Current evaluation metrics for generative models typically rely on strong regularity conditions. For example, they assume samples are well-curated without outliers adversarial perturbation, real or generative models have bounded densities, etc. However, practical scenarios are wild: both real and generated samples can be corrupted with noise from various sources, and the real data can be very sparsely distributed without density. In this work, we consider more general and practical situations, wherein both real and generated samples can have noises coming from the sampling procedure, remained uncertainty due to data or model, etc. See Appendix G.2 for more on practical scenarios.
Overview of our metric We design our metric to evaluate the performance very conservatively. Our metric is based on topologically significant data structures with statistical confidence above certain level. Toward this, we apply KDE as function f to define filtration, which allows
3

---

us to upproximate the support with data through { bo) Since the significance of data comprising the support is determined by the life length of homological features. we calculate ea that enables us to systematically separate short/long lifetimes of homological features. We then estimate the supports with topologically significant data structure via superlevel set f [Ca 00) and finally, we evaluate fidelity and diversity with the estimated supports. We have collectively named this process TRpP&R. By its nature, TopP&R is bounded and yields consistent performance under various conditions such as noisy data points, outliers, and even with long-tailed data distribution.
3.1 Topological precision and recall
To facilitate our discussion, we rewrite the precision in Section 2 as precisionp())) Q (supp(P) (uupp(Q)) /Q (supp(Q)) and define the precision of data points as
p))cision,())) = دز= Σj=1 1 (Y, supp(Q))
1 (Y,
which is just replacing the distribution Q with the empirical distribution 778 Σド= dy, of Y in the precision. Similarly,
recallo( = Ei= (X, E supp(Q) ssupp(P))
Ei= 1 (X, E supp(P))
In practice, supp(P) and supp(Q) not known priori and need to be estimated, and these estimates should be robust to noise since we allow it now. For this, we use the KDE ph., (z) =
-= of X and the bootstrap bandwidth ex of [ph.. - -hn, where h, > 0 and significance level E (0, (Section 2). Then, estimate the support of P by the superlevel set at cxf" as supp(P) Ph, oo), which allows filter out noise whose KDE values are likely to be small. Similarly, the support of Q is estimated: supp(Q) = [ey, 20), where @i... (x) =
and Σ K
1 mAd. ΣØ= K
( is the KDE of y and cy is the bootstrap bandwidth of l| It. loco
For the robust estimates of the precision, we apply the support estimates to precision (J) and recallo( (xt) and define the topological precision and recall (TopP&R) as
1= 1 (Y, E supp(P) (supp(())
TopP, x x\))
Σ/= 1 1Y, supp(Q))
Σ j=1 (pn., (Y₁) > c22 @n... (Yz) > ey)
(3)
Σ)= 1 (9)... (Y₁) > cy)
TopBy = E i=1 1 (Â) (X₂) > cy. ph, (Xi) > cx) Σ (ph. (X,) cx)
(4)
The kernel bandwidths hn and Itm are hyperparameters, and we provide guidelines to select the optimal bandwidths hn and fim in practice (See Appendix H.4).
3.2 Bandwidth estimation using bootstrapping
Using the bootstrap bandwidth ex as the threshold is the key part of our estimator (TopP&R) for robustly estimating sPp As we have seen in Section the bootstrap bandwidth ex filters out the topological noise in topological data analysis. Analogously, using cx allows robustly estimate sPpp(P). When X; is an outlier, its KDE value (X,) is likely to be small as well as the values the connected component generated by X,. So those components from outliers are likely to be removed in the estimated support P7 |cx, o0). Higher dimensional homological noises are also removed. Hence, the estimated support denoises topological noise and robustly estimates supp(P). See Appendix [ a more detailed explanation.
Now that we are only left with topological features of high confidence, this allows us to draw analogies confidence intervals in statistical analysis, where the uncertainty of the samples is treated
'The computation of and its practical interpretation is described in Algorithm

---

by setting the level of confidence. In the next section, we show that TopP&R not only gives a more reliable evaluation score for generated samples but also has good theoretical properties.
3.3 Addressing the curse of dimensionality
As discussed in Section 3.2 getting the bootstrap bandwidth cx with a theoretical guarantee plays a key role in our metrie, and the choice of KDE filtration function is inevitable, in Remark 44.4 However, computing the support of high-dimensional feature with KDE demands significant computation, and the accuracy is low due to low density values. This hinders efficient and correct evaluation in practice. To address this issue, we apply a random projection into a low-dimensional space by leveraging the random projection effectively preserves information regarding distances and homological features, composed of dileessionnnll features, in -i-dmemniional representation. Furthermore, we have shown that random projection does not substantially reduce the influence of noise, nor does it affect the performance of TopP&R under various conditions with complex data (Section and L.8.
Lemma (Lemma [B.1]. This lemma posits that using
4 Consistency with robustness of TopP&R
The key property of TopP&R is consistency with robustness. The consistency ensures that, the precision and the recall compute from the data approaches the precision and the recall from the distribution as we have more samples. The consistency allows to investigate the precision and recall of full distributions only with access to finite sampled data. TopP&R achieves consistency with robustness, that is, the consistency holds with the data possibly corrupted by noise, This is due to the robust estimation of supports with KDE with confidence bands.
We demonstrate the statistical model for both data and noise. Let P. Q, be as in Section and let X2 3 be real data and generated data without noise. X. 2 y are understood as multisets, i.e., elements can be repeated. We first assume that the uncorrupted data are IID.
Assumption 1. The data 2 = (XP... ...., X and yo (YP (Yp...... ..... Vo are IID from P and Q, respectively.
In practice, the data is often corrupted with noise. We consider the adversarial noise, where some fraction of data are replaced with arbitrary point cloud data.
Assumption Let {px kee be a sequence of nonnegative real numbers. Then the observed data X and y satisfies x2) = nPp and y mppm-
In the adversarial model, we control the level of noise by the fraction but do not assume other conditions such IID boundedness, to make our noise model very general and challenging.
For distributions and kernel functions, we assume weak conditions, detailed in Assumption A1 and A2 in Appendix D Under the data and the noise models, TopP&R achieves consistency with robustness. That is, the estimated precision and recall are asymptotically correct with high probability even if up to portion of 1/vn 1/vm replaced by adversarial noise. This is due to the robust estimation of the support with the kernel density estimator with the confidence band of the persistent homology.
Proposition 4.1. Suppose Assumption
A1
A2 hold. Suppose a - O. lon + O, nl - co.
nh, d 22 + O, and similar relations hold for hm Pm- Then,
[TopP x3) precision (3)) Op (Q(Bn.m) + Pm))
TopRy - recallo = Op (P(An,)) + Pm))
for fixed sequences of sets (An,m JammInmmer InMEN {Bam JUmmN with P(An,m) - and Q (Bn,m) + 0 as n,m Theorem 4.2. Under the same condition as in Proposition 4.1
Topp x precision p (Q) = Oz (Q( + pm)) TopRy - recall o (P) = O2 (P(An,m) + Pm))
Since P(An.m) - and (m + O, these imply consistencies of TopP&R. fact, additionally under minor probabilistic and geometrical assumptions, P(An,m) and Q(Bm,m) are of order hm + hn-
5

---

Lemma 4.3. Under the same condition as in Proposition 4.7 and additionally under Assumption A3
A4 P(An.m) = (h, + hm) and (B,m,,) = (h
Remark 4.4. Consistency guarantees from Proposition and Theorem 4.2 are in principle due to the uniform convergence of KDE over varying bandwidth ht, (Proposition D.2) Once we replace estimating the support with KDE by k-NN or something else, we wouldn't 't have consistency guarantees. Hence, using the KDE is an essential part for the theoretical guarantees of TopP&R
Our theoretical results in Proposition 4. and Theorem 4.2 are novel and important in several perspectives. These results are among the first theoretical guarantees for evaluation metries for generative models far are aware of. Also, as in Remark D.I assumptions are very weak and suitable for high dimensional data. Also, robustness to adversarial noise is provably guaranteed.
Experiments
A good evaluation metric should not only possess desirable theoretical properties but also effectively capture the changes in the underlying data distribution. To examine the performance of evaluation metrics, we carefully select a set of experiments for sanity checks. With toy and real image data, we check 1) how well the metric captures the true trend of underlying data distributions and 2) how well the metric resists perturbations applied to samples.
We compare TopP&R with several representative evaluation metrics that include Improved Precision and Recall (P&R) [33 Density and Coverage (D&C) 14, Geometric Component Analysis (GCA) 27 and Manifold Topology Divergence (MTD) 28 (Appendix F.. Both GCA and MTD the recent evaluation metrics that utilize topological features to some extent; GCA defines precision and recall based on connected components of /' and Q, and MTD measures the distance between two distributions using the sum of lifetimes of homological features. For all the experiments, linear random projection to 32 dimensions is additionally used for TopP&R, and the shaded area of all figures denotes the + standard deviation for ten trials. For fair comparison with existing metries, we have utilized 10k real and fake samples for all experiments. For more details, please refer to Appendix H.I
5.1 Sanity checks with toy data
Following [1]. we first examine how well the metric reflects the trend of y moving away from X' and whether it is suitable for finding mode-drop phenomena. addition to these, we newly design several experiments that can highlight TopP&R's favorable theoretical properties of consistency with robustness in various scenarios.
5.1.1 Shifting the generated feature manifold
We generate samples from X' ~ NO, I) and y ~ M(11, 7) in R&4 where 1 is vector of ones and is an identity matrix. We examine how each metric responds to shifting y with μ E [-1, 1] while there are outliers at 3 E R24 for both X' and y (Figure - We discovered that GCA struggles to detect changes using its default hyperparameter configuration, and this issue persists even after performing an exhaustive hyperparameter sweep. Since empirical tuning of the hyperparameters is required for each dataset, utilizing GCA in practical applications proves to be challenging (Appendix (F). Both improved P&R and D&C behave pathologically since these methods estimate the support via the k-nearest neighbor algorithm, which inevitably overestimate the underlying support when there are outliers. For example, when μ 0.5, Recall returns high-diversity score, even though the true supports of X and ) are actually far apart. In addition, P&R does not reach in high dimensions even when X y) D&C [1] yields better results than P&R because it consistently uses X' (the real data distribution) as a reference point, which typically has fewer outliers than y (the fake data distribution).
However, there no guarantee that this will always be the case in practice [17 [8]. If outlier is present in X, D&C also returns an incorrect high-fidelity score at μ > 0.5. On the other hand, TopP&R shows stable trend unaffected by the outlier, demonstrating its robustness.
5.1.2 Dropping modes
We simulate mode-drop phenomena by gradually dropping but one mode from the fake distribution ) that is initially identical to X (Figure Here, we consider the mixture of Gaussians with seven

---

Fidelity
Diversity
Outlier
Tight Pe Bu
Tight je Cick acail
0.2
02
0.0
OCA ponsaai
0.0
1.0
-055
130
U15
10
-100
+0.5
0,
1.0
Centerof of the fake distribinioyy
Centorof the fale distribmony
Figure 2: Behaviors of evaluation metrics for outliers real and fake distribution, For both real and fake data, the outliers are fixed at 3 1264, and the parameter μi is shifted from -1 to
distribution
Real distribution
Fake distribution
Fake distribution
Fidelity
Diversity
Fidelity
Diversity
1.4
1.0
Ronall
1.2
1.2
DCA secal
0.5
1.0
0.5
Topl
0.3
0.8
0.3
-11 Cowegg
0.0
0.6
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0,4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
Consectration on the first mode
Coscentration on the first msdde
Coscentraton on the first mode
Concentation on the first mode
(a) Sequential mode dropping
(b) Simultaneous mode dropping
Figure Behaviors of evaluation metrics for (a) sequential and (b) simultaneous mode-drop scenarios. The horizontal axis shows the concentration ratio on the distribution centered at = 0.
Fidelity
Diversity
Fidelty
Diversity
0.8
1.00
0.6
0.75
0.75
0.6
0.4
0.50
Tugt
Dessi So
0.25
02
0.0
0.00
0.00
0.0
0.00
0.0d
0.08
0.12
0.15
0.00
0.04
0.08
0.15
0.00
0.04
0.08
0.12
0.15
0.00
0.04
0.08
0.12 Q15
Ratio of oullies in thed (%5)
Ratio oullier dataa (%)
ol oullierin thass (%)
Ratio odliier (hediaa
Non- scalter noise
(b) Non-IID swap nokse
Figure 4: Behaviors of evaluation metrics Non-IID perturbations. We replace certain percentage of real and fake data (a) with random uniform noise or (b) by switching some of real and fake data.
modes in 64 We keep the number of samples in constant so that the same amount of decreased samples are supplemented to the first mode which leads fidelity to be fixed to 1. We observe that the values of Precision fail to saturate, i.e., mainly smaller than 1, and the Density ty fluctuates to a value greater than 1, showing their instability and unboundedness. Recall and GCA do not respond to the simultaneous mode drop, and Coverage decays slowly compared to the reference line. In contrast, TopP performs well, being held at the upper bound of in sequential mode drop, and TopR also decreases closest the reference line in simultaneous mode drop.
5.1.3 Tolerance to Non-IID perturbations
Robustness to perturbations is another important aspect we should consider when designing metric. Here, we test whether TopP&R behaves stably under two variants of noise cases (see Section G.3): 1) scatter noise: replacing X, and Y, with uniformly distributed noise and 2) swap noise: swapping the position between X; and Y,. These two cases all correspond to the adversarial noise model of Assumption 2 We set X Nn I) E R64 and y N(p 1, E R64 where μ = 1, and thus an ideal evaluation metric must return zero for both fidelity and diversity. In the result, while the GCA precision is relatively robust to the seatter noise, GCA recall tends to sensitive to the swap noise. In both cases, we find that P&R and D&C are more sensitive while TopP&R remains relatively stable until the noise ratio reaches 15% the total data, which is a clear example of the weakness of existing metrics to perturbation (Figure
7

---

Fidelit
Diversity
Fidelity
Diversity
Releiive Ub:
outile COBe
2:00
Topit
1.30
1.50
01.98
1.00
1.30
1.00
1.42
Conngrrk.ia
Tep
1.14
12.54
1.00
0.00
0.00 DCE
p.10 0.15 (,20 (U25
(L30
0.00
DCG 0.10 8.5 0.20 0.25 0.30
0.00
000 0.10 1515 0.20 0L25 0.30
0.00 006 n10 015 0.20 0.5 0.30
Ratio of oulber to inters (%)
Faso inliss (16)
Ratio of outber iitiees (%))
(a) Non-IID Scatter noise
(b) Non-IID Swap nolse
Figure 5: Comparison of evaluation metrics Non-IID perturbations using FFHQ dataset. We replaced certain ratio of X and y (a) with outliers and (b) by switching some of real and fake features.
Gausdaa noks
Gaussaa blur
Salt and pepper
(c) Black redtangle
Fl-smr
1.0
0.5
0.0
0.0
2.0 Naise intessity
1.0
3,000
1.0
2.0
3000
1.0
20
3,0000
1.0
2.0
30
Noise interaily
Nosse intensily
Noise intersily
Figure 6: Verification of whether TopP&R make accurate quantitative assessment of noisy image features. Gaussian noise, gaussian blur, salt and pepper. and black rectangle noise are added to the FFHQ images and embedded with T4096.
5.2 Sanity check with Real data
Now that we have verified the metrics on toy data, we move on to real data. Just like in the toy experiments, we concentrate on how the metrics behave in extreme situations, such outliers. mode-drop phenomena, perceptual distortions, etc. We also test different image embedders, including pretrained VGG16 22. Inception V3 33), and SwAV 31
5.2.1 Dropping modes in Baby ImageNet
We have conducted an additional experiment using Baby ImageNet [15 investigate the sensitivity of TopP&R to mode-drop in real-world data. The performance of each metric (Figure A4) is measured with the identical data while simultaneously dropping the modes of nine classes Baby ImageNet, in total of ten classes. Since our experiment involves gradually reducing the fixed number of fake samples until nine modes the fake distribution vanish, the ground truth diversity should decrease linearly. From the experimental results, consistent to the toy result of Figure 3 both D&C and P&R still struggle to respond to simultaneous mode dropping- In contrast, TopP&R consistently exhibit high level of sensitivity to subtle distribution changes. This notable capability of TopP&R can be attributed its direct approximation of the underlying distribution, distinguishing from other metrics. In addition, that TopP&R captures the trend well even when there are minority sets (Appendix [.I). This again shows the reliability of TRpP&R.
perform the experiments dataset with long-tailed distribution and find
5.2.2 Robustness to perturbations
To test the robustness of our metric against the adversarial noise model of Assumption 2 we test both scatter-noise and swap noise scenarios with real data (see Section G.3) the experiment, following Kynkäänniemi et al. [13. we first classify inliers and outliers that are generated by StyleGAN M. For scatter noise we add the outliers to the inliers and for swap noise we swap the real FFHQ images with generated images. Under these specific noise conditions, Precision shows similar or even better robustness than Density (Figure 3). On the other hand, Coverage is more robust than Recall. In both cases, TopP&R shows the best performance, resistant noise.
5.2.3 Sensitiveness to the noise intensity
One of the advantages of FID is that it is good at estimating the degrees of distortion applied to the images. Similarly, we check whether the Fl-score based on TopP&R provides a reasonable
evaluation according to different noise levels. As illustrated in Figure 6 and A5 X' and y are sets

---

Table 1: Generative models trained on CIFAR-I0 are ranked by FID, KID, MTD, and Fi-scores based on TopP&R, D&C and P&R, respectively. The X' and ) are embedded with V3 VGG16, and SWAV. The number inside the parenthesis denotes the rank based each metric.
Model
StyleGAN2
ReACGAN
BigGAN
PDGAN
ACGAN
WGAN-GP
FID (4)
3.78 (1)
3.87 (2)
4.16 (3)
31)
33.39 (5)
107.68 (6)
Inception
KID (L)
0.002 (1)
0,012 (3)
0.011 (2)
0.025 (4)
0.029 (5)
0.137 (6)
TopP&R (t)
0.9769 (1)
0.8457 (2)
0.7751 (3)
0.7339 (4)
0.6951 (5)
0.0163 (6)
D&C (f)
0.9626
0.9409 (3)
1.1562 (1)
0.4383 (4)
0.3883 (5)
0.1913 (6)
P&R (f)
0.6232 (1)
0.3320 (2)
0.3278 (3)
0.1801 (4)
0.0986 (5)
0.0604 (6)
MTD (4)
2.3380 (3)
2.2687 (2)
1,4473 (1)
7,0188 (4)
8,0728 (5)
11.498 (6)
TopP&R (t)
0.9754 (1)
0.5727 (3)
0.7556 (2)
0.4021 (4)
0.3463 (5)
0.0011 (6)
VGG16
D&C (1)
0,9831 (3)
1.0484 (1)
0,9701 (4)
0.9872 (2)
0.8971 (5)
0.6372 (6)
P&R (t)
0.6861 (1)
0.1915 (3)
0.3526 (2)
0.0379 (4)
0.0195 (5)
0.0001 (6)
MTD (1)
25.757 (4)
25.826 (3)
34.755 (5)
24.586 (2)
23.318 (1)
41.346 (6)
TopP&R (t)
0.9093 (1)
0.3568 (3)
0.5578 (2)
0.1592 (4)
0.1065 (5)
0.0003 (6)
SWAV
(T)(
1.0732 (1)
0,9492 (3)
1,0419 (2)
0.6328 (4)
0.4565 (5)
0,0721 (6)
P&R (t)
0,5623 (1)
0.0901 (3)
0,1459 (2)
0.0025 (4)
0.0000 (6)
0.0002 (5)
MTD (L)
1.1098 (1)
1.5512 (3)
1,3280 (2)
1.8302 (4)
2.2982 (5)
4.9378 (6)
of reference and noisy features, respectively. The experimental results show that TopP&R actually reflects well the different degrees of distortion added to the images while a similar topology-basee method MTD shows inconsistent behavior to the distortions.
5.2.4 Ranking between generative models
The alignment between FID (or KID) and perceptual evaluation has been well-established in prior research, and these scores are widely used as a primary metric in the development of generative models due to its close correspondence with human perception. Consequently, generative models have evolved to align with FID's s macroscopic perspective. Therefore, we believed that the order determined by FID at high level captures to some extent the true performance hierarchy among models, even if it may not perfectly reflect it. In other words, if the development of generative models based on FID leads to genuine improvements in generative performance and if there is meaningful correlation, similar rankings should be maintained even when the representation or embedding model changes. From this standpoint, while other metrics exhibit fluctuating rankings, TopP&R consistently provides the most stable and consistent results similar to both FID and KID. To quantitatively compare the similarity of rankings across varying embedders by different metries, we have computed mean Hamming Distance (MHD) (Appendix H.6) where lower value indicates more similarity. TopP &R, P&R, D&C, and MTD have MHDs of 1.33, 2.66, 3.0, and 3.33, respectively.
6 Conclusions
Many works have been proposed recently to assess the fidelity and diversity of generative models. However, none of them has focused on the accurate estimation of support even though it is one of the key components in the entire evaluation pipeline. In this paper, we proposed topological precision and recall (TopP&R) that provides systematical fix by robustly estimating the support with both topological and statistical treatments. To the best of our knowledge, TopP&R is the first evaluation metric that offers statistical consistency under noisy conditions, which may arise in real practice. Our theoretical and experimental results showed that TopP&R serves as a robust and reliable evaluation metric under various embeddings and noise conditions, including mode drop, outliers, and Non-III perturbations. Last but not least, TopP&R provides the most consistent ranking among different generative models across different embeddings via calculating its Fl-score.

---

Acknowledgements
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) NN..2.20057.011 Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2020-0-01336 Artificial Intelligence Graduate School Program (UNIST), No.2021-0-02068, Artificial Intelligence Innovation Hub, 22022-0-00999, (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making, Generation with Knowledge-band Deep Logic Neural Network).
Comprehensive Video Understanding and
References
[1] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[2] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8110 8119, 2020.
[3] Tero Karras, Miika Aittala, Samuli I Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 180. 2018.
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances Neural Information Processing Systems, 33:6840-6855 2020.
[6] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan large diverse datasets. arXiv preprint arXiv:2202.00273, 2022.
[8] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems, 34, 2021.
[9] Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. Advances in Neural Information Processing Systems, 33:21357-21369 2020.
[10] Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[11] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilihrium. Advances neural information processing systems, 30, 2017.
[12] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in Neurul Information Processing Systems, 31. 2018.
[13] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
[14] Muhammad Ferjad Nacem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jacjun Yoo. Reliahle fidelity and diversity metries for generative models. In International Conference on Machine Learning, pages 7176-7185. PMLR, 2020.
[15] Minguk Kang, Joonghyuk Shin, and Jaesik Park. Studiogan: taxonomy and benchmark of gans for image synthesis. arXiv preprint arXiv:2206.0944 09479, 2022.
[16] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. arXiv preprint 220.::2200000222 2022.
10

---

[17] Geoff Pleiss. Tianyi Zhang. Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the under the margin ranking. Advances in Neural Information Processing Systems, 13:17044-17056 2020.
[18] Zhengwen Li, Rummin Wu, and Tao Gan. Study on image data cleaning method of early esophageal cancer based on vgg nin neural network. Scientific Reports, 12(1):1-10 2022.
[19] Gumnar Carlsson. Topology and data. Bull. Amer. Math. Soc. (N.S.), 46(2):255 308 2009. ISSN 0273-0979.
doi: 10. 0011090902733-0099990--022299999
URL. httpa://doi.org/// 1090/
50273-0979-09 -9--9999999------
[20] Herbert Edelsbrunner and John L. Harer. Computational topology. American Mathematical Society, Providence, RI, 2010. ISBN 978-0-8218-49225 doi: 10. 10.1090/mbk/069. UtL/https::/dbl..orgg 1090/mbk/069 An introduction.
[21] Frédéric Chazal and Bertrand Michel. An introduction topological data analysis: Fundamental and practical aspects for data scientists. Frontiers Artif. Intell., 4:667963, 2021. doi: 10.3389/frai.2221.66999
httpp: org/11 10. 3389/frai www...2222..2211 667963
[22] Larry Wasserman. Topological data analysis. Annu. Rev. Stat. Appl.. 5:501-535, 2018. ISSN
2326-8298. doi: 10. 0tt444//aaauresssstiieee-- 100045. URL
1146/
annurey- wttp:e/ssttaatitictt110011------
[23] Allen Hatcher. Algebraic topology. Cambridge University Press, Cambridge, 2002. ISBN 0-521-79160-X 0-521-79540-0.
[24] Frédéric Chazal, Brittany Fasy, Fabrizio Lecci, Alessandro Rinaldo, Aarti Singh, Larry Wasserman. On the bootstrap for persistence diagrams and landscapes. Modelirovanie Analiz Informacionnyh Sistem,
20, 11 2013. doi: 10.18255/1811 111-11811111 1015-2013-66 111-120.
[25] Frédéric Chazal, Brittany Teresc Fasy, Fabrizio Lecei, Alessandro Rinaldo, and Larry Wasserman. Stochastic convergence of persistence landscapes and silhouettes. J. Comput. Geom., 6(2):140-611 2015.
[26] Brittany Terese Fasy, Fabrizio Lecci, Alessandro Rinaldo, Larry Wasserman, Sivaraman Balakrishnan, and Aarti Singh. Confidence sets for persistence diagrams. Ann. Statist. 220612230--2399, 2014. ISSN 0090-5364. doi: 14114/11--AS15555 http::/t:////o org/10 11114/444-01152
[27] Petra Poklukar, Anastasiaa Varava, Danica Kragic. Geomea: Geometric evaluation of data
representations. In International Conference on Machine Learning, pages 8588-8598. PMLR, 2021.
[28] Serguei Barannikov, Ilya Trofimov, Grigorii Sotnikov, Ekaterina Trimbach, Alexander Korotin, Alexander Filippov, and Evgeny Burnaev. Manifold topology divergence: a framework for comparing data manifolds. Advances Neural Information Processing Systems, 34, 2021.
[29] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: /409. 1556, 2014.
[30] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architccture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 8818-22 2826822,222 2016.
[31] Stanislay Morozov, Andrey Voynov, and Artem Babenko. On self-supervised image representations for gan evaluation. In International Conference on Learning Representations, 2020.
[32] A.W. der Vaart. Asymptotic Statistics. Asymptotic Statistics. Camhridge University Press, 2000. ISBN 9780521184504. URL htRppttts:/ //tp:ke..//o/eeee/eeeeeeoo http://oooooooooolooooo/oooo google http://oooeiiccc http://ooe....ddEEEEEeeeesssas
[33] Michael R. Kosorok. Introduction to empirical processes and semiparametric inference. Springer Series in Statistics. Springer. New York, 2008, ISBN 978-0-387-74977.5
doi: 10. 10.10077998800-3377779793 URL
httpa://doi. /ttp:/0000000//77----877----1100
[34] William B. Johnson and Joram Lindenstrauss. Extensions Lipschitz mappings into a Hilbert space. In Conference in modern analysis and probability (New Haven, Conn,, 1982), volume 26 of Contemp. Math. pages 189 206, Amer. Math. Soc., Providence, RI, 1984, doi: 10. 00.10900comm//0//777770000 URL https: org/ www0///omm00////7//00/0000

---

[35] Kasper Green Larsen and Jelani Nelson. The LiinssoonLLideestrauusss lemma is optimal for linear dimensionality reduction. In 43rd International Colloquiun on Automata, Languages, and Programming. volume 55 LIPlcs. Leibniz Int. Proc. Inform. pages Art. No. 82, 11. Schloss Dagstuhl. Leibniz-Zent. Inform., Wadern, 2016.
[36] Kasper Green Larsen and Jelani Nelson. Optimality the Codnson--indeestraunt lemma. 58th Annual IEEE Symposium on Foundations of Computer Science- 2017, pages 633-638. 638. IEEE Computer Soc., Los Alamitos, CA, 2017.
[37] Jisu Kim, Jachyeok Shin, Alessandro Rinaldo, and Larry Wasserman. Uniform convergence rate of the kernel density estimator adaptive to intrinsic volume dimension. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3398-3407. PMIR, 09-15 Jun 2019. URL Utt.tt //proceedings. mlr. pww.preeare /kimm/mmm........ html
[38] Michael H. Neumann. Strong approximation of density estimators from weakly dependent observations by density estimators from independent observations. Anm. Statist., 2665):2014-2048, 1998. ISSN 0090-5364. doi: 10.12 //14// 01/44/os/1022//0 1024691 367. WRW.tttttt/////... httpp:://oii. http://oo...gg....o
[39] Herbert Federer. Curvature measures. Transactions of the American Mathemattical Society, 93:418-492 1959. ISSN 0002-9947. doi: 10.2307/1993504, httpss//// httttts:://oii.rg///ttlt org/oo..//....... 111.111.3771999911
[40] Christoph Thäle. years sets with positive reach-a survey. Surv. Math. Appl., 3:123-165.
2008. ISSN 1843-7265. doi:
URI. htttpa//doi.orz orw.w/.......... 1007/
[41] Erik Carlsson, Gunnar Carlsson, and Vin De Silva. An algebraic topological method for feature identification. International Journal of Computational Geometry & Applications, 16(04):291-314, 2006.
[42] Frédéric Chazal, Brittany Fasy, Fabrizio Lecci, Bertrand Michel, Alessandro Rinaldo, Alessandro Rinaldo, and Larry Wasserman. Robust topological inference: Distance to measure and kernel distance. The Journal of Machine Learning Research, (8(1):5845-5884 2017.
[43] Hubert Wagner, Chao Chen, and Erald Vuçini. Efficient computation of persistent homology for cubical data. In Topological methods data amalysis and visualization II, pages 91-106. Springer, 2012.
[44] George R Terrell and David W Scott. Variable kerel density estimation. The Annals of Statistics, pages 1236-1265, 1992.
[45] Richard W Hamming. Error detecting and error correcting codes. The Bell system technical journal, 29(2): 147-160, 1950.
[46] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[47] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density- based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93- 104, 2000.
[48] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference data mining, pages 413-422. IEEF, 2008.
[49] Jiycon Han, Hwanil Choi, Yunjey Choi, Junho Kim, Jung- Woo Ha, and Jacsik Choi. Rarity score: A new metric to evaluate the uncommonness synthesized images, arXiv preprint arXiv:2206.085499 2022.
12

---

Appendix
A More Background on Topological Data Analysis
Topological data analysis (TDA) 19 is recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data. A key object in TDA is persistent homology, which quantifies salient topologiaal features of data by observing them in multi-resolutions.
A.1 Persistent Homology
Persistent homology is multiscale approach to represent the topological features. For filtration JF and for each Ng = Nu {0), the associated k-thh persistent homology PH+F is a collection of k-th dimensional homologies (o,.Fo equipped with homomorphisms (4 -HFF. HaFFoaas induced by the inclusion Fr C Fb.
Persistent homnology.
Persistence diagram. For the k-th persistent homology PH&F, the set of filtration levels which specific homology appears is always interval lb. d) [-00, oo). The corresponding k-th persistence diagram is a multiset of points (IR U {oc})² consisting of all pairs (b, d) where [6, d) is the interval of filtration values for which a specific homology appears in PH₃F.
A.2 Statistical Inference of Persistent Homology
As discussed above, a homological feature with long life-length is important information in topology while the homology with a short life-length can be treated as non-significant information or noise. The confidence band estimator provides the confidence set from the features that only include topologically and statistically significant (statistically considered as elements in the population set) under certain level of confidence. And to build confidence set, first need to endow metric the space of persistence diagrams.
Bottleneck distance. The most fundamental metric to measure the distance between two persistence
diagrams is the bottleneck distance.
Definition A.1. The bottleneck distance between two persistence diagrams D₁ and D2 is defined by
d D2) = inf sup ||p oo,)lo,,
7ET' pED;
where the set I' consists of all the bijections y D1 Diag + D2 Diag, and Diag is the diagonal <xx,x) E RR) C R22 with infinite multiplicity.
One way of constructing the confidence set uses the superlevel filtration of the kernel density estimator and the bootstrap confidence band. Let X' = (X1, X2, X,, as given points cloud, then the probability for the distribution of points can be estimated via KDE defined as following: (r()) = ndd compute Pr. and P/. which are the KDE of X and the KDE of bootstrapped samples X", respectively.
1 Σn K (=-x,) where h is the bandwidth and as a dimension of the space. We
Now, given the significance level a and h >> 0, let confidence band gx be bootstrap bandwidth of
Gaussian Empirical Process 32
33
Vnl Ip..
1% llx. Then it satisfies P(vn frr. ps. lloc
(x)= a, as in Proposition D.2 in Section D Then Bag (Pr, ex). the ball of persistent homology
centered at Pn, and radius cx = gx /n. in the bottleneck distance dp, a valid confidence set as lim inf, IP ( E Bag (Pr, cx) 2 a. This confidence set has further interpretation that in the persistence diagram, homological features that above twice the radius 2cx from the diagonal are simultancously statistically significant.
B Johnson-Lindenstraus. Lemma
JobhssonLLindenstie Lemma is stated as follows:
Lemma B.1
Lemma). (34 Lemma 1]
13

---

Let 0 < é < and & C IR be a set of points with size n. Then for d = O (min fn. D, log n))
there exists linear map f IR I such that for all x, y E X.
(1-e) 1 - y/l = 1lf(()) f(y)ll S (1 + |) ||=
(5)
Remark B.2. This Johnson-Lindenstrral Lemma is optimal for linear maps, and nearly-optimal even if we allow non-lineat maps: there exists set of points X' C IRD with size n, such that d should satisfy: (a) d = Q (E log for a linear map f RDD + IR satisfying (5) to exist [35] Theorem 3],
and (b) d = Ω (e-2 (o=()))))))) for a map J : RDD + IR satisfying 5 to exist 36 Theorem I].
C Denoising topological features from outliers
Using the bootstrap bandwidth ex as the threshold is the key part of our estimators TopP&R for robustly estimating supp(P). When the level set P7 [cx, oo) is used, the homology of [,x oo) consists of homological features whose (birth) 2 Cx and (death) Σ cx, which are the homological features in skyblue area in Figure AT In this example, consider three types of homological noise, though there can be many more corresponding to different homological dimensions.
There can be O-dimensional homological noise of (birth) < cx and (death) cx. which is the red point the persistence diagram of Figure A1 This noise corresponds to the orange connected component on the left. As in the figure, this type of homological noise usually corresponds to outliers.
There can be O-dimensional homological noise of (birth) >> ex and (deathi) > cx, which is the green point in the persistence diagram of Figure A1] This noise corresponds to the connected component surrounded by the green line on the left. As in the figure, this type of homological noise lies within the estimated support, not like the other two.
There can be 1-dimensional homological noise of (birth) < Cx and (death) < cx,, which is the purple point in the persistence diagram of Figure A1 This noise corresponds the purple loop on the left.
These homological noises satisfy either their (birth) cx and (death) < Cx or their (birth) cx and (death) > cx simultaneously with high probability, so those homological noises are removed in the estimated support p/ [cx, oo), which is the blue area in the left and the skyblue area in the right in [A]
We would like to further emphasize that homological noises are not restricted to O-dimension lying outside the estimated support (red point in the persistence diagram of Figure AT). 0- dimensional homological noise inside the estimated support (green point in the persistence diagram of Figure AII. 1-dimensional homological noise can also arise, and the bootstrap bandwidth cx allows to simultaneously filter them.
D Assumptions on distributions and kernels
For distributions, we assume that the order of probability volume decay P(B(I, is at least yd Assumption A1. For all x E supp(P) and E supp(Q),
P P(B(x,r)
Q (B((.))) r-0
lim inf
O, lim inf
> 0.
P-+
pd
rd
Remark D.I. Assumption Al analogous to Assumption 2 of Kim al. [37]. but is weaker since the
condition is pointwise on each æ E RRd. And this condition is much weaker than assuming a density on R. for example, distribution supported on a low-dimensional manifold satisfies Assumption A1 This provides a framework suitable for high dimensional data, since many times high dimensional data lies on low dimensional structure hence its density on Rd cannot exist. See Kim et al. [37] for a more detailed discussion.
For kernel functions, we assume the following regularity conditions: Assumption A2. Let Rdd + be a nonnegative function with ||]], = 1, and satisfy the following:
| =
|| < co.
14

---

Data and KDE levels
Persistence Diagram
15
Birth
0-dim
0-dim
0-dim
1-dim
Ca
1.5
Death
Figure A1: To robustly estimate the support, we use the bootstrap bandwidth to filter out topological noise (orange) and keep topological signal (skyblue). Then TopP&R is computed on this support.
(i) (())= >
(ii) K has a compact support.
(iii) K is Lipschitz continuous and of second order.
Assumption A2 allows to build a valid bootstrap confidence band for kernel density estimator (KDE). See Theorem 12 of (66 Theorem 3.4 of (38
Proposition D.2 (Theorem 3.4 of (8).. Let X (X1,..., X be HDD from a distribution P. For > 0, les ph,, Ph be kernel density estimator for and its bootstrap x", respectively, and for (0, 1), let cx be the bootstrap quantile from vnh lipn P% pll. For h + 0.
4+d
(Wnhh, lipn. Phnlloc cx =a
log nh!
Assumption A1 from 0.
A2 ensures that, when the bandwidth ht average KDEs bounded away
Lemma D.3. Let P be a distribution satisfying Assumption A1 Suppose K is nonnegative satisfying K(0)= and continuous Suppose {hn, JeE given with h, 2 0 and h. - 0. Then for all x E supp(P),
lim inf ph. (I) 0.
Proof. Since K(0)= 0 and K continuous at 0, there ro > such that for all y E (00,
K 2 2 (() > 0. And hence
(x)
/
hd
1
K
I I-
h
dP(w)
K(0),
2hd
(h h
(I-V
B(0 dP(x)
(00)
> 2hd
P (B(x)roh)))
Hence as hn + 0,
lim inf Ph. (x) O.
W-00c
15

---

Before specifying the rate of convergence, we introduce the concept of reach. First introduced by [9), the reach a quantity expressing the degree of geometric regularity of a set. Given closed subset A C Rdd the medial axis of A, denoted by (ed(A), is the subset of TR consisting of all the points that have at least two nearest neighbors on A.
Med(A) f E R A: 3q: + Q2 A, llq1 =|| 1l
x|| d(x, A)}
where d(x, A) = inf 1 - =|| denotes the distance from a generic point E I to A, The reach of A is then defined as the minimal distance from A Med(A).
Definition D.4. The reach of closed subset Rd defined as
reach( A)
inf inf d (g. Med(A)) gEA,rEMed(A) qEA
1lg x=|.
Now, for specifying the rate of convergence, we first assume that distributions have densities away from 0 and
Assumption A3. P and Q have Lebesgue densities p and q that, there exists 0 < Puuin I Pmmx < 00,
0 quin Imax < with for all E supp(P) and y E supp(Q).
Puin S p(x) S Pmax
9win I <(()) S qmax-
We also assume weak geometric assumptions on the support of the distributions supp(P) and supp(Q), being bounded and having positive reach. Assumption A4. We assume supp(P) and supp(Q) are bounded. And the support of P and Q have positive reach, i.e. reach (supp(P)) 0 and reach (sach(sspp(Q)) 0.
Sets with positive reach ensure that the volume of its tubular neighborhood grows in polynomial order, which is Theorem 26 from [40] and originally from [3)]
Proposition D.5. Let A be set with reach( ()) 0. Let A, fa Rd d(x, A) S r). Then for r < rraah(A)) there exists au....... E R U }oo} that satisfies
d-1
Ad = Σ arr
Amm0
where >d is the usual Lebesgue measure Tf..
E Details and Proofs for Section
For random variable X and a E (0, 1), let qx.a be upper quauntilee of X, i... P(X 2 9.) cquivalently, PXX < )a) 1 - Let pr. be the KDE on X For finite set X, we use the notation Cx.a for a-bootstrap quantile satisfying TP ix.,.. - Px* x < Cx X') = - a, where X is the bootstrap sample from X. For distribution P,
= or
the notation CP,.. for a-quantile satisfying P (Ilpr.. Ph Il < (P.a) = 1 - a, where pr., is kernel density estimator of IID samples from P. Hence when X' is not IID samples from P. the relation of Proposition| D.2 may not hold.
Claim E.1. Let Z N 1) be sample from standard normal distribution, then for a (0, 1),
12.a = θ (Vlog(1/a))
And for 0 0
(Bay log//o)(1/a))
gZ.a 02,0+6
Proof. Let NO, 1), then for all 2,
x²
x²
2x cxp
<
P(Z >
x
< exp
2
2
Then = C. [og( gives
C/2
a C/2
<
P(2 x)
2C (log(////))
(//((
16

---

And hence
12,a = =6 (Vlog((1/a))
and in particular,
22,0
= (avlog(1/o))
exp
Now for 0
92,a
ta
(92,a
exp
exp
dt = ô
2
12,a+8
a2,00
I (az,a 42,0+8) exp
And hence
a2,0+
= (5((x ( (o(a+))/oog(1/))
17,a (2,0088 S Śdexp
2
and
$3,0
92.a - 97,00 2 Sexp
=
Then from d a.
22,a a2,a+0 = 0 (00o//1o((//a))
Claim E.2. Let X, Y be random variables, and 0 5 < Suppose there exists e satisfying
P( - Y > c) <
Then,
9x,+05 - C aY S ax +
Proof. Note that Qx.a-5 satisfies P(X > (x) = 8. Then from 6
(V 9x (Y 8x 8 G, X Y) P(X YJ c)
<( > 4x + =a
And hence
gY 8x 8 +
Then changing the role of X and Y gives
gx,+88 5 a.
Lemma E.3.
(i) Under Assumption
and A2
Pn |||||
p pn aoo
hd
(ii) Under Assumption 7 and A2
with probability 1 8,
Cxo,a065 -O
Pn hd
Pu. log(1/8) nh.dd
Pu
Pn log(1/5) nh.2d
+
S Cx.a S cx +0
hd +
(ii)) Suppose Assumption 7 ald 442d
A2 hold, and let a, on e (0,1). Suppose nh. 0o, 8,
((
and nh d 12,5 0. Then with probability 48n-
pr polloc < Cx.. CP.a-3...
17

---

Proof. (i)
First, note that
Xi
(-- x? )
(1) (n)
K
K
nh!
Then under Assumption A2
X;
(-xxx))
-- X
IPA - Ph 10c x
K
- K
nhd
h
h
i=1
1 nhd
| oo
I
(X4 + x!!
Then from Assumption 2
Σi= (X: & x S npn. and hence
| 1l Pa
1pA. pol S
hd
(ii)
Let X, X2 be bootstrapped samples of X. X with the same sampling with replacement process.
Let ph. Ph be KDE of 26 and X. respectively. And, note that
pr phll - llōn - ph llol S ll polloo + |12 ph bo
Let Lg be the number of elements where A' and Xg differ, i.e., Lo = 1 1661 = |x then
La ~ Binomial (n, Pr)), and
||]] Tob nhd
And hence,
| (npp La)
|1 Ph - ||ān - Ph bl S
nhd
Then by Hoeffding's s inequality, with probability
8.
n llg((1/0)
Ls S npn
2
by applying this to 7 with probability 1 1- 8,
p ph - || 1111 50
Pn hdd
+
Pn log(1/6) 1/6) nh2d
Hence applying Claim
E.2 implies that, with probability 1-
Pri 1d
log (og(1/)) 11/2d
(ooooo(()) n2/2d
Cx
0
Cx S Cx9
hdd
(iii)
Let 5, = (
Log of
be from RHS of Proposition D.2 then for large enough 6, 2 8,.
Note that Proposition
D.2 implies that for all a E (0,
CP.tt S CXo.a S CP.a-
(8)
Now, nh. - oo implies that / /nhg (ph.. Phh.) converges to Gaussian process, and then Claim E.2
implies that Cp = (v
log( and nhd
18

---

log( log(1/a)
CPaa - CP.a+5. = 8n
(9)
nhe
Then under Assumption 2 since nhn 22,0, o(1) and npn 2 1 implies p,o00 o(1). and then
Pn h
Pn. log((16)
=0 bnay
log(1/a) nhe
(10)
nh?
Then
8 9.. 10p
implies that
-o
0 (dnogy
1og(1/a) nhe
I Cxo, -0
Pn. hed
Pro log((1 (1/))
C8 S CP,
+
(11)
nh?
and
a- +0
Pn
Pn log(1/6) nh2
cP.a-28, +0 bnay
log(1/a) nhe
+
C..a-33n.. (12)
he
Now, from the definition of Cp,0++35, with probability 1 38nr
|| -Palloc S CCP.++35..
(13)
And (ii) implies that, with probability 1 1-dra 8n.
Cx ,c
(
Pn
+
On log(1/6) nh?dd
S Cx2a S ax +0 hd
Pn
Pn. log(1/0)
(14)
hd
nd?d
Hence by combining
14
with probability a 46
pp
pulloo
Cp,
S cx -0
Pr
On (1/))
+
hdd
nh2d 2d
S Cx
S Cx Cx 6n +0
Pn
+
Pn log(1 /6) nh?d
hdd
< Cp.a-36.'
Corollary E.4. Suppose Assumption
A2 hold, and let a E (0, 1). Suppose on
O
logn ahe
4+2d
and nh -d p 5,5 5-2 - Then with probability 1 - - 45m
(i) Let dn (0,1) and suppose nld - xc, 5,1 = 0 --t=
4+d
log nnd
and nh dl 12,5
Then with probability 1 44.
Ph. (2cp.0 38,,, 00) C (cx 0o) C Supp((Pr..)
(ii) Let 8m E (0, 1) and suppose mhd + 0o, 8p =0 (
4+dd
log mhut
4422
and mh - d p2 8m
0. Then with probability 1 48m-
00) 9h (20Q,a-35... 20) C gill ao 00) C supp(QQ.....
19

---

Proof. (1)
Lemma FL.3
(iii) implies that with probability 1 46n, ll phll cx CP.a-36.- This
implies
Ph.. [20p -33.: 00) C lex
I 00) C supp(Phn.).
(ii)
This can be proven similarly to (i).
Claim E.S. For a nonnegative measure μ and sets A, D,
B)AnB) - (CC = +
Proof.
S (AAA ((CAAn))CC)U((nBB))) ( +)AABB)))
From here, let Pn. and Qm be the empirical measures on X' and V respectively, i.e., P. D dx, and Qm 2 Sy
Lemma E.6. Suppose Assumption 7 hold. Let A C Rd.
Then with probability 1 - 8,
log(2/6)
|Px - PI S P. +
2n
and in particular,
log(2/0)
(a(A) S P(A) + Pn +
Proof.
Let p0 be the empirical measure on X i.e., p 1 n
6xp- By using Hoeffding's
inequality,
P ( P) (A) 2 t) < 2 (-2nt²).
and hence with probability 1-
log(2/8)
p - (A)
2n
And Pa -Po (A) is expanded as
1 n n
Pn Pol (A)
Σ E) A)I
Under Assumption 2. Ei= I (X: # x
S npn, and hence
1 Σ |1(x E A) 1(XP E A)
Pn - p.l (A)
1
m
<
Σ
T [X, # x²) Pn-
n
i=1
Therefore, with probability 1-8 8,
log(2/8)
IP, - (A) < Pn +
2n
20

---

Claim E.7. Suppose Assumption
1 A1
A2
hold, and let ò om E (0,1). Suppose nh +
mhn
7 00. _m² O
log nh
dm
O
log in mih
nhn"" p2, + 0, and
nhim P pmn0m 6m
O. Let
Ba. (supp(P)))) [2cp, x0)) U Ug (0, 200) U Usupp(Ph..) (suppp)))))))
(15)
ii With probability 1 - 2a - 46, - 88m.
10m (Ph, (ox,
00) r 9rmm loy, 00)) - Qm (supp(P) (sspp(Q)))
log(1/6)
c Q( + Pm +
(ii) With probability 1 - 20 80m.
Qm (fhm [cy, oc)) - Qm (supp(Q))) sc (o (9 (0, (00)) + Pm +
1og(1/6)
(iii) With probability 1 - 2a - 48,, - 95m.
Qm lex,
lex, 00) n arul ley, z0)) Q (supp(
s c Q( + Pm + V log(1/0)
(iv) With probability 1 - 2xx 90mm
(0, (00)) (
log(1/6)
1Qm (Trn (oy, 00)) - - ((
+ Pm +
(v) As n. > 0o, Ban + 0. And in particular,
Q(Bam) + 0.
Proof. (i)
From Corollary E.4 (i) and (ii), with probability 2a 4(6n + 8m))
Qm (PT.. [2cp, oo) 9hm [2cQ. oc)) S Qm (Pm, [cx, oc) n 9 [ey, oo))
< Q (supp Ph..) n supp (Qnm))
(16)
Then from the first inequality of (16), combining with Claim
E.5
gives
Qm (pmi) [cx, 0o) n बull [cy, oo)) - Qm (
2 Qm (Ph,, [2cp, oc) n 9h
-1 [2cq, oo)) Qm (supp (Ssupp())Ssupppppppp nsupp(Q))
2 (Qm (supp( PT. [2cr, oo)) Qm (supp(Q) / 1eg x)))
(17)
And from the second inequality of (16), combining with Claim E.5
gives
Qm (Pm. [cx, o0) N.ll [cy, oo)) - Qm (P) n supp(Q))
- Qm (supp( PA..) n supp(Q - Qm (supp n supp(())
< Qm (supp( Ph..) (supp( (P) + Qm (suup((Q....)suppppppp (ssppp())))))
(18)
And hence combining (1) and 18 gives that, with probability 2a - 4(6, + 6m).
1Qm (pn,) [cx, oo) n Tr (cy, oo)) - Qm (supp(P) (sspp(())
< max ( (2m (supp (P) (. [2cr, 2o)) Qm (supp(Q))\r.!! Qm (supp )(sppp()) + Qm (supp(Qn..) (eap((((n..)sup(())) }(supp(Q)))
(19)
21

---

Now we further bound the upper bound of (I9). From Lemma E.6 with probability -
8m.
log(2/8)
Qm (supp(P)\rr.) (2cp, ool) s Q (supp(P) PN)Ph, [2cp, 20)) + Pm + And similarly, with probability 1 dm
(20)
2m
(2g(2/6)
Qm (supp(Q))nn,, 1200, oo)) 2 (supp(Q)\(n.!! 1200, oo)) + Pm +
2m
= Q (9T. (0) + Pm + log(2/0)
(21)
2m
And similarly, with probability 1 5m
log(2/8)
@m (supp( (h.pp(P..)) (P,,,), (sspp(()) S S (ssuup(Ph.))suuup(( + Pm + V
(22)
2m
And similarly, with probability 8m
log(2/8)
Qm (aapp(Qr...nsuupplle s (supp(Qin.)suuppp((() + Pm.
2m
log(2/0)
Pm +
(23)
2m
Hence by applying 20 21p.
to
with probability 2a 48,
86m-
Qm (Pm, [ex, 20) n Trm
Cyy, oc))
Qm (supp(P) (supp(()))
C (Q (suppp( (Popp(()eee)) PM (2cp, oo)) U Ih (0, 2nQ) Usupp((P.p... \s.,,)))upp((P))
1og(1/0)
+ + v
m
-c Q(Bm.()))) Pm
log(1/8)
m
where Br,m is from 15
(ii)
This can be done similarly to (i).
(iii)
Lemma E.6 gives that with probability 8m
log(2/8)
1mm (supp(P) suppp())) (supp(P)) < Pm +
2m
Hence combining this with (i) gives the desired result.
(iv)
Lemma E.6 gives that with probability 1 - 1-8m
log(2/8)
Qm (supp(Q)) - S Pm + Hence combining this with (ii) gives the desired result.
2m
(v)
Note that Lemma D.3 implies that for æ supp(P), lim inf Ph. (x) 0, Ph. (x) > 2cp for large enough n. And hence as - oc,
supp (P) [2ep, xc) + 0.
(24)
22

---

And similar argument holds for dupps Q7 22cq. 00), so as m - 00.
(upp(Q)(r,
+ 0.
(25)
Also, since K has compact support, for any x supp (P), supp(Ph..) once hn < ((2,,supp)) P)).
Hence as n + 0o,
supp (Mpp(Pr.))supppa Ssupp (P) -
(26)
And similarly, as m + 00,
Qup suppp(Q) 0.
(27)
Hence by applying
to the definition of Bn,m in 15] gives that n, m + 00,
Bu. 0.
And in particular,
Q(Bum) + 0.
Below we state more formal version of Proposition 4.1
Proposition E.8. Suppose Assumption
AZ A2 hold. Suppose a + 0, 8n -
0, 6
hn - O, nh + oc, and nh," ",,,5 + O, and similar relations hold for Umm Pm- Then there exists some constant C not depending on anything else such that
log 11 O
log(1/6)
TopPxx(2) precision c ((mmm) + +m +
m
log(1/0) n
TopRy(X)) recallo S P(Anm) + Pn +
where
Anm = (supp(Q)) (imi (eq oc)) U Ph. (0, 2cp) Supp((hh))))))) (uppp()))))supp(() Ba == (supp( P)) (P 2cp, xc)) U Thm -1 (0, 2cQ) U suppp( Msuppp)))
And n, m CO, P(An.m) 0 and Q (m,.m) hold.
Proof of Proposition E.8
Now this an application of Claim E.7 (i) (v) the definitions of
()) and recallo(2)) (X)) ( in T and 2 and the definitions of TopP x (V) and (X) in and
Similarly, we state a more formal version of Theorem 4.2
Theorem E.9. Suppose Assumption A2 hold. Suppose a - 0, 8n +
0, 6
4nd 4422
logn nh!!!
hn - 0, nhd + oc, and nh,, 12.0 + O, and similar relations hold for "
hm Pm- Then there exists some constant C not depending on anything else such that
[lo
TopP precision,(()) c Q(Bm,m) + Pm +
m
log(1/8)
TopRy recallo (reall())reaall((()) SC
(((nm) + Pn +
n
where and Bu are the same as Proposition E.8 Q(Bm,m) + hold.
Again as n, m - oo, P(An..) - 0 and
23

---

Proof of Theorem E.9
Now this is an application of Claim E.7 (iii) (iv) (v) to the definitions of
precision,(() and recallo (P) and the definitoons of TopP, and Ty) in (3) and
Proof of Lemma 4.3 Recall that Ba,m is defined in 15 as
Bu.m (supp(P) (PT) (2ep, oc)) Th... (0, 2cQ) Usupp(Ph.)) (supp(())
and hence Q(B be upper bounded
((n.,)) S Q (supp(P)) PT.., [2cp, 1 oo)) + Q (Thm (0, 2cQ)) + Suuppp(r...suppppp (28)
For the first term of RHS of (28). suppose supp(K) K) (R, 1) for convenience, and let
Ahh, [I E supp(P) d(a;, R (upp()(())) h- Then from Assumption A3 for all A-h..
Pho (x) 2 Pmin holds. Hence for large enough N such that for 2 N. cp
Pmins then
Ph. (Ø) 2 Puin for all x E A-h,, and hence
Q (supp(P))PP,
(2cp, 20)) & Q (supp(P) PM))).))
(29)
Also, (P) being bounded implies that the coefficients do, dd-1 in Proposition D.5 for supp(P) P) are in fact finite. Then y S Imax from Assumption A3 and Proposition D.5 implies
Q (ssup(()) = (hr)).
(30)
And hence combining 29 and
30 gives that
Q (supp(P) PT. [2ep, 12ep, oo)) = ((w))
(31)
For the second term of RHS of 28 with a similar argument,
(9 (0,2ccq) (hm))
(32)
Finally, for the third term of RHS of 28 Proposition D.5 implies
Q (supp( Px.,)) (supp)) P)) Oh,,).
(33)
Hence applying 31..
32
33 to
28
gives that
Q(Bmm O(hn, + hm).
F
Related Work
Improved Precision & Recall (P&R). Existing metries such IS and FID assess the performance of generative models with a single score while showing usefulness in determining performance rankings between models and are still widely used. These evaluation metries have problems that they cannot provide detailed interpretations of the evaluation in terms of fidelity and diversity. Sajjadi et al. [12] tried solve this problem by introducing the original Precision and Recall, which however has limitation as it is inaccurate due to the simultaneous approximation of real support and fake support through "k-means clustering". For example, if we evaluate the fake images having high fidelity and large value of k, many fake features can be classified in small cluster with no real features, resulting in a low fidelity score (precisionp == (suupp( (P)). Kynkääänniemi et al. [33 focuses on the limitation from the support estimation and presents an P&R that allows us to more accurately assess fidelity and diversity by approximating real support and fake support separately:
M
precision x.))
1
Σ J(Y,, recall(X,))
Σ J(X,,)),
M
N
where J(Y,, [1, Y, E supp(P),
00 otherwise.
Here, supp(P) ()) is defined as the union of spheres centered on each real feature X, and whose radius is the distance between kth nearest real features of X;
24

---

Density& & Coverage (D&C), Naeem et al. [14] has reported that P&R cannot stably provide accurate fidelity and diversity when real or fake features are present through experiment. This limitation of P&R is that it has a problem with approximating overestimated supports by outliers due to the use of "ee-ntrestt neighborhood" algorithm. For a metric that robust to outlying features, Naeem et al. [14] proposes new evaluation metric that only relies on the real support, based on the fact that a generative model often generates artifacts which possibly results in outlying features in the embedding space:
M N
N Σ
density(2 = M 3
1
Σ
1 1y ((X;), coverage(X,))
= N as a.t. Y,e ((X))
where S(X:) N(D,
In the equation, (X.) means the kth-nearest neighborhood distance of X; and [X,,, NN,NNNDD (X,)) denotes the hyper-sphere centered at X; with radius (X,). However, D&C is only partial solution because it still gives inaccurate evaluation when real outlying feature exists. Unlike coverage, density is not upper-bounded which makes it unclear exactly which ideal score generative model should achieve.
Geometric Evaluation of Data Representations (GCA), Poklukar et al. 27 proposes a metric called GCM that assesses the fidelity and diversity of fake images. GCM uses the geometric and topological properties of connected components formed by vertex V (feature) and edge (connection). Briefly, when the pairwise distance between vertices is less than certain threshold e, connected component is formed by connecting two vertices with an edge. This set connected components can be thought of as graph G (V.E), and each connected component separated from each other in G is defined as a subgraph 9; (i.e., g = U,G.). GCA uses the consistency c(G.) and the quality ((G,) to select the important subgraph Gú that is formed on both real vertices and fake vertices (v' X U )). <(Ge) evaluates the ratio of the number of real vertices and fake vertices and ((G)) computes the ratio of the number of edges connecting real vertices and fake vertices to the total number of edges in Gi. Given the consistency threshold Te and quality threshold f2, the set of important subgraphs is defined as S(Bes (g( <G.)>> n. Gi. Based on this, GCA precision and recall are defined as follows:
|S 1/ 1 v
S lv 1
GCA precision
GCA recall
In above, Sd and g represent subset (V v,E)) of each graph S and G, respectively. One of the drawbacks is that the new ppper-parameters c, Ye, and 7) need to be set arbitrarily according to the image dataset in order to evaluate fake images well. In addition, even if we use the that seem most appropriate, it difficult to verify that the actual evaluation results are correct because they do not approximate the underlying feature distribution.
Manifold Topology Divergence (MTD). Barannikov et al. 28 proposes a new metric that uses the life-length of the k-dimensional homology of connected components between real features and fake features formed through Vietoris-Rips filtration [41). MTD is simply defined as the sum of the life- length of homologies and is characterized by an evaluation trend consistent with the FID. Specifically, MTD constructs the graph by connecting edges between features with distance smaller than the threshold c. Then the birth and death of the k-dimensional homologies in G are recorded by adjusting the threshold from to co. The life-length 1, (h) is obtained through death; birth; of any k-dimensional homology h, and let life-length set of all homologies as L(h). MTD repeats the process obtaining (hh) on randomly sampled subsets X" E X' and y" E y each iteration, and defined as the following:
MTD(X.)) 1 n
L.(h)) where L.hh) is the life-length set defined at ith iteration.
However, MTD has a limitation of only considering a fixed dimensional homology for evaluation. In addition, MTD lacks interpretability as it evaluates the model with single assessment score.
25

---

G Philosophy of our metric & Practical Scenarios
G.1 Philosophy of our metric
All evaluation metrics have different resolutions and properties. The philosophy of our metric is to propose reliable evaluation metric based on statistically and topologically certain things. In a real situation, the data often contain outliers or noise, and these data points come from a variety of sources (e.g. human error, feature embedding network). These errors may play outliers, which leads to an overestimation of the data distribution, which in turn leads to false impression of improvement when developing generative models. As discussed in Section 2 P&R and its variants have different ways to estimate the supports, which overlook the possible presence and effect of noise. Naeem al. [4] revealed that previous support estimation approaches may inaccurately estimate the true support under noisy conditions, and partially solved this problem by proposing to only use the real support. However, this change goes beyond the natural definition of precision and recall, and results in losing some beneficial properties like boundedness. Moreover, this is still a temporary solution since real data can also contain outliers. We propose a solution to the existing problem by minimizing the effects of noise, using topologically significant data points and retaining the natural definition of precision and recall.
G.2 Practical Scenarios
From this perspective, we present two examples of realistic situations where outliers exist in the data and filtering out them can have a significant impact on proper model analysis and evaluation. With real data, there are many cases where outliers are introduced into the data due to human error
[8] Taking the simplest MNIST as an example, suppose our task of interest is to generate Since image number 7 is included in data set number 4 due to incorrect labeling (see Figure I of [17), the support of the real data in the feature space can be overestimated by such outliers, leading to an unfair evaluation of generative models (as in Section 5.1.2 and 5:2.2): That is, the sample generated with weird noise may be in the overestimated support, and existing metrics without taking into account the reliability of the support could not penalize this, giving good score to poorly performing generator.
A similar but different example is when noise or distortions in the captured data (unfortunately) behave adversely on the feature embedding network used by the current evaluation metrics (as in Section (51.2] and 5.2.2); e.g., visually it is the number 7, but it is mismapped near the feature space where there are usually and becomes an outlier. Then the same problem as above may occur. Note that in these simple cases, where the definition of outliers is obvious with enough data, one could easily examine the data and exclude outliers priori to train generative model. In the case of more complicated problems such as the medical field [18], however, it is often not clear how outliers are to be defined. Morcover, because data are often scarce, even outliers are very useful and valuable in practice for training models and extracting features, making it difficult to filter outliers in advance and decide not to use them.
On the other hand, we also provide an example where it is very important to filter out outliers in the generator sample and then evaluate them. To evaluate the generator, samples are generated by sampling from the preset latent space (typically Gaussian). Even after training is complete and the generators' outputs are generally fine, there's a latent area where generators aren't 't fully trained. Note that latent space sampling may contain samples from regions that the generator does not cover well during training ("unfortunate outlier"). When unfortunate outliers are included, the existing evaluation metrics may underestimate or overestimate the generator's 's performance than its general performance. (To get around this, it is necessary to try this evaluation several times to statistically stabilize it, but this requires a lot of computation and becomes impractical, especially when the latent space dimension is high.)
Especially considering the evaluation scenario in the middle of training, the above situation is likely to occur due to frequent evaluation, which can interrupt training or lead to wrong conclusions. On the other hand, we can expect that our metric will be more robust against the above problem since it pays more attention to the core (samples that form topologically meaningful structures) generation performance of the model.
26

---

G.3 Details of the noise framework in the experiments
We have assessed the robustness of TopP&R against two types of non-lID noise perturbation through toy data experiments (in Section 5.1.3) and real data experiments (in Section (32.2). The scatter noise we employed consists of randomly extracted noise from a uniform distribution. This noise, even when extensively added to our data, does not form a data structure with any significant sigmal (topological feature). In other words, from the perspective of TopP&R, the formation of topologically significant data structures implies that data samples should possess meaningful probability values in the feature space. However, noise following a uniform distribution across all feature dimensions holds very small probability values, and thus, it does not constitute a topologically significant data structure. Consequently, such noise filtered out by the bootstrap bands cx' or ey that we approximate (see Section 2)
The alternate noise utilized in our experiments, swap noise, possesses distinct characteristics from seatter noise. Initially, given the real and fake data that constitute important data structures, we introduce swap noise by randomly selecting real and fake samples and exchanging their positions. In this process, the swapped fake samples follow the distribution of real data, and conversely, the real samples adhere to the distribution of fake data. This implies the addition of noise that follows the actual data distribution, thereby generating significant data structures. When the number of samples undergoing such positional swaps remains small, meaningful probability values cannot be established within the distribution. Statistically, an extremely small number of samples cannot form a probability distribution itself. As a result, our metric operates robustly in the presence of noise. However, as the count of these samples increases and statistically significant data structures emerge, TopP&R estimates the precise support of such noise as part of their distribution. By examining Figure Table A12 it becomes evident that conventional metrics struggle with accurate support estimation due to vulnerability to noise. This limitation results in an inability appropriately evaluate aspects involving minority sets or data forming long distributions.
well as
G.4 Limitations
Since the KDE filtration requires extensive computations in high dimensions, a random projection that preserves high-dimensional distance and topological properties is inevitable. Based on the topological structure of the features present in the embedded low dimensional space, we have shown that TopP&R theoretically and experimentally has several good properties such as robustness the noise from various sources and sensitivity to small changes in distribution. However, matching the distortion from the random projection the noise level allowed by the confidence band is practically
The bootstrap confidence band is of order (nhg)= and hence for the distortion from the random projection to match this confidence band, the embedded dimension should be of order Ω (nha, l) from Remark (B.2). However, computing the KDE filtration in dimension Q (nhg, a logn) is practically infeasible. Hence in practice, the topological distortion from the random projection is not guaranteed to be filtered out by the confidence band, and there is possibility of obtaining a less accurate evaluation score compared to calculations in the original dimensions.
infeasible:
Exploring the avenue of localizing uncertainty at individual data points separately presents an intriguing direction of research. Such an approach could potentially be more sample-efficient and disregard less data points. However, considering our emphasis on preserving topological signals, achieving localized uncertainty estimation this manner is challenging within the current state of the art. For instance, localizing uncertainty in the kernel density estimate P is feasible, as functional variability is inherently local. To control uncertainty at a specific point xr we primarily need to analyze the function value p(x) at that point. In contrast, localizing uncertainty for homological features situated at point æ requires the analysis of all points connected by the homological feature. Consequently, even if our intention is to confine uncertainty to a local point, it demands estimating the uncertainty at the global structural level. This makes localizing the uncertaintly for topological features difficult given the tools in topology and statistics we currently have.
27

---

H Experimental Details
H.1 Implementation details of embedding
We summarize the detailed information of our embedding networks implemented for the experiments. A4 and P&R and D&C are computed from the features of ImageNet pre- trained VGGI6 (fc2 layer), and TopP&R is computed from features placed in IR with additional random linear projection. In the experiment in Figure A10 the SwAV embedder is additionally considered. We implement ImageNet pre-trained InceptionV3 (fe layer), VGG16 (fe2 layer), and SwAV embedding networks with random linear projection to 32 dimensional feature space to compare the ranking of GANs in Table I
In Figure 3 4
A10
H.2 Implementation details of confidence band estimator
Algorithm 1 Confidence Band Estimator
# KDE: kemel density estimator
grid scarch for the confidence band
# h: kernel bandwidth parameter
for q E [min(f) (0), max(f)
# number of repeats
count = 0
# Ô: set of difference
for element E 0 do
if element > q then cownt count
Given x= (X1, X2, X Dh KDE(X')
end if end for
for iteration = 1, 2, do
# Define X with bootstrap sampling
define the band threshold
X = random sample times with repeat from X
if count/k then
# Ph replaces population density
9a. = end if
P7 KDE(X"")
# compute 0 with bootstrap samples
end for
Append 0 with pr |/
define estimated confidence band
end
Ca /a/////
H.3 Choice of confidence level
For the confidence level a, we would like to point out that a is not the usual hyperparameter to be tuned: It has statistical interpretation of the probability the level of confidence allow error, noise, etc. The most popular choices are a = 0.1, 0.05, 0.01, leading to 90%, 95%, 99% confidence. We used a 0.1 throughout our experiments.
H.4 Estimation of Bandwidth parameter
As we discussed Section 2 since TopP&R estimates the manifold through KDE with kernel bandwidth parameter h, we need to approximate it. The estimation techniques for h are as follows: (a) a method of selecting that maximizes the survival time (S(h) or the number significant homological features (N (h)) based information obtained about persistent homology using the filtration method, (b) a method using the median the k-nearest neighboring distances between
features obtained by the balloon estimator (for more details, please refer to [42] [43 and 44)). Note
that, the bandwidths h for all the experiments in this paper are estimated via Balloon Bandwidth Estimator.
For (a), following the notation Section A the ith homological feature of persistent diagram be (be, d.), then we define its life length as I, (7)) d: - by at kernel bandwidth h. With confidence band Co (h), we select h that maximizes one of the following two quantities:
N(h) # 4(h) > ca (h)), S(h) Σ1((h)) co (h))+-
Note that, denote the confidence band c, as c, (h) considering the kernel bandwidth parameter of KDE in Algorithm
For (b), the balloon bandwidth estimator is defined as below:
28

---

Algorithm/ 2 Balloon Bandwidth Estimator
# h: kernel bandwidth
d(X (dxi ||x x1
# KND: kth nearest distance
i.c., {d( {d(X, X1), (1),((
d(xids: X2)....}
# idx: index
Calculate kth nearest neighbor distance
# sort: sort in ascending order
end for
Given
X,)
Given KND (KND), KND KND...
for idr = 1, ..2.....1 do # Compute L2 distance between Xute and X!
Define the estimated bandwidth h median(KND)
H.S
Computational complexity
TopP&R is computed using the confidence band Ca and KDE bandwidth h. The computational cost of the confidence band calculation (Algorithm IT and the balloon bandwidth calculation (Algorithm (2)
O(k n d)) and O(n² d)), respectively, where represents the data size, d denotes the data dimension, and stands for the number of repeats (k- = 10 in our implementation). The resulting computational cost of TopP&R is O(k n d). In our experiments based real data, calculating TopP&R once using real and fake features, each consisting of 10k samples in a 4096-dimensional space, takes approximately 3-4 minutes. This computation speed is notably comparable to that of P&R and D&C, utilizing CPU-based computations. Note that, P&R and D&C that we primarily compare in our experiments and due to the computation of pairwise distances, the computational cost of these algorithm is approximately On d).
algorithms based k-nearest neighborhood for estimating data distribution,
Table A1: Wall clock, CPU time and Time complexity for evaluation metrics for 10k real and 10k fake features 4096 dimension. For TopP&R, r is random projection dimension (r < d). The results measured by the time module in python.
Time
TopPR (Ours)
Wall Clock
Imin 43s
Imin 40s
Imin 58s
CPU time
2min 44s
2min 34s
2min 21s
Time complexity
O(n² d)
Onn" d)
O(k n² r)
H.6 Mean hamming distance
Hamming distance (HD) 45 counts the number of items with different ranks between A and B, then measures how much proportion differs the overall order, i.e. for A; E and B, E B, (A, B) En= 1[k: A j Bi} where k is the number of differently ordered elements. The mean HD is calculated as follows to measure the average distances of three ordered lists: Given three ordered lists A, B, and C. HD (A,, B) + HD(A, C) + HD(B,C))/3
H.7 Explicit values of bandwidth parameter
Since metric adaptively reacts to the given samples of P and Q, have two h's per experiment. For example, in the translation experiment (Figure 2), there are 13 steps in total, and each time we estimate h for P and Q, resulting in total of 26 h's. To show them all at a glance, we have listed all values in one place. We will also provide the code that can reproduce the results in our experiments upon acceptance.
Table A2: Bandwidth parameters h of distribution shift in Section
5.1.1
f- shift
shift
real h
8.79
8.60
8.58
8.67
8.63
8.71
8.66
8.73
8.54
8.63
8.54
8.75
8.78
fake h
8.46
8.86
8.60
8.49
8.44
8.80
8.55
8.63
8.74
8.61
8.58
8.83
8.60
29

---

Table A3: Bandwidth parameters h of sequential mode drop in Section 5.1.2
Steps
0
2
real
10.5
10.6
10.5
10.7
10.5
10.8
10.6
fake h
11.0
10.2
10.0
9.82
9.57
9.29
8.78
Table A4: Bandwidth parameters simultaneous mode drop in Section 5.1.2
Steps
0
2
3
4
5
6
7
8
10
real
10.5
10.6
10.6
10.8
10.3
10.6
10.7
10.6
10.4
10.4
fake h
10.6
10.5
10.7
10.3
10.4
10.0
9.88
9.31
9.19
9.07
8.79
Table A5: Bandwidth parameters h of non-lll noise perturbation in Section
5.1.3
Steps
5
6
7
8
9
8.76
8.71
8.59
8.85
8.71
8.61
8.97
9.00
8.95
Scatter
Take
8.61
8.93
8.46
8.72
8.67
8.68
9.07
8.88
real
8.65
8.44
8.76
8.62
8.55
8.75
8.96
8.59
8.53
8.74
Swap
fake
8.52
8.97
8.87
8.94
8.84
9.05
8.94
8.90
Table A6: Bandwidth parameters h of FFHQ truncation trick in Section I.]
VP
v
V
real
6,26
6.41
6.31
6.23
6.06
6.24
6.20
6.61
6.39
6.28
6.26
fake
1.04
1.67
2.36
2.90
3.41
3.93
4.31
4.90
5.51
5.60
6.46
Table A7: Bandwidth parameters h of CIFAR10 sequential mode drop in Section 5.2.1
Steps
7
8
9
real h
7.09
6.95
6.96
6.92
7.02
6.81
6.86
6.88
6.78
6.67
fake h 6.87
6.75
6.88
6.64
6,40
6.64
6.41
6.44
6.62
6.04
Table A8: Bandwidth parameters h of CIFAR10 simultancous mode drop in Section 5.2.
Steps
2
3
4
real h 6.78
6.95
7.05
6.62
6.85
6.51
6.94
6.91
6.80
7.00
6.78
fake h
6.75
6.69
7.14
6.68
6.75
6.84
6.61
6.68
6.50
6.32
6.17
Table A9: Bandwidth parameters FFHQ non- Mon-IID noise perturbation in Section
5.2.2
Steps
8
9
real h
7.24
7.03
7.01
7.45
7.38
7.66
7.46
7.36
7.79
7.66
Scatter
fake
5.41
5.77
5.67
5.49
5.71
5,65
5.82
6.08
5.81
6.02
real h
7.25
7.20
6.99
6.67
7.07
6.96
7.03
6.94
6.83
6.86
Swap
fake
6.87
6.63
6.60
6.91
6.74
6.77
6.92
6.83
6.83
6.82
Additional Experiments
L.1 Survivability of minority sets in the long-tailed distribution
An important point to check in our proposed metric is the possibility that a small part of the total data (i.e., minority sets), but containing important information, can be ignored by the confidence band. emphasize that since our metric takes topological features into account, even minority sets are not filtered conditioned that they have topologically significant structures. We assume that signals or data
30

---

Table A10: Bandwidth parameters FFHQ noise addition in Section
5.2.3
Noise intensity
real h
6.52
6.27
6.41
6.49
Gaussian noise
fake
6.32
6.15
4.06
2.64
real h
6.08
6.26
6.26
6.47
Gaussian blur
fake
6.12
5.13
4.60
3,40
real h 6.06
6.28
6.49
6.44
Black rectangle
fake
6.36
6,66
6.31
5.93
Table A11: Bandwidth parameters h of GAN ranking in Section 5.2.4
StyleGAN2
ReACGAN
BigGAN
PDGAN
ACGAN
WGAN-GP
real
2.605
2.629
2.581
2.587
2.615
2.606
Inception
fake h 2.226
2.494
2.226
2.639
2.573
2.063
real
7.732
8.021
8.175
7.792
7.845
8.036
VGG16
fake h 7.880
6.839
7.178
6.174
6.839
3.615
real h 0.774
0.765
0.785
0.780
0.780
0,822
SwAV
fake h 0.740
0.667
0.6746
0.619
0.627
0,502
that are minority sets have topological structures, but outliers exist far apart and lack a topological structure in general.
To test this, we experimented with CIFARIO 44), which has 5,000 samples per class. We simulate a dataset with the majority of six classes (2,000 samples per class, 12,000 total) and the minority set of four classes (500 samples per class, 2,000 total), and an ideal generator that exactly mimics the full data distribution. As shown in the Table [12] the samples in the minority set remained after the filtering process, meaning that the samples were sufficient to form a significant structure. Both D&C and TopP&R successfully evaluate the distribution for the ideal generator. To check whether our metric reacts to the change in the distribution even with this harsh setting, we also carried out the mode decay experiment. We dropped the samples of the minority set from 500 100 per class, which can be interpreted as an 11.3% decrease in diversity relative to the full distribution (Given (a) ratio of the number of samples between majority and minority sets 12, 000 2, 000 = and (b) 80% decrease in samples per minority class, the true decay in the diversity is calculated as (146) x 0.8 11.3% with respect to the enitre samples). Here, recall and coverage react somewhat less sensitively with their reduced diversitics as 3 p.p. and 2 p.p.. respectively, while TopR reacted most similarly (9 p.p.) to the ideal value. In summary, TopP&R shows much more sensitiveness the changes in data distribution like mode decay. Thus, once the minority set has survived the filtering process, our metric is likely to be much more responsive than existing methods.
Table A12: Proportion of surviving minority samples in the long-taile tailed distribution after the noise exclusion with confidence band cx. The p.p. indicates the percentage points.
Before mode drop
After mode drop
Proportion of survival (before / after filtering)
100% + 59%
100% + 57%
TopP&R (Fidelity / Diversity)
0.99 / 0.96
0.99 0.87 (9 p.p
P&R (Fidelity Diversity)
0.73 / 0.73
0.74 p.p J)
D&C (Fidelity/ Diversity)
0.99 / 0.96
1.02 p.p.)
I.2 Experiment on Non-ITD perturbation with outlier removal methods
We compared the performance of metrics denoised dataset using Local Outlier Factor (LOF)
47
and Isolation Forest (IF) [88. The experimental setup identical to that of Figure (see Section for details of our noise framework). In this experiment, we applied the outlier removal method before calculating P&R and D&c. From the result, P&R and D&C still not provide a stable evaluation, while TopP&R shows the most consistent evaluation for the two types of noise without changing its trend.
G.3
31

---

Fidelity
Diversity
Fidelity
Diversity
08
08
1.0
06
0.8
0.8
06
04
0.6
04
0.4
022
0.0
00
00
070
1L04
015
0700
0004
[LOB
13
615
0.00
0004
(LOB
15:15
0100
1106
13:0
1212
0.15
Rabio of outberinthe the data
Ratio of outtberin the data (%)
Ratoo of outheer the ditta (%)
Ratio of outhier the dataa (%)
Non-IID scatter noise
Non- swap noise
Figure A2: Behaviors of evaluation metries on Non-IID perturbations. The dotted line in the graph shows the performance of metrics after the removal of outliers using the IF. We use "Clean" as a prefix to denote the evaluation after IF.
Fidelity
Diversity
Fidelity
Diversity
0.0
1.0
1.0
0.6
0.6
08
013
0.5
0.5
0.4
04
044
044
02
02
0.0
0.0
0.0
0.0
0.0
of cuttbrrint data
of outber data
Ratio of outller data
Ratio outleer data
(a) Non- scather nolse
(b) Non- swap
Figure A3: Behaviors of evaluation metrics on Non-IID perturbations. The dotted line in the graph shows the performance of metrics after the removal of outliers using the LOF. We use "Clean" as prefix to denote the evaluation after LOF.
Since TopP&R can discriminate topologically significant signals by estimating the confidence band (using KDE filtration function), there is no need to arbitrarily set cutoff thresholds for each dataset. On the other hand, in order to distinguish inliers from specific datasets using LOF and IF methods, a threshold specific to the dataset must be arbitrarily set each time, so there a clear constraint that different results are expected each time for each parameter setting (which is set by the user). Through this, it is confirmed that our evaluation metric guarantees the most consistent scoring based on the topologically significant signals. Since the removal of outliers in the TopP&R is part of the support estimation process, it is not appropriate to replace our unified process with the LOF or IF. In detail, TopP&R removes outliers through a clear threshold called confidence band which is defined by KDE, and this process simultaneously defines KDE's super-level set as the estimated support. If this process is separated, the topological properties and interpretations of estimated support also disappear. Therefore, is not practical to remove outliers by other methods when calculating TopP&R
I.3 Sequential and simultaneous mode dropping with real dataset
Diversity
Diversity
1.0
1.0
0.8
0.8
0.6
0.6
0.4
Taxi
Tepl
0.2
.a. Reall =
. Reall 9o
Ceverue 12
0.0
0,2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.0
Visualization on Baby ImageNet dataset Baby ImageNet
Concentrsion on the first mode (a) Sequential mode dropping
Concentraion on the first mode
(b) Simultaneous mode dropping
Figure A4: Comparison of evaluation metrics under sequential and simultaneous mode dropping scenario with Baby ImageNet[[5]
1.4 Sensitiveness to noise intensity
The purpose of this experiment (Figure A5) is to closely observe the noise sensitivity of metrics in F As the noise intensity is incrementally increased until all the metrics converge to O, the most ideal outcome for the metries is to exhibit a linear trend to capture these changes most effectively. From the results, it is able to observe that both TopP&R and P&R exhibit the most linear trend in their evaluation outcomes, while D&C demonstrates that least capability to reflect differences based on noise intensity.
32

---

(n) Gaussaa noise
(b) Gaussian blur
(c) Salt and pepper
(d) Black rectangle
1.0
0.5
PAR MAC
0.3
PMA
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1,0
0.0
0.2
0.4
0.6
088
1.0
0.0
0.2 04 0.6 0.8 1.0
Notseintersity
Notseintersiity
Noise-interiitt
Nosse-intrsslyy
Figure A5: Verification of whether TopP&R can make accurate quantitative assessment of noise image features. Gaussian noise, gaussian blur, salt and pepper, and black rectangle noise are added to the FFHQ images and emedded with T4096. The noise intensity is linearly increased until all metrics converge to zero.
1,5 Evaluating state-of-the-art generative models on the ImageNet
We conducted our evaluation using the ImageNet dataset, which covers a wide range of classes. We employed commonly accepted metrics in the community, including FID and KID, to rank the considered generative models effectively. The purpose of this experiment is not only establish the applicability of our ranking metric for generative models, akin to existing single-score metrics, but also to emphasize its interpretability. In Table A13 we calculated the HD between P&R variants and the more reliable KID metric. This comparison aimed to assess the alignment of our metric with established single-score metrics. The results consistenly showed that TopP&R's 's evaluation results are the most consistent with single-score metrics across various embedding networks. Conversely, P&R exhibited an inconsistent evaluation trend compared to KID, which struggled to effectively differentiate between ReACGAN and BigGAN. Note that, Kang et al. [15] previously demonstrated that BigGAN performs worse than ReACGAN, However, surprisingly. P&R consistently rated BigGAN as better than ReACGAN across all embedding networks. These findings imply that the P&R's unreliable support estimation under noisy conditions might hinder its ability to accurately distinguish between generative models.
Table A13: Ranking results based on FID. KID, MTD, TopP&R, P&R and D&C for ImageNet (128 x 128) trained generative models. SwAV. VGG16, and Inception V3 represent the embedders, and the numbers in parentheses indicate the ranks assigned by each metric to the evaluated models. The HD) the hamming distance between KID and metrics.
Model
ADM
StyleGAN-XI
ReACGAN
BigGAN
HD
FID (L)
9.8715 (1) 10.943 (2)
24.409 (3)
33.935 (4)
0
KID (L)
0,0010 (1) 0.0025 (2)
0,0062 (3)
0.0205 (4)
Inception
TopP&R (t)
0,9168 (1)
0.8919 (2)
0,8886 (3)
0.7757 (4)
0
D&C (f)
1.0333 (2)
1.0587 (1)
0.8468 (3)
0,5400 (4)
2
P&R (1)
0.6988
0.6713 (2)
0.4462 (4)
0.5079
2
MTD (L)
1.7308 (2)
1.5518 (1)
2.2122 (3)
3.0891
2
TopP&R (1)
0.9778 (1)
0.8896 (2)
0.8376 (3)
0.5599 (4)
0
VGG16
D&C (t)
1.0022 (3)
0.9159 (4)
1.1322 (1)
1.1020 (2)
4
P&R (1)
0.7724 (2)
0.7785 (1)
0.4092 (4)
0.5237 (3)
4
MTD (L)
15.748 (3)
22.313 (4)
10.268 (1)
13,992 (2)
TopP&R
0.8851 (2)
0.9102 (1)
0.6457 (3)
0.4698 2
SWAV
D&C (1)
1.0717 (1)
0.8898 (4)
1.0654 (2)
1.0578 (3)
3
P&R (1)
0.6096 (1)
0.5596 (2)
0.1335 (4)
0.1544 (3)
2
MTD (L)
0.4165 (3)
0.7789 (4)
0,3601 (2)
0,3200 (1)
4
33

---

Table A14: Ranking results based TOPP&R, P&R, and D&C for ImageNet (128 128) trained generative models. SwAV, VGG16, and Inception V3 represent the embedders, and the numbers in parentheses indicate the ranks assigned by each metric to the evaluated models. Note that, the random projection is applied to TopP&R, P&R, and D&C.
Model
ADM
StyleGAN-XL
ReACGAN
BigGAN
SWAV VGG16 Inception
TopP&R (1)
0.9168 (1)
0.8919 (2)
0.8886 (3)
0.7757 (4)
D&C w/ rand proj (1)
1.0221 (2)
1.0249 (1)
0.9218 (3)
0.7099 (4)
P&R w/ rand proj (1)
0.7371 (1)
0.7346 (2)
0.6117 (4)
0.6305 (3)
TopP&R (f)
0.9778 (1)
0.8896 (2)
0.8376 (3)
0.5599 (4)
D&C w/ rand proj (1)
1.0086 (3)
0.9187 (4)
1,0895 (1)
1.0803 (2)
P&R w/ rand proj (t)
0.7995 (2)
0.8056 (1)
0,5917 (4)
0.6304 (3)
TopP&R (t)
0.8851 (2)
0.9102
0.6457 (3)
0.4698 (4)
D&C w/ rand proj (1)
1.0526 (3)
0.9479 (4)
1.1037 (2)
1.1127 (1)
P&R w/ rand proj (1)
0.6873 (1)
0.6818 (2)
0,4179 (3)
0.4104 (4)
1.6
Verification of random projection effect in generative model ranking
We also applied the same random projection used by TopP&R to P&R and D&C in Section 1.6] From the results of the A14 it is evident that the application of random projection P&R leads different rankings than its original evaluation tendencies. Similarly, the rankings of D&C with random projection still exhibit significant variations based on the embedding. Thus, we have demonstrated that random projection does not provide consistent evaluation tendencies across different embeddings, while also highlighting that this characteristic is unique TopP&R. To quantify these differences among metrics, we computed the mean Hamming Distance (MHD) between the results in the embedding spaces of each metric. The calculated MHD scores were 1.33 for TopP&R, 2.67 for P&R, and 3.33 for D&C, respectively.
1.7
Verifying the effect of random projection to the noisy data
Fidelity
Diversity
Fidelity
Diversity
1.0
1,0
0.7
0.7
0.7
0.5
0.5
0.5
0.5
02
0.2
022
0.2
Dowtly Bumia
Doall
1.0
0.55
00
100
0.5
00
1.0
110
0.5
41
0.0
10
Center the fake clstributiongs
Center the fake distribution
Center the fake distribationu
Center the fake distribution
(4) All metries with random projection to 32-dimensional space
(b) All metrics without random proj ectioini
Figure A6: Behaviors of evaluation metrics for outliers on real and fake distribution. For both real and fake data, the outliers are fixed at 3 E Rd where d is a dimension, and the parameter μ is shifted from 1 to 1. (a) depicts the comparison of all metrics using random projection from 64 dimensions to 32 dimensions. (b) illustrates the difference between metries without using random projection, where real and fake data are considered in 32 dimensions.
We experiment to ascertain whether utilizing random projection address the drawbacks of existing metrics that are susceptible to noise, and we also conduct tests to validate whether TopP&R possesses noise robustness without using random projection. Through Figure A6
(a). it is evident that P&R and D&C still retain vulnerability to noisy features, and is apparent that random projection itself does not diminish the impact of noise. Furthermore, Figure A6 (b) reveals that TopP&R, even without utilizing a random projection, exhibits robustness to noise through approximating the data support based on statistically and topologically significant features.
Building upon the insight from the toy data experiment in Figure A6 that random projection does not confer noise robustness to P&R and D&C, we aim to further demonstrate this fact through a real data experiment. In the experiment depicted in Figure A7 we follow the same setup as the previous experiment in Figure 5 applying the same random projection to P&R and D&C only. The results of
34

---

Fidelity
Diversity
Fiddity
Diversity
1.0g
16
Tige
1.05
1.6
Tapel
1000
14
1.00
1.4
0.9
Toge
1.2
0.50
Typ
Preckkmn
1.2
1.0
0.0
0,04
0.08
0.2
1.5
0.0
0.04
0.08
0.12
15
0.0
0.01
0.08
0.12
1,5
0.0
0.04
0.08
0.12
1.5
Ratio Df odfiies in dad (96)
Ratio of outlier (%)
Radio of oullier'n dalaa (36)
Ratio odtier in inteeeaalaaaaa dala
(a) Non-IID scatter noise
(b) Non-IID swap noise
Figure A7: Comparison of evaluation metrics on Non-IID perturbations using FFHQ dataset. We replaced certain ratio of 2' and y (a) with outliers and (b) by switching some of real and fake features. P&R and D&C were computed using the same 52-dimensional random projection.
this experiment affirm that random projection does not provide additional robustness against noise for P&R and D&C, while TopP&R continues to exhibit the most robust performance.
I.8
Robustness of TopP&R with respect to random projection dimension
Sequential mode drop
Simultaneous mode drop
Sequential mode drop
Simultaneous mode drop
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
Becal 20
Becal 10
Recll 8: 1)
Recll 8: 1 Conragg is
0.0
Coveagg -1
0.0
Conerage (1:
0.0
Coveagg (
0.0
0.2
0.6
0.8
1.0
0.6 0.8
1.0
0.6
0.8
1.0
0.0
0.4
1.0
Concectration on the mode Concectration on the first mode
Concentration on the mode
Concectratio on the first mode
(a) Random proj eejecttion to 64-dimensional space
(b) Random projection to dimeneinnii space
Figure A8: Comparison of evaluation trends of TopP&R with respect to random projection dimension in both sequential and simultaneous mode drop scenarios. Baby ImageNet dataset is used for the comparison, and TopP&R is computed using random projections (a) 64 dimensions and (b) 128 dimensions.
To investigate whether the dimension random projection influences the evaluation trend of TopP&R, we conduct mode dropping experiment using the Baby ImageNet dataset, similar to the experiment in Section I.3 Results in (a) and (b) of Figure A8 respectively compare the performance of TopP&R using random projection in dimensions of 64 and 128 with other metrics. From the experimental results, observe that TopP&R is the most sensitive to the distributional changes, akin to the tendencies observed in the previous toy mode dropping experiment. Furthermore, upon examining the differences across random projection dimensions, TopP&R shows consistent evaluation trends regardless of dimensions.
I.9 Trucation trick
is parameter for the truncation trick and is first introduced in 4 and 1. We followed the approach in [4] and [T] GANs generate images using the noise input z, which follows the standard normal distribution (0, T) uniform distribution (1 Suppose GAN inadvertently samples noise outside of distribution, then it is less likely to sample the image from the high density area of the image distribution p(=) defined in the latent space of GAN, which leads to generate an image with artifacts. The truncation trick takes this into account and uses the following truncated distribution. Let be the mapping from the input to the latent space. Let = fz). and 2 E=f())], where is either from (O0 I) or ((-1, 1). Then we use w' = T + z/( w w) as a truncated latent vector. If the value of & increases, then the degree of truncation decreases which makes images have greater diversity but possibly lower fidelity.
I.10 Toy experiment of trade-off between fidelity and diversity
We have designed a new toy experiment to mimic the truncation trick, With data distributions of 10k samples, X ~ Mi 0, /) E 32 and y Niu - 0.6, o²) E R232, we measure the fidelity and diversity while incrementally increasing a from 0.7 to 1.3. For smaller a values, the evaluation metric should exhibit higher fidelity and lower diversity, while increasing should demonstrate decreasing
35

---

Tnge"
1.0
Tistt
y- "y-Mi m0.0)
1.50
1.25
0,8
1.00
0,6
o"= 1.3)
0.75
0.4
0.50
0.25
022
0.00
0.0
0.70
0.85
100
1.15
1.30
0.70
115 Standardd deviationn y (r)
0.85
1.0
1330
Sunderddeviition
(c)
Figure A9: Trade-off between fidelity and diversity in toy experiment. In this experiment, the f of X' was set to O, and the of y was set to 0.6. By incrementally increasing from 0.7 to 1.3, the evaluation trend based changes in the distribution is measured.
fidelity and higher diversity. From the results in Figure A9 both TopP&R and P&R exhibit trade-off between fidelity and diversity.
I.11 Resolving fidelity and diversity
emerated Images
VGG16
SwAV
5.0
4.0
3.0
2.0
1.0
1.0
0.5
Thylt
0.0
Decrsead y
Iroesed
Thuncttonn Pat
Thuncaioon Psa
Figure A10: Behaviors of metrics with truncation trick. The horizontal axis corresponds to the value of di denoting the increased diversity. The images are generated via StyleGAN2 with FFHQ dataset
To test whether TopP&R responds appropriately to the change in the underlying distributions in real scenarios, test the metric the generated images of StyleGAN2 2 using the truncation trick []. StyleGAN2, trained on FFHQ as shown by Han et al. [49] tends to generate samples mainly belonging to the majority of the real data distribution, struggling to produce effectively. In other words, while StyleGAN2 excels in generating high-fidelity images, it lacks diversity. Therefore, the evaluation trend this experiment should reflect steady high fidelity score while gradually increasing diversity. Han et al. [49 demonstrates that the generated image quality of StyleGAN2 that lies within the support region exhibits sufficiently realistic visual quality. Therefore, particularly in the case of TopP&R, which evaluates fidelity excluding noise, the fidelity value should remain close to As shown in Figure every time the distribution is transformed by vi TopP&R responds well and shows consistent behavior across different embedders with bounded scores in [0, 1], which are important virtues as an evaluation metric. On the other hand, Density ty gives unbounded scores (fidelity 1) and shows inconsistent trend depending on the embedder. capped in value, it difficult interpret the score and know exactly which value denotes the best performance (e.g., in our case, the best performance is when TopP&R 1).
samples
Because Densiti ttytttyyt
36


Reviews:
[{"review_id": "kiaARuA0M2", "reviewer": "Reviewer_PyMg", "summary": "The paper addresses the need for more reliable evaluation metrics for deep generative models. The existence of noise in real data, such as mislabeled data and adversarial examples, affects the reliability of existing metrics and may lead to false impressions of improvement when developing generative models. To overcome these issues, the paper proposes a new evaluation metric based on the idea of Topological Data Analysis (TDA) and statistical inference. The proposed metric uses persistent homology from TDA and bootstrap bandwidth with confidence level to distinguish between significant patterns (signal) and noise in the data, providing a more robust and reliable measure of generative model performance.", "questions": "1. This paper employs the median of pairwise k-nearest distances between samples to estimate fixed-width kernels for KDE. I am curious whether this method is as effective as using sample-adaptive varying kernels, especially for datasets with multi-modal or long-tailed distributions. Adaptive kernels that adjust to local data characteristics might offer advantages in capturing complex and diverse patterns.\n\n2. Concerning the experiments presented in Table 1, which dataset was used for these experiments? Can the proposed metric demonstrate effectiveness when applied to large-scale datasets, such as ImageNet?\n\n3. There are some recommended experiments that are missing in the paper but would strengthen the effectiveness of the proposed metric:\n- Evaluating metrics on Diffusion models such as ADM or EDM, which are the leading and prominent advancements in generative models, would be beneficial to demonstrate the practicality of the proposed metrics.\n- Experiment for being robust to outliers in real-world dataset as in D&C paper.\n- Ablation study regarding the hyperparameters alpha and h for real-world datasets.\n\n4. To enhance the readability of the paper, the captions of figures should be more detailed.\n", "limitations": "Limitations are discussed in the supplement. Potential negative social impact is unlikely for a paper comparing distributions.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Given the recent substantial advancements in generative models, having a metric that is robust and reliable under various real-world noises holds significant importance. The authors present an approach that aims to overcome the limitations of previous metrics, using straightforward yet well-founded methods. Additionally, the paper introduces theoretical properties that show the metric's consistency in the presence of adversarial noises. The paper is well-written and easy to follow. ", "weaknesses": "1. If I understand correctly, the authors use random linear projection only for the proposed metric, which might be unfair since it is unclear whether the improvements in robustness to outliers or noises stem from the low-dimensional space or the proposed methods. A comparison with previous methods operating in the same low-dimensional space would be more comprehensive. \n\n2. The proposed metric's algorithm relies on multiple iterations for KDE filtration, raising questions about its practicality. It would be beneficial to include a comparison of the computational cost and overall runtime between the proposed metric and previous ones. Gaining insights into its efficiency compared to existing metrics will be valuable for both potential users and researchers. Additionally, it is essential to consider whether the metric can efficiently scale to handle a large number of datasets. However, the paper does not seem to provide details about the number of datasets used in the experiments, which leaves uncertainty about its scalability.\n\n3. In section 5.1.2, the authors assert that precision values smaller than 1 indicate a problem. However, this lacks sufficient justification especially with a small number of samples. Additionally, the behavior of TopP for simultaneous mode dropping in Figure 3 exhibits worse instability compared to other metrics. \n\n4. In the experimental results for mode dropping experiments with CIFAR-10, TopR does not exhibit consistent behavior with the toy experiments in the main paper (Figure 4) and sometimes performs worse than Recall and Coverage.\n\n5. In section 5.2.3, the authors argue that the proposed metric is more sensitive to different levels of diverse noise, similar to FID. However, the results in Figure 6 do not appear to provide compelling evidence for this claim, except for the Gaussian noise case. Notably, in the salt and pepper noise case, TopPR, PR, and DC exhibit equal behavior, and in the black rectangle scenario, TopPR performs worse than PR. To strengthen the authors' claim about the proposed metric's sensitivity to diverse noise levels, additional explanation and experiments are necessary.\n\n6. The sole reliance on FID as the ground truth ranking for models in section 5.2.4 might be insufficient and unconvincing. It is not clear how much relying on FID as GT is reliable. Including more widely used metrics, such as KID or human evaluations, would enhance the credibility of the result. However, even with the FID, the results do not seem to show the clear superiority of the proposed metric. Additionally, the results with HD across different embeddings also raise questions about the effect of the random projection layer, as other metrics do not utilize such a layer.\n\n7. Some statements in the paper are unfounded. For instance, the rationale behind the ideal metric returning zero (line 263) is not clear. Moreover, the assumption in line 278 (\"ground truth diversity should linearly decrease\") is not correct because the supp(Q) remains the same with simultaneous mode dropping. In that sense, TopR works the worst of all. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer 1\n**R1-1. Adaptive kernels.** (G4)  \n&nbsp;  \n**R1-2, W2.**  \n**The dataset utilized in Table 1:**  (G1-2) CIFAR-10. In the revised version, we have mentioned this in the caption.  \n**Effectiveness on a large-scale dataset:**  (G1) Yes, it scales well. For all the experiments, we used 10k real and 10k fake samples, a standard choice for most of the evaluation metrics. We have added this detail to our paper.  \n**Computational complexity:** (Table B2) TopP&R takes approximately 1 min 58 seconds of wall clock time for a single measurement on 10k samples, showing an efficient computational speed for model comparison. We have added this to our supplementary material.  \n&nbsp;  \n**R1-3.**  \n**The outlier experiment similar to D&C:** (Section 5.2.2 and Figure 5) Please note that it has already been conducted in a completely identical manner, using outliers and inliers from the FFHQ dataset.  \n**Comparison of state-of-the-art generative model** (G1-3)  \n**Ablation study concerning hyper-parameters:** (G3)  \n&nbsp;  \n**R1-4. Improve captions:** Thank you for bringing this to our attention. In our revised paper, we have now updated the captions.  \n&nbsp;  \n**R1-W1, W6. Add KID:** (G1-2, G1-3), **Impact of random projection:** (G2)  \n&nbsp;  \n**R1-W3. Justification to precision values smaller than 1 indicate a problem:** Consistent with our results, Naeem et al. (2020) noted that precision does not approach 1, even when comparing two sets of samples from the same distribution, **irrespective of the sample size**. This makes determining a meaningful precision value difficult when assessing actual generative models. While TopP may exhibit cases where it does not converge to 1, it still shows a consistent evaluation trend, and it is difficult to argue that TopP values less than 1 significantly differ from the values that should converge. The key aspect of this experiment is the sensitivity of TopR to small changes in the mode. As the reviewer pointed out, even if TopP shows minor variations in this experiment, we believe that in many of our other experiments, TopP&R demonstrated more advantages than drawbacks.  \n&nbsp;  \n**R1-W4. Mode dropping experiments on CIFAR-10 are not strong:** (G1-4) We believe that the reviewer is referring to Figure 3 (not Figure 4). We agree with the reviewer that, compared to the results of Figure 3, it is hard to say that TopR is clearly better than the others in CIFAR-10 experiments. To further investigate the differences between TopP&R and the existing metrics, we conducted experiments using a large-scale dataset, baby imagenet, which has higher image resolution and diversity than CIFAR-10. In this large-scale dataset, TopP&R and the other metrics exhibit similar trends as seen in the toy data experiment–TopP&R shows sensitivity to distribution changes and exhibits precise responses.  \n&nbsp;  \n**R1-W5. The authors argue that the proposed metric is more sensitive to different levels of diverse noise, similar to FID:** We wish to clarify that we did **not** assert that TopP&R is \"more\" sensitive to varying levels of diverse noise compared to all other metrics. Our comparison was limited to MTD, a topology-based method like ours, but one that exhibited inconsistent reactions to the distortions. Other than that, as it can be seen in Section 5.2.3, we simply claimed that TopP&R effectively captures the various levels of distortion introduced to the images. Our objective was to demonstrate that TopP&R offers a consistent evaluation across different noise levels. We did not intend to suggest that our metric shows the best response.  \n&nbsp;  \n**R1-W7.  \nThe rationale behind the ideal metric returning zero (line 263):** As the reviewer pointed out, it is indeed appropriate to modify the statement in line 263 to “has low values” (not zero).  \n\n**Assumption in line 278  (\"ground truth diversity should linearly decrease\") is not correct:** \nIt is correct. The $supp(Q)$ shrinks as the number of samples of each mode simultaneously decreases. This is because the samples are from the Gaussian distribution of high dimensional space, where they are sparsely distributed. Hence the deletion of the samples actually results in shrinking the support, which leads to a linear decrease in ground truth diversity. \nThis experiment aligns with Naeem et al.’s (2020) work, aiming to achieve an ideal linear response to simultaneous mode drop. The experiment in Section 5.2.1 starts with identical $supp(P)$ and $supp(Q)$ and progressively reduces samples of nine modes within $supp(Q)$ in a linear fashion until these nine modes vanish, while replenishing the first mode with the decreased number of samples. We would like to clarify that there might have been a misunderstanding regarding this aspect. Although the total sample count within $supp(Q)$ remains unchanged, the distribution itself experiences the linear disappearance of nine modes, ultimately leaving only one mode. Thus, in terms of diversity, the reduction indeed follows a linear trend.  Therefore, it is crucial for the metric to capture the linear reduction in diversity within $supp(Q)$. The results in G1-4 of general comments further confirm that TopP&R responds most effectively to such linear trends."}, {"review_id": "1H1G5XPYiz", "reviewer": "Reviewer_YKk3", "summary": "The paper discusses robust estimation of precision and recall in high dimensional and potentially noisy data. The approach relies on using kernel density estimation to approximate the underlying distributions and using a bootstrap estimation of confidence interval to ignore support with low probability mass. The authors show the effectiveness of this approach extensively using simulated and real data.", "questions": "- Figure 1b: In the persistence plot, what does the shaded area imply? Can more blue points be added from the example on the right showing where in the persistence plot they land? Should the birth in this case always be at zero?\n- Line 128: while c_X and c_Y are referred to as bootstrap bandwidth, is there a link between them and h_n and h_m, i.e., bandwidth of the kernel density estimate?\n- Line 261: what does \"swapping the position between Xi and Yj\" imply?\n- Line 275: sensitiveness or sensitivity?", "limitations": "Limitation of the method has not been discussed in details. More discussion on the computational complexity will be helpful.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is very well written and discusses the motivation and approach clearly. The problem addressed in the paper is very relevant as robust estimation of performance metrics is crucial for better comparing different approaches. The paper presents several interesting simulated and real examples to elaborate the method.", "weaknesses": "Although the estimation of confidence interval to remove 'noisy' samples is interesting, it would be great to see a discussion on estimating this uncertainty at individual points separately than working with the worst uncertainty, i.e., the inf norm over the kernel density.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer 2  \n**R2-1. Shaded area in Figure 1b:** For the sake of clarity in our explanation, it would be beneficial to provide a concise description of our Persistence Diagram (PD). At first glance, our PD might appear to be a swapping of the x-label and y-label, a confusion commonly associated with the PD derived from the kernel density estimator (KDE). **This confusion stems from the fact that the PD for KDE is generated through a 'super-level' filtration, distinct from the more typical 'sub-level' filtration used for most PDs, such as a Vietoris-Rips PD**. This distinction arises because KDE assigns high values to data points, making a 'decreasing' filtration value more appropriate. As a result, in this context, homological features are born at high values and die at low values, leading to a birth > death relationship. In particular, the PD for KDE almost never has homological features with birth = 0, while the PD for Vietoris-Rips has a lot of 0-dimensional features with birth = 0. To ensure comparability with the visualization of the PD from sub-level filtration, it has become customary to position death on the x-axis and birth on the y-axis.  \n\nThe shaded area corresponds to the orange area and the light green area in Figure A1. The orange area is where homological features have their birth < $c_\\alpha$ and death < $c_\\alpha$, and they usually correspond to outliers. The light green area is where homological features have their birth > $c_\\alpha$ and death > $c_\\alpha$, and they usually correspond to homological insignificant features lying within the estimated support. For either case, the shaded area corresponds to homological features that are statistically insignificant, and can be either (1) from noise, or (2) from signal but not eminent enough compared to the sample size. See Section C for a more detailed discussion.  \n&nbsp;  \n**R2-2. link between $c$ and $h$:** As in Algorithm 1 in **Section H.2**, the kernel bandwidth $h$ performs a function akin to the parameter $k$ employed in P&R and D&C. Notably, the parameter $k$ can influence the inherent breadth of the support, thereby affecting its scope. Employing a KDE that incorporates this parameter, the probability values of data features are measured. Subsequently, during the support estimation process, the probability values of features are used to approximate the bootstrap bandwidth ($c_X$ or $c_Y$). Thus, there exists a link between the kernel bandwidth $h$ and the bootstrap bandwidth, as they are intertwined in the approximation process.  \n&nbsp;  \n**R2-3. \"swapping the position between $x_i$ and $y_j$\":** This means the exchange of randomly selected datapoints $x_i\\in\\mathcal{X}$ and $y_j\\in\\mathcal{Y}$, such that each is transferred to the dataset to which other belongs (i.e., $x_i\\in\\mathcal{Y}$ and $y_j\\in\\mathcal{X}$).  \n&nbsp;  \n**R2-4. Sensitiveness or sensitivity?:** Thank you for your comment and we have revised that part.  \n&nbsp;  \n**R2-L1. Computation cost:** (Table B2) We have relatively similar computational complexity. We analyze the computational complexity and time consumption in Table B2 in the pdf we provide at this rebuttal.  \n&nbsp;  \n**R2-W1. Discussion on estimating this uncertainty at individual points separately:** Great point. As the reviewer pointed out, localizing the uncertainty at individual data points separately is an interesting direction of work, and it will be more sample efficient and disregard fewer data points. However, considering that we are preserving the topological signals, such a direction of localizing the uncertainty is almost hardly possible given the state of the art. For example, it is possible to localize the uncertainty of KDE $\\hat{p}$, since the functional variability is local; to control the uncertainty of $\\hat{p}$ at some point $x$, (roughly saying) we just need to analyze the function value at that point $\\hat{p}(x)$. However, to localize the uncertainty of the homological feature lying at some point $x$, we need to analyze all the points that the homological feature lies on; hence even if we only want to control the uncertainty at a local point, it necessitates controlling the uncertainty on the global structure. This makes localizing the uncertainty for topological features difficult given the tools in topology and statistics we currently have. We have incorporated this discussion into the limitations section."}, {"review_id": "0RrXoLkLBw", "reviewer": "Reviewer_fxDp", "summary": "The paper introduces TopP&R, a comprehensive evaluation metric for generative models that enhances the accuracy of sample quality assessment in comparison to existing metrics. TopP&R incorporates topological data analysis and statistical inference, effectively estimating supports through the application of Kernel Density Estimator (KDE) under topological conditions. This metric demonstrates remarkable robustness towards outliers, successfully detects changes in distributions, and provides consistency guarantees and robustness for high-dimensional data with minimal assumptions. Notably, TopP&R is the pioneering metric that prioritizes robust support estimation, ensuring statistical consistency even when confronted with noise. The paper presents compelling theoretical findings and experimental evidence that strongly validate the effectiveness of TopP&R.", "questions": "1. How does TopP&R handle the trade-off between precision and recall in evaluating generative models?\n\n2. Can you provide more details on the noise framework and statistical inference used in TopP&R?\n\n3. Have you considered comparing TopP&R to other evaluation metrics based on topological features, such as Persistence Landscape?\n\n4. Can you provide more information on the implementation of TopP&R and its applicability to different types of generative models?\n\n5. How does the choice of threshold for retaining topologically and statistically significant features affect the performance of TopP&R?\n\n6. Have you explored other techniques, like Gaussian mixture models or non-parametric methods, for estimating the density function in TopP&R?", "limitations": "Based on my understanding, the limitations of the TopP&R method mentioned in this paper can be summarized as follows: \n\n1. The method requires a large number of samples to estimate the confidence band accurately, which can be a challenge for datasets with limited samples. \n\n2. The reliance on the choice of kernel function and bandwidth can affect the performance of TopP&R, and the need for a theoretical guarantee on the choice of bandwidth can be difficult to obtain in practice. \n\n3. The method assumes a smooth density function and independence between samples, which may not hold for all types of data, and the reliance on topological features may not capture all aspects of the data distribution.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a novel evaluation metric, Topological Precision and Recall (TopP&R), for generative models. It combines topological data analysis and statistical inference to estimate support robustly and detect distributional changes accurately.\n\nThe paper provides theoretical and experimental evidence to demonstrate the reliability and consistency of the TopP&R metric. It accurately assesses sample quality, even in the presence of outliers, non-independent and identically distributed perturbations, and other noise conditions. The paper also presents a systematic approach to estimate support using Kernel Density Estimator (KDE) under topological conditions.\n\nThe paper clearly explains the motivation, limitations of existing metrics, and the design and implementation of the TopP&R metric. It includes visualizations and examples to enhance clarity and uses clear language to facilitate understanding.\n\nThe paper addresses the need for more reliable evaluation metrics for generative models. The proposed TopP&R metric fills this gap by providing a robust evaluation of sample quality and ensuring statistical consistency. Its findings have implications for improving the evaluation and development of generative models.", "weaknesses": "Limited accessibility: The paper may be challenging for readers unfamiliar with persistent homology and topological data analysis to grasp the methodology and results. Providing additional background information and explaining concepts in more detail would improve clarity.\nLimited experimentation: The experiments are confined to a few datasets and models. Conducting experiments on a broader range of datasets and models would demonstrate the effectiveness of TopP&R in diverse scenarios.\n\nInsufficient discussion of limitations: Although the authors mention some limitations, such as the requirement for a large number of samples to estimate the confidence band, a more comprehensive discussion of the limitations would help readers understand the scope and applicability of TopP&R.\n\nRecommendations for improvement:\n\nCritique existing metrics: Highlight limitations of commonly used metrics, such as Inception Score and Fréchet Inception Distance, due to unreliable support estimates from sample features.\n\nAddress noise vulnerability: Explain how TopP&R overcomes vulnerability to noise by considering topologically and statistically significant features with confidence.\n\nPresent experimental validation: Provide comprehensive experimental results comparing TopP&R to existing metrics, demonstrating its superior accuracy and consistency.\n\nDiscuss real-world applicability: Explore practical implications and applications of TopP&R in various domains, showcasing its relevance beyond theory.\n\nShare open-source implementation: Make TopP&R's implementation publicly available, ideally on platforms like GitHub, to promote reproducibility, validation, and further advancements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer 3  \n**R3-1. Trade-off between precision and recall:** To quickly check whether TopP&R exhibits trade-off between fidelity and diversity as in improved P&R, we used $\\mathcal{X}\\sim{\\mathcal{N}(0,1)}$ and $\\mathcal{Y}\\sim{\\mathcal{N}(0.6, \\sigma^2)}$ with a sample size of 10k and 32 dimensions. We compared the metrics while gradually increasing $\\sigma$ from 0.7 to 1.3. The results of the experiment are shown in the table below.\n| $\\sigma$     | 0.7  | 0.85 | 1.0  | 1.15 | 1.3  |\n|----|------|------|------|------|------|\n| Precision | 0.85 | 0.61 | 0.36 | 0.11 | 0.04 |\n| Recall    | 0.0  | 0.05 | 0.22 | 0.69 | 0.89 |\n| TopP      | 0.93 | 0.68 | 0.28 | 0.04 | 0.01 |\n| TopR      | 0.0  | 0.01 | 0.16 | 0.84 | 0.99 |\n\n&nbsp;  \n**R3-2, W4. Provide more details on the noise framework and statistical inference:** Added. Noise refers to samples with low statistical probability values, not contributing significantly to the overall distribution. We discussed two non-IID noise types in Section 5.1.3 and 5.2.2: scatter noise and swap noise. Given the scatter noise is derived by adding noise sampled from a uniform distribution across all data dimensions, its influence remains limited due to its low probabilities. Consequently, evaluation metrics are expected to remain relatively stable even when the level of scatter noise increases. On the other hand, swap noise involves randomly selecting samples from two sets and exchanging their classes. At first, only a few swaps may not form a meaningful distribution, as the swapped samples don’t necessarily represent the distribution of the opposite set. However, as the level of swap noise increases, it can lead to the formation of significant distributions due to the heightened probability of class swaps. \n\n**How TopP&R overcomes vulnerability to noise:** The significant data structures possess meaningful probability values, while the opposite exhibits lower values. In this context, TopP&R approximate robust data support identifying significant data structures by the bootstrap band $c_\\alpha$. In our paper, we have provided a detailed explanation of the noise framework we employed and described how our algorithm should effectively respond to such noise.  \n&nbsp;  \n**R3-3. Metrics utilizing topological features:** To the best of our knowledge, our paper compares all the metrics utilizing topological features  (e.g., GCA and MTD). We did not include a separate comparison with previous metrics such as Geometry Score (Khrulkov et al. 2018), as they are known to exhibit inferior performance compared to the modern topological feature-based metrics like MTD, and are also notoriously slow in computation, making them impractical for real-world usage.\n\nIt is possible to define metrics using methods like the persistence landscape. For instance, one could construct landscapes using the filtration method for both real and fake datasets, and then define a new metric using statistical tests such as the t-test on the real and fake features. However, these methods utilizing persistence landscapes focus only on topological structures and forget distributional features not captured by persistence landscapes. As a result, **they may not effectively capture geometric transformations like translation in distributions**. Due to these characteristics, it is challenging to consider such methods particularly suitable as metrics for comparing generative models.  \n&nbsp;  \n**R3-4, W6, W7. Applicability to different types of generative models:** (G1-2, G1-3) TopP&R can be applied to all fields where generative models are used. For example, TopP&R can evaluate models that produce sound signals using appropriate networks to obtain features. Moreover, when comparing distribution similarity between various numeric data, TopP&R can be directly used even without an embedding network, treating each observation value as a feature.  \n\n**Details of implementation, Code release:** (Table B2, Introduction) Please note that we have already provided the link to our code at the end of Section 1. Our metric primarily utilizes the Numpy, sklearn, and scipy libraries, and we utilize the linear layer provided by Pytorch. The metric computation is achievable through CPU operations. In terms of computation speed, our metric requires approximately 1 min 58 seconds of wall clock time to compare a real and fake dataset with a sample size of 10k. This indicates that our approach offers an appropriate calculation speed for effectively comparing generative models. Moreover, our algorithm is very user-friendly, as conducting experiments using our method only requires a simple one-line of code.  \n&nbsp;  \n**R3-5, L2. Choice of threshold:** (G3).  \n&nbsp;  \n**R3-6. Other techniques for estimating density:** (G4).  \n&nbsp;  \n**R3-W1. Limited experimentation:** (G1).  \n\n**Addressing accessibility:** Given the constraints of limited page space and the conference’s nature, we designed the paper to allow readers to grasp the robustness of TopP&R at a high-level without becoming entangled in intricate TDA details. However, we want to emphasize that certain aspects of TDA were intentionally condensed in the main paper. For readers seeking a deeper understanding, we have provided a comprehensive TDA explanation in the appendix. We will make diligent efforts to furnish more detailed explanations to address areas that may require further clarity.  \n&nbsp;  \n**R3-W2, L1. Sample size required for estimating confidence band:** (G1-1, Table B2) TopP&R uses 10k real and fake samples, identical to all other metrics, and does not require a larger sample size to demonstrate its performance. In addition, TopP&R is efficient to compute (Table B2).  \n&nbsp;  \n**R3-W3. Critique existing metrics:** We have revised the manuscript to better emphasize the aspects suggested by the reviewer in our experimental results.  \n&nbsp;  \n**R3-W5. Experimental validation (showing TopP&R’s accuracy and consistency):** (G1, G2-1)."}, {"review_id": "e0fsna5Lp0", "reviewer": "Reviewer_UDSM", "summary": "The paper presents a novel evaluation metric called Topological Precision and Recall (TopP&R) for generative models. Existing metrics often suffer from unreliable support estimation and yield inconsistent results. In contrast, TopP&R systematically estimates supports by retaining topologically and statistically significant features with confidence. It provides robust evaluation, accurately capturing changes in samples even in the presence of outliers and non-independent and identically distributed perturbations. The metric combines ideas from Topological Data Analysis and statistical inference, offering a reliable approach to support estimation. The paper includes theoretical and experimental evidence to demonstrate the effectiveness and robustness of TopP&R, making it a valuable contribution to the evaluation of generative models.", "questions": "No additional questions in addition to the weaknesses listed above.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel evaluation metric, Topological Precision and Recall (TopP&R), specifically designed for generative models. This metric addresses the limitations of existing evaluation metrics by focusing on robust support estimation, which is crucial for accurately assessing sample quality.\n\nTopP&R provides a robust evaluation that is resilient to outliers and non-independent and identically distributed perturbations. The metric systematically estimates supports by retaining topologically and statistically significant features, ensuring reliable evaluation results even in challenging scenarios.\n\nThe paper demonstrates that TopP&R offers statistical consistency in its results. By incorporating concepts from Topological Data Analysis and statistical inference, the metric provides a systematic approach to distinguishing signal from noise and effectively capturing relevant features in the data.\n\nThe paper provides both theoretical and experimental evidence to support the effectiveness of TopP&R. The theoretical analysis establishes the robustness and consistency guarantees of the metric, while the experimental results demonstrate its superior performance compared to existing evaluation metrics.\n\nTopP&R is a versatile evaluation metric that can be applied to various generative models. Its systematic approach to support estimation makes it suitable for evaluating the performance of models across different domains and datasets.\n\nThe authors provide code implementation for TopP&R, making it readily accessible for researchers and practitioners. This facilitates the adoption and reproducibility of the metric, contributing to the transparency and advancement of the field.\n\nOverall, the paper's strengths lie in its introduction of a novel evaluation metric, its robustness and reliability in assessing sample quality, and its theoretical and experimental validation, demonstrating its effectiveness and broad applicability in the evaluation of generative models.", "weaknesses": "The paper mentions Algorithm 1 in line 107 but does not provide the algorithm or any further explanation in the paper. This lack of clarity leaves readers confused and makes it difficult to understand the specific steps and procedures involved.\n\nLine 139 mentions the bootstrap bandwidth, but the paper does not clearly explain how this bandwidth is computed. It is important to provide details on the methodology used for computing the bandwidth to ensure transparency and reproducibility.\n\nThe paper lacks a clear discussion on the dimensionality of the problem that the proposed method can handle. It does not explain how the authors choose the dimensionality of the space for random projections or how they ensure that these projections capture relevant underlying features. Additionally, the paper does not provide theoretical results supporting the choices made in Section 3.4 regarding dimensionality and random projections.\n\nIn the experimental section, the authors choose a projection dimension of 32 without providing a clear justification or explanation for this specific choice. It is important to provide reasoning or empirical evidence to support the selection of this dimension and explain its impact on the performance of the method.\n\nWhile the paper includes simulations to demonstrate the robustness and effectiveness of the proposed method, it lacks a thorough evaluation on real-world data where changes are not simulated. It would be valuable to see the performance of the method in real data scenarios to assess its practical applicability and validate its effectiveness in capturing genuine changes.\n\nAddressing these weaknesses would strengthen the paper by providing clarity, offering detailed explanations for methodological choices, and conducting comprehensive evaluations on real-world data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer 4  \n**R4-W1, W2. Explanation of line 107 and 139 (Algorithm 1):** Added. We apologize for missing the appropriate reference link to “Appendix H.2” for the algorithm. Unfortunately, this happened while we were breaking the manuscript into two (main and supplementary) documents for the submission.  \n&nbsp;  \n**R4-W3, W4 How the dimension of random projection was chosen:** (G2-2).  \n&nbsp;  \n**R4-W5. Evaluation on real-world data:** (G1)."}, {"review_id": "tXUmQoyBhi", "reviewer": "Reviewer_4EiX", "summary": "The paper observes that the existing metrics for evaluating fidelity and diversity of generative models can be unreliable when outliers or non-IID perturbations are present. Thus, the authors propose a new metric based on Kernel Density Estimation under topological conditions that is more robust in the earlier scenarios. The main contributions of this paper are the metric that is robust to outliers and theoretical guarantees for robustness.", "questions": "1. The authors claim that robustness is required in practical evaluations of models since these cases are “wild”, i.e., might corrupt/outlier samples or the distributions might be sparse without density. I am curious as to why the proposed metric seems to perform similarly as improved PR with real generative models in Tab. 1? Also, Tab. 1 does not tell which dataset these models were trained and evaluated on.\nSince the proposed metric is composed of an outlier removal and the Top P&R evaluation, how much of the performance in toy/synthetic datasets can be explained by the outlier removal part? Would adding a similar outlier removal to existing improved PR and DC improve their performance in the toy experiments?\n\n2. I think detecting outliers might prove to be difficult in practice, compared to the toy experiments presented in the paper. Real datasets do not ground-truth “outlier labels” and in some datasets, e.g. in medical imaging, the distribution has a very long tail as there might be more data from patients without a certain disease than with the disease. In this case, modeling the patients with the disease might be more interesting than modeling patients without the disease. However, when evaluating Top P&R the contributions of the patients with the diseases might be suppressed as they are deemed as outliers.\n\n3. How does the dimensionality of the random projection affect the performance of the metric?\n\n4. Is the bandwidth parameter $h$ different for each model in Tab. 1? If so, is it possible that some models obtain artificially better scores because their outlier removal is different than for the other models? Are Top P&R scores with different bandwidth parameters directly comparable?", "limitations": "The authors adequately addressed the limitations of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The paper assesses a very important topic of evaluating generative models and proposes a metric that provides robust estimates of sample fidelity and diversity in several toy/synthetic examples. Additionally, bringing concepts from topological data analysis (TDA) to evaluation of generative models might open new future directions in metric research.", "weaknesses": "The paper is difficult to follow for readers that are not very familiar with TDA and it could be improved significantly by adding intuition and a visual introduction to the most important concepts TDA and precision and recall. Furthermore, the concept of persistent homology, persistence diagram and confidence band estimation are discussed on a general level in Sec. 2 but the discussion seems disconnected how these are related to the proposed metric – adding concrete and intuitive examples would make this section more approachable. Unfortunately, I believe that in the current state paper is not very accessible to a wider audience. \n\nThe novelty of the work is challenging to evaluate and potentially limited. In particular, the proposed metric seems to rely on the removal of outliers before evaluating the Top P&R, and the Top P&R part of the metric seems quite similar to previous work. Ideally, I would hope to see real world use cases with state-of-the-art models, where the metric provides more reliable evaluation (now Tab. 1 Top P&R yields very similar results to improved P&R).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer 5  \n**R5-1. Dataset used for Table 1:** (G1-2) CIFAR-10. We apologize for not mentioning the datasets. We revised the caption of Table 1 in the revised paper.  \n\n**Reason why TopP&R shows similar performance to P&R in Table 1:** (G1-3, G1-4). TopP&R shows better performance than the others when we apply it to more large-scale datasets, such as ImageNet and Baby ImageNet.  \n\n**Application of outlier removal to improved P&R and D&C:** We had already compared P&R and D&C with TopP&R by applying the outlier removal method (e.g., Local Outlier Factor or Isolation Forest) to these methods in Section I.2. The experiments were conducted using the same setup as the experiments depicted in Figure 4 of the main text. By examining the results in Figures A2 and A3, it becomes evident that applying the outlier removal method to P&R and D&C still does not make them robust to noise.  \n&nbsp;  \n**R5-2. How TopP&R behaves with minority sets (e.g., long-tailed distributions):** If we correctly understand the reviewer's question, it pertains to whether TopP&R can effectively capture differences when there are variations in a minority set of patient data samples within a larger patient dataset, where only a small number of samples exhibit a specific disease. In our **Section I.1** of Appendix, we have already conducted experiments comparing how various metrics, including TopP&R, respond when changes occur in the minority set. This comparison was carried out to assess how different metrics react to variations in such a scenario.  \n\nIn **Table A11 of Section I.1**, our utilized data assumes a situation where out of a total of 10 classes, six data classes are majority classes, containing a substantial number of samples, while the remaining four minority classes have relatively small data counts, resulting in a long-tailed data distribution. In this context, we conducted experiments by further reducing the number of samples within the minority set, aiming to observe how TopP&R and other metrics respond to even the slightest changes in this distribution. Contrary to the reviewer's anticipation, the experimental results revealed that, among the metrics, TopP&R is the most sensitive in reflecting evaluation values influenced by subtle differences in the distribution. As detailed in Section C in Appendix, due to the inherent nature of TopP&R, if the minority set possesses a topologically and statistically significant data structure, these data points are deemed crucial features in approximating support. Therefore, considering all minority sets as outliers when estimating TopP&R’s support is an incorrect notion.  \n&nbsp;  \n**R5-3. The impact of changing the dimension of random projection on TopP&R:** (G2-2).  \n&nbsp;  \n**R5-4. Comparison of TopP&R scores across different bandwidth parameters:** (G3) In addition to our responses regarding $c_\\alpha$ and $h$ in the general comments, $h$ being too small leads to the KDE with unstable topological features. On the other hand, $h$ being too large leads to the KDE with its topological features dissolved, so the choice of $h$ cannot be arbitrary but should be proper.  \n\n**Whether certain models can achieve better performance due to different settings of the bandwidth parameter in Table 1:** (G1-2, G1-3) As stated in the response to the general comments, we have demonstrated that TopP&R provides a generally consistent ranking with metrics such as FID and KID, which are commonly used to assess performance differences among models in the field of generative modeling.  \n&nbsp;  \n**R5-W1. The paper is not easily understandable for individuals unfamiliar with TDA (+ more discussion between confidence band and proposed metric):** We can relate to the feedback provided by the reviewers, as this aspect was a significant concern during the paper writing process. Considering the limited page and the nature of the conference to which we submitted the paper, our approach was to structure the paper in a way that enables readers to understand the robustness of TopP&R at a high-level without getting entangled in intricate details of TDA. However, it is important to note that some aspects of TDA have been intentionally condensed in the main paper. For readers seeking a more comprehensive understanding of our work, we have provided the detailed explanation of TDA in the appendix.  \n\nFurthermore, in response to the concerns raised by the reviewer, we will strive to provide more detailed explanations and visual introductions to address the aspects that require additional intuition. Our aim is to enhance the paper by offering a more comprehensive and visually informative presentation.   \n&nbsp;  \n**R5-W2. Novelty of this work:** Our proposed method introduces a novel theoretical approach by combining the traditional topic of consistency in statistical inference with topological data analysis. While existing literature has explored consistencies under i.i.d. settings without considering noise, and consistencies under noise frame framework, the focus on statistical inference in geometrical or topological settings with noise is relatively rare (e.g., Genovese, Christopher R., et al. (2012)). Our contribution lies in bridging this gap and providing a framework that incorporates both the consistency of topological analysis and the consideration of noise. This not only encompasses the theoretical properties of our method, called TopP&R, but also extends to identifying an appropriate noise framework and adapting statistical inference techniques for geometrical and topological data analysis.  \n&emsp;&emsp;\\- Genovese, Christopher R., et al. \"Manifold estimation and singular deconvolution under Hausdorff loss.\" (2012): 941-963.  \n\n**Real world use cases with SOTA model:** (G1-3)."}]

Author rebuttal:
We deeply appreciate your feedback. Your insights greatly helped us to refine our presentation and better illuminate the key advantages of our metric. Your input has also greatly enhanced our discussion about the metric's limitations, which not only clarifies our current work but also highlights promising future directions.

**Reviewers can find the additional experiments we conducted in the attached PDF file: Table B1\~5 and Figure B1\~5.**  
&nbsp;  
**G1. Detailed info. of Table 1 & Scalability to large-scale data with more recent generative models (ADM, StyleGAN-XL)**

**G1-1.** All the evaluation in this paper was conducted using **10k real and 10k fake samples**. 

**G1-2. Model ranking experiment (Table 1):** The models were trained on **CIFAR-10**. Here, our aim was to highlight the consistency of TopP&R across different embeddings and its congruence with widely accepted metrics in the field, FID. Based on the reviewer's recommendation, we additionally included **KID (Table B1)**. A foundational aspect of an evaluation metric is its capacity for stability and consistency. Ideally, the ranking it provides for different models should remain largely stable, irrespective of the embedding network used. Our results show that TopP&R consistently outperforms in terms of stability, as evidenced by the MHD values across different embedding networks (TopP&R: 1.33, P&R: 2.66, D&C: 3.0, MTD: 3.33). Furthermore, as generative models have developed with an emphasis on aligning with FID's (or KID’s), we posit that the rankings determined by FID (or KID) can, to some extent, reflect the true performance hierarchy among models, even if it doesn't capture it perfectly. Our results show that TopP&R displays the strongest alignment with FID and KID, which resonates with the historical trajectory of model development.

**G1-3. Ranking SOTA models (Table B3):** We evaluated generative models trained on **ImageNet**, including **ADM and StyleGAN-XL**. The results consistently show that TopP&R maintains stable rankings across various embeddings and aligns most closely with FID and **KID**. 

**G1-4. Mode dropping on Baby ImageNet (Figure B1):** In contrast to the CIFAR-10 outcomes where TopP&R produced results comparable to other metrics, **with the large-scale dataset**, both the sequential and simultaneous mode dropping scenarios highlight **TopP&R's precise assessment, which is in line with our observations from the toy experiments**. Conversely, P&R had difficulties in accurately evaluating the simultaneous mode drop.  
&nbsp;  
**G2. Random projection (RandProj)**  
**G2-1. The impact of RandProj (Table B4, Figure B3 and B4):** To discern if TopP&R's enhanced performance stems from RandProj, we applied RandProj to both P&R and D&C. This led to inconsistent results in model ranking (Table B4), suggesting that **the performance of TopP&R is attributed to its distinct features, rather than just the RandProj**. This trend was further observed in the "shifting the generated feature manifold (Section 5.1.1)" (Figure B3), and the "Tolerance to Non-IID perturbations (Section 5.1.3)" (Figure B4). Both experiments were conducted with and without RandProj. Across these studies, a consistent theme emerged: both P&R and D&C failed to exhibit a stable evaluation trend in relation to noise, irrespective of the use of RandProj.

**G2-2. The effect of dimensionality (Figure B5):**
We examine the **impact of changing the dimension of RandProj** on TopP&R. We employed the same experimental setup as the "Mode dropping experiment (Figure B1)" and observed how the evaluation trend of TopP&R changes by altering the dimension of RandProj. By Johnson-Lindenstrauss Lemma (Johnson et al. 1986), it is known that RandProj preserves the topological structure of data, given the output dimension isn't excessively reduced (see Section G.3). When we use higher dimensions than 32, our results show that the trend of TopP&R stays consistent and stable. However, using higher dimensions leads to greater computational overhead. Thus, for pragmatic computation purposes, we opted for a dimension of 32.  
&nbsp;  
**G3. Choosing parameters of TopP&R**  
**G3-1. Confidence level of bootstrap bandwidth ($\alpha$):** $\alpha$ is not a typical tuning hyperparameter, but it holds a statistical interpretation of the probability or degree of confidence in accommodating errors, noise, and similar factors. The most commonly chosen values include $\alpha=0.1, 0.05, 0.01$, corresponding to confidence levels of 90%, 95%, and 99% respectively. In our experiments, we consistently employed $\alpha = 0.1$.

**G3-2. Kernel bandwidth ($h$):** $h$ serves to adjust the initial KDE probability values of features. $h$ being too small leads to the KDE with unstable topological features, and $h$ being too large leads to the KDE with its topological features dissolved, so $h$ should be properly chosen. Ideally, $h$ can be chosen using topologically significant features, but this incurs computational challenges. Instead, in Algorithm 2 of Section H.4 we use a heuristic and automatic approximation of $h$ that effectively adapts to various embeddings and arbitrary datasets. From various experimental results, this $h$ performs effectively in practice.  
&nbsp;  
**G4. Different methods rather than KDE**
Please see our Remark 4.4. The super-level set of KDE excludes topological noise by the KDE filtration: a confidence band establishes a statistical threshold for distinguishing between signal and noise in the KDE filtration. However, if we switch from using the super-level set of KDE to other methods like k-NN or any alternative, the estimated support is not a result of KDE filtration. Consequently, the topological noise present cannot be assessed through KDE filtration, and the confidence band cannot be applied to determine signal or noise in support. Consequently, the guarantee of excluding topological noise is lost when using different methods; this is not known yet.

Meta-review: This paper develops a novel, promising approach to evaluating deep generative models. The authors have done a good job addressing all of the reviewer's concerns.
Decision: Accept (poster)