Robust Manifold Estimation Approach for Evaluating Fidelity and Diversity
Authors: Pum Jun Kim
Yoojin Jang
Jisu Kim
Jaejun Yoo
Venue: Submitted to ICLR 2023 (2023)

Abstract: We propose a robust and reliable evaluation metric for generative models by introducing topological and statistical treatments for a rigorous support manifold estimation. Existing metrics, such as Inception Score (IS), Frechet Inception Distance (FID), and the variants of Precision and Recall (P&R), heavily rely on support manifolds that are estimated from sample features. However, the reliability of their estimation has not been seriously discussed (and overlooked) even though the quality of the evaluation entirely depends on it. In this paper, we propose Topological Precision and Recall (TopP&R, pronounced “topper”), which provides a systematic approach to estimating support manifolds, retaining only topologically and statistically important features with a certain level of confidence. This not only makes TopP&R strong for noisy features, but also provides statistical consistency. Our theoretical and experimental results show that TopP&R is robust to outliers and non-independent and identically distributed (Non-IID) perturbations, while accurately capturing the true trend of change in samples. To the best of our knowledge, this is the first evaluation metric focused on the robust estimation of the support manifold and provides its statistical consistency under noise.

Under review as a conference paper at ICLR 2023
ROBUST MANIFOLD ESTIMATION APPROACH FOR EVALUATING FIDELITY AND DIVERSITY
Anonymous authors Paper under double-bling review
ABSTRACT
We propose a robust and reliable evaluation metric for generative models by introducing topological and statistical treatments for rigorous support manifold estimation. Existing metrics, such as Inception Score (IS), Fréchet Inception Distance (FID), and the variants of Preci sion and Recall (P&R), heavily rely on support manifolds that are estimated from sample features. However, the reliability of their estimation has not been seriously discussed (and overlooked) even though the quality of the evaluation entirely depends on it. In this paper, we propose Topological Precision and Recall (TopP &R, pronounced "topper"). which provides a systematic approach to estimating support manifolds, retaining only topologically and statistically important features with certain level of confidence. This not only makes TopP &R strong for noisy features, but also provides statistical consistency. Our theoretical and experimental results show that TopP& &R is robust to outliers and -oneindeeenndnt and identically distributed (Non-IID) perturbations, while accurately capturing the true trend of change in samples. To the best our knowledge, this is the first evaluation metric focused the robust estimation of the support manifold and provides its statistical consistency under noise.
INTRODUCTION
In keeping with the remarkable improvements of deep generative models (Karras al. 2019 2020
202I Brock et al. 20188 Ho et al. 2020; Kingma & Welling 2013) Sauer et al. 2022; 2021 Kang
Park 2020), evaluation metrics that can well measure the performance of generative models have
also been continuously developed (Salimans et al.) 2016 Heusel et al. 2017 Sajjadi et al. 2018
Kynkäänniemi al., 2019 Naeem al. (020). For instance, Inception Score (IS) (Salimans et al.
2016) measures the Kullback-Leible divergence between the real and fake sample distributions. Fréchet Inception Score (FID))Heusse et al. 2017) calculates the distance between the real and fake support manifolds using the estimated mean and variance under the multi Galtiian assumption. The original Precision and Recall (Sajjadi et al. (2018) and its variants (Kynkäänniemi al. 2019) Naeem et al. 2020) measure the fidelity and diversity by investigating whether the generated image belongs to the real image distribution and the generative model can reproduce all the real images in
the distribution. respectively.
Considering the eminent progress of deep generative models based on these existing metrics, some may question why we need another evaluation study. In this paper, we argue that we need more reliable evaluation metrics now precisely, because deep generative models have reached sufficient maturity. To provide more accurate and comprehensive ideas and to illuminate a new direction of improvements in the generative field, we need a more robust and reliable evaluation metric. In fact, has been recently reported that even the most widely used evaluation metric, FID, sometimes doesn't tt match with the expected perceptual quality, fidelity, and diversity, which means the metrics are not al. 2022). In addition to this, in practice, not only do generated samples but also real data in the wild often contain lots of artifacts (Pleiss et al. 20200] Li et al. 2022). and these have been shown to seriously perturb the existing evaluation metrics, giving false sense of improvements (Naeem et al. 2020 Kynkäänniemi et al. 2022).
always working properly (Ky (Kynkäänniemi
An ideal evaluation metric must capture the real signal of the data, while being robust to noise. Note that there is an inherent tension in developing metrics that meets these goals. On one hand, the metric should be sensitive enough so that it can capture real signals lurking in data. On the other hand, it

---

Under review as a conference paper at ICLR 2023
Real images Generated
Feauue embeller
(a) Probubility support estimation via KD and bootstrap
(b) Confidence band estimation
(c) TopP and TopR evalustion
images
Ah Kemel Density Estimatoe for fotrree
(b) Given Confidence interval o, count significant difference and estimating cy satisfying
lim Inf n- P. (llA. All < cx) 21-a
Boutstru Sampling
Birth
Topological noise
A
0g
8/
P
Topological signal
Ph
|1.
supp Pn [cz,
Deach
(c) Given (P)p for real features x and (tpp (Q) for generated featurcs y.
2 10, E (P) n (U))
TopPx (9))
(Upp(P)
2 10, (()))
2n 1(X e (upp(() (P))
(Upp(Q)
(X))
E 1(X, E (P))
Figure 1: Illustration of the proposed evaluation pipeline. The proposed metric TopP & is defined in the following three steps: (a) Confidence band estimation with bootstrapping in section (b)
Robust support estimation, and (c) Evaluation via TopP& &R in section
must ignore noises that hide the signal. However, sensitive metrics are inevitably susceptible to noise to some extent. To address this, one needs systematic way answer the following two questions: 1)
what is signal and what is noise? and 2) how do we draw a line between them?
One solution can be to use the idea of statistical inference and topological data analysis (TDA). Topological data analysis (TDA) (Carlsson 2009) is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data. A key object in TDA is persistent homology. which observes how long each topological feature would survive over varying resolutions and provides a measure to quantify its significance; i.e., if some features) persist longer than others over varying resolutions, we consider them as topological signal and vice versa as noise.
In this paper, propose to combine these ideas to form more robust and compact feature manifold and overcome various issues from the conventional metrics. Our main contributions are as follows: we introduce (1) an approach to directly estimate a support manifold via Kernel Density Estimator (KDE) derived under topological conditions; (2) a new metric that is robust to outliers while reliably detecting the change of distributions on various scenarios; and (3) a theoretical guarantee of consistency with robustness under very weak assumptions that is suitable for high dimensional data; (4) combining
a noise framework and a statistical inference in consissnstte under noise framework have studied in much literature, but not quite in geometrical or topological setting.
2 BACKGROUND
To lay the foundation for our theoretical analysis, we introduce the main idea of persistent homology and its confidence estimation techniques that bring the benefit of using topological and statistical tools for addressing uncertainty in samples. In later sections, we use these tools to analyze the effects of outliers in evaluating generative models and provide more rigorous way of scoring the samples based on the confidence level we set. For space reasons, we only provide a brief overview of the concepts that are relevant to this work and refer the reader Appendix [A] or (Edelsbrunner & Harer 2010 Chazal & Michel 2021 Wasserman 2018 Hatcher 2002) for further details.
2.1 NOTATION
For any x and r >> O, we use the notation (a(x,r) (y d(y. x) < <r} be the open ball in distance d of radius r. We also write B(z, r) when d is understood from context. For distribution P on Rd, we let sPpp(P) == (a E gd P(B(I, r)) > for all r > 0} be the support of P. Throughout the paper, we refer sPpp don't necessarily require the (geometrical) manifold structure on supp(P). For a kernel function K: Rd + R, a dataset X' (XI,... X,) C Rd and bandwidth h > 0, let the kernel density estimator (KDE) as (x) n > K (==X.) and we let the average KDE Ph E [p)).
support manifold of P, simply support, manifold, but
2

---

Under review as a conference paper at ICLR 2023
We denote by P, the probability distributions in R2 of real data and generated samples, respectively. And we use = {Xi....; [x]] C I2 and y = {Y,,... Ym } C Rd for real data and generated samples possibly with noise, respectively.
2.2
CONFIDENCE BAND ESTIMATION
Statistical inference has recently been developed for topological data analysis (Chazal et al. 2013
2015 Fasy al. 2014). Topological data analysis consists of features reflecting topological characteristics of data, and it of question to distinguish features that are indeed from geometrical structures and features that are insignificant or due to noise. To statistically separate topologically significant features from topological noise, we use a confidence band. Given the significance level a, let confidence band cx be the bootstrap bandwidth of l| - Pi Then satisfies lim inf, IP (1li%. lo < cx) 2 a, as in Proposition in Appendix This confidence band can be used determine simultaneously significant topological features while filtering out noise features. The algorithm for computing cx is described below.
X
Algorithm 1 Confidence Band Estimator
1: # KDE: kernel density estimator
15:
for E [min(0), max(0)] do count
2: # R.S.: random sample with replacement
16:
3: # k: number of repeats
17:
for element E 0 do
4: # 0: set of difference
18:
# count significant difference
5: Given X
{X1, X x,)
19:
if element > q then count count
6: PA. (D
20:
7: for iteration = 1, ...., do
21:
end if end for
8:
compute θ with bootstrap samples x== R.S.S. times from X
22:
9:
23:
# define the band threshold
10:
# 5% replaces population density
24:
if count = a then
11: Ph KDE(X")
25:
qa 9 end if
12:
Append 0 with v lin- 1ho
26:
27:
end for
13: end for
28:
# define estimated confidence band
14: # grid search for the confidence band
29: Co 9a/ /n
ROBUST SUPPORT MANIFOLD ESTIMATION FOR RELIABLE EVALUATION
Current evaluation metrics for generative models typically rely on strong regularity conditions. For example, they assume samples are well-curated without outliers or adversarial perturbation, real or generative models have bounded densities, etc. However, practical scenarios wild: both real and generated samples can be corrupted with noise from various sources, and the real data can be very sparsely distributed without density.
In this work, we consider more general and practical situations, wherein both real and generated samples can have noises that come from sampling procedure, remained uncertainty due data model, etc. For more detailed discussions on the philosophy our metric, please see Appendix E
3.1 TOPOLOGICAL PRECISION AND RECALL
In the ideal case where we have full access to the probability distributions P and Q, we define the precision and the recall of distributions as
precision, (Q) Q (supp(P)),
(Ccall((P)::((supp))))))
These correspond to the max precision and the max recall in Sajjadi et al. (2018). We tweak the precision precision, Q (supp(P) supp(Q)) /Q (supp(Q)), and define the precision of data points
Em 1Y, E supp(P) n ssupp(Q))
precision,,
En (Y, supp(Q))
3

---

Under review as a conference paper at ICLR 2023
which is just replacing the distribution Q by the empirical distribution プラ= δy; of Y in the precision. We similarly define the recall of data points as
recallo(2) Ei= 1 (X; supp(Q) n supp(P)) Ex= (X, E supp(P))
However, in practice, supp(P) P) and supp(Q) are not known a priori and need to be estimated, and since we allow noise, these estimates should be robust to noise. For this, we use the kernel density estimator (KDE) and the bootstrap bandwidth to robustly estimate the support. Given hn > 0 and
significance level a E (0, 1), we use the KDE Ph., (2) == nng Ei= K
x==; of X, and we use the hn
bootstrap bandwidth cx of || P%. 1l from Section 2 Then we estimate the support of P by the superlevel set cx as suppp(P) - P%.,, [cx, 00). Similarly, we let @i... (1)
mh 2) K Amm
be the KDE of y and let ey be the bootstrap bandwidth of | Them alo and then we use supp(Q) कل. [ey, oo). Using the superlevel set at cx allows to filter out noise whose KDE values are likely to be small.
For the robust estimates of the precision, we apply the support estimates to the precision of data points, and define the topological precision (TopP) as
Σラ= 1 (Y3 E supp(()) suupp(()) ΣØ= 1 (pn., (V3) > cx, Th.m. (Vz) > ey) Σラ= 1 (Y, E supp(())
TopP, x (D)
Eド= (9 (Y₁) > cy)
And similarly define the topological recall (TopR) as
Top = Ei= (@h. (X,) cy. Ph. (Xi) > cx)
E 1 (ph,, (X,) > cx)
The kernel bandwidths h. and hm are hyperparametens that users need choose. We also provide guideline select the optimal bandwidths h, and hm in practice. (See Appendix F33
3.2 BANDWIDTH ESTIMATION USING BOOTSTRAPPING
Using the bootstrap bandwidth cx' as threshold is the key part of our estimators TopP &R for robustly estimating supp(P). As we have seen in Section 2 the bootstrap bandwidth ex acts as a threshold for filtering out the topological noise in topological data analysis. Analogously, using ca as a threshold allows to robustly estimating (Ppp(P). When X, is an outlier, its KDE value (X,) is likely to be small, and the KDE values at the connected component generated by X; is likely to be small as well. So those components from outliers are likely to be removed in the estimated support P/. [exx oo). Higher dimensional homological noises also removed. Hence. the estimated support denoises topological noise and robustly estimates supp(P). See Appendix B for more detailed explanation.
Now that are only left with topological features of high confidence. this allows to draw analogies to confidence intervals in statistical analysis, where the uncertainty of the samples is treated by setting the level of confidence. In the next section, we show that Top? &R not only gives more reliable evaluation score for generated samples but also has a good theoretical properties.
4 CONSISTENCY WITH ROBUSTNESS OF TOPP&R
The key properties of TRpP&R is consistency with robustness. The consistency ensures that, the precision and the recall we compute from the data approaches the precision and the recall from the distribution as we have more samples. The consistency allows to investigate the precision and the recall of the full distributions only with access to finite sampled data. TopP &R achieves consistency with robustness, that is, the consistency holds with the data possibly corrupted by noise. This is due to the robust estimation of the support with the kernel density estimator with confidence bands. This section is devoted to the theoretical analysis of consistency of TopP& & with robustness.
We demonstrate the statistical model for the data and the noise. Let P. Q, X, y be in Notation in Section [ and let X y be real data and generated data without noise. X', JL, X², y are understood multisets, iee., elements can be repeated. We first assume that the uncorrupted data are IID..
4

---

Under review as a conference paper at ICLR 2023
Assumption 1. The data X {Xp..... ..... X and yo (YP,..... ...... ..n) are IID from P and Q, respectively.
In practice the data is often corrupted with noise. We consider the adversarial noise, where some fraction of data are replaced with arbitrary point cloud data.
Assumption 2. Let {pk JREN be a sequence of nonnegative real numbers. Then the observed data X and y satisfies 22) = npn, and | = mpm
In the adversarial model, we control the level of noise by the fraction p. but do not assume other conditions such as IID or boundedness, to make our noise model very general and challenging.
For distributions and kernel functions, we assume weak condition, detailed in Assumption Al and A2 in Appendix [ Under the data and the noise models, TopP& &R achieves consistency with robustness. That is, the estimated precision and recall is asymptotically correct with high probability even if up to a portion of // / or 1/ 1/ym are replaced by adversarial noise. This is due to the robust estimation of the support with the kernel density estimator with the confidence band of the persistent homology.
Proposition 1. Suppose Assumption A hold. Suppose hn - 0, nhn - oo, nh, P 0, and similar relations hold for hm Pm- Then
[TopP> x ())) precision p (>)) + O,
[TopRy(22) - recallo 0,
in probability.
Theorem 2. Under the same condition as in Proposition !
[Topl', x x))) precisionp(()) 0,
ToppBy (X) recallo (P) + 0,
in probability.
Our theoretical results Proposition and Theorem 2 are novel and important in several perspetives. These results are among the first theoretical guarantees for evaluation metrics for generative models as far as we are aware of. Also, as in Remark 3 assumptions are very weak and suitable for high dimensional data. Also, robustness adversarial noise is provably guaranteed.
EXPERIMENTS
A good evaluation metric must correctly capture the changes of the underlying data distribution. To examine the performance of evaluation metries, we carefully select a set of experiments for sanity checks. With toy and real image data, we check 1) how well the metric captures the true trend of underlying data distributions and 2) how well the metric resist perturbations applied to samples. 'The shaded area of the figures denotes the standard deviation for ten trials.
5.1 SANITY CHECKS WITH TOY DATA
Following Naeem et al. 2020), we first examine how well the metric reflects the trend of ) moving away from and whether is suitable for finding mode-drop phenomena. addition to these, we newly design several experiments that can highlight TopP & R ''s favorable theoretical properties of consistency with robustness in various scenarios.
5.1.1 SHIFTING THE GENERATED FEATURE MANIFOLD
For this experiment, generate samples for X' NN I) and y ~ (M11,, I) in R64 where 1 is vector of ones and is an identity matrix. We then examine how each metric responds to shifting ) with μ [] while there are outliers at 3 R64 for both X' and y (Figure 2 Here, find that both improved P&R and D&C behave pathologically when there are outliers. Since these methods are based on the k-nearest neighbor algorithm and ignore the fact that there can be outliers in both real and fake data, they inevitably overestimate the underlying support when there are outliers. For example, when X' lies between ) and the outlier at y = 3, Recall returns high-diversity score, even though the true supports of X' and y are actually far apart. In addition, P&R does not reach in high dimensions even when X' = y. Nacem et al. (2020) circumvented these problems by proposing D&C that always use X' (the real data distribution) cases is assumed to have fewer outliers than y (the fake data distribution). However, there is no guarantee that this will be the case in practice. When there is an outlier in X, D&C also returns incorrect high-fidelity score at μ > 0.5. On the other hand, TopP& & shows a stable trend unaffected by outliers, demonstrating the robustness of our method.
reference point, which in most
5

---

Under review as a conference paper at ICLR 2023
Fideliy
Diverity
1.0
0.7
0.7
Oullier
0.5
0.2
Togll Procias a
Topei Roall Ouei
0.0
0,0
-05
Centar of the take destribttions
Center the fake distributionjj
Figure 2: Behaviors of evaluation metrics for outliers real and fake distribution. The horizontal axis corresponds to the value of μ..
Real distribution
Real distribution
Fake distribution
Fake distribution
Fidelity
Diversity
Fidelity
Diversity
1.4
1.0
Toaft
1.4
Precidin
Seadiy 1
0.7
Covmage 3
1.2
0.7
0.0
0.0
0.0
0.4
0.5 0.8
1.0
0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.2 0.4 0.6 0.8
1.0
0.0
0.2
0.4
0.6 0.8
Concecrraion the first mode
Concentration the first mode
Concentrazion on the first modt mode
Concentration the firs mode
(a) Sequential mode dropping
(b) Simultaneous mode dropping
Figure 3: Behaviors of evaluation metrics for (a) sequential and (b) simultaneous mode dropping
scenarios. The horizontal axis shows the concentration ratio the distribution centered 0.
Fidelity
Diversity
Fidelity
Diversity
0.8
1.00
1.00
0.6
0.75
0.6
0.50
Togfi
0.50
Togi
04
Covauge Quwd)
0.2
0,25
0.0
0.00
0.00
0.0
0100
0.04
0.08
0.12
0.15
0.000
0:04
0.08
0.12
0.15
0.00
0.04
0LO8
0.12
0,15
0.00
0.04
0.88 0,12 Q15
Raiio of auther in the (%))
Ratio outber in the data (%)
oa outberin the data (%)
Ratio of outher in (%6)
Non-IID scatter noise
Non-IID swap noise
Figure 4: Behaviors of evaluation metrics on Non-IID perturbations. We replace certain percentage of real and fake data (a) with random uniform noise and (b) by switching.
5.1.2 SEQUENTIALLY AND SIMULTANEOUSLY DROPPING MODES
For this experiment, we consider the mixture of Gaussians with seven modes in R&1, We simulate mode-drop phenomena by gradually dropping all but one mode from the fake distribution y that is initially identical to X' (Figure 3). As in the illustration of mode-drop experiment, when the number of samples in a particular mode decreases, we kept the number of samples in constant so that the same amount of decreased samples are supplemented
the first mode which leads fidelity to be fixed to 1. From the result, we observe that the values of P Precision sinn fail to saturate, i.e., mainly smaller than and the Density fluctuates a value greater than 1 indicating their instability and unboundedness. In terms of diversity, Recall 1 does not respond to the simultaneous mode drop, nor does the improved metric Cover age show fast decay as the reference line. Compared these methods, Top? performs well, being held at the upperbound of in sequential mode dropping, and TopR also decreases closest to the reference line in simultaneous mode drops.
5.1.3 TOLERANCE TO NON-IID PERTURBATIONS
Robustness to perturbations is another important aspect should consider when designing metric. Here, we test whether TopP& &R behaves stably under two variants noise cases; 1) scatter noise: replacing X, and Y, with uniformly distributed noise and 2) swap noise: swapping the position between X, and Y,- These two cases all correspond to the adversarial noise model of Assumption
2 We set X' ~ N(2 = 0, I) E 04 and y ~ N(μ I) E R&4 where μ = 1, and thus an ideal evaluation metric must return zero for both fidelity and diversity. In both cases, find that P & and D&C are more sensitive while TopP &R remains relatively stable until the noise ratio reaches 15% of the total data, which is clear example of the weakness of existing metrics to perturbation (Figure

---

Under review as a conference paper at ICLR 2023
Generated Images
VGG16
Sw.V
5.0
2.0
Density
4.0
Coverage
1.3
Precision
2,0
1.0
Recall
1,0
0.5
TopP
TopR
Decreased Y Incressed V
0.0
0.2
01.4
0.6
0.8
1.0
0.0
0.2
01.4
0.6
01.8
1.0
Trancation Psi
Truncation Psi 4
Figure 5: Behaviour of metrics with truncation trick. The horizontal axis corresponds to the value of a denoting the increased diversity. The images are generated via StyleGAN2 with FFHQ dataset.
5.2 SANITY CHECK WITH REAL DATA
Now that we have verified the metrics on toy data using Gaussians, we test them on real data. Just like in the toy experiments, we concentrate how the metrics behave in extreme situations, such as outliers, mode-drop phenomena, perceptual distortions, and etc. We also test the image embedder for evaluation, including pretrained VGG16 (Simonyan & Zisserman 2014), Inception V3 (Szegedy et al. 2016), and SwAV (Morozov et al., 2020). Here, linear random projection to 32 dimension is additionally used for TopP&R. &R. For more experimental details, please refer to the Appendix F.1
5.2.1 RESOLVING FIDELITY AND DIVERSITY
To test whether TopP& & responds appropriately to the change in the underlying distributions in real scenarios, we test the metric on the generated images StyleGAN2 (Karras et al. 2020) using the truncation trick (Karras et al. 2019), As shown in Figure 5 every time the distribution is transformed by v, TopP& &R responds well and shows consistent behavior across different embedders with bounded scores in [0, 1], which are important virtues an evaluation metric. On the other hand, Density gives unbounded scores (fidelity > 1) and shows inconsistent trend depending on the embedder. Because Densi ty is not capped in value, it is difficult interpret the score and know exactly which value denotes the best performance (e.g.. g.. in our case, the best performance is when TopP & = 1). Since TopP &R pays more attention the consistent behavior of model by examining what the model primarily generates, rather than relying on the entire sample, which contains results by chance, the fact that TopP is kept at 1.0 means that StyleGAN2 produces high-quality images most of the time. Thus, this behavior ("TopP remains constant") does not mean that TopP is inferior regular precision for checking the trade-off between fidelity and diversity, but rather reveals its property focusing on different perspectives than the others.
5.2.2 SEQUENTIALLY AND SIMULTANEOUSLY DROPPING MODES IN CIFAR- 10
We conduct an additional simultaneous mode drop experiment to verify TopP&R's actual sensitiveness on the real data set (CIFAR-10). The performance of each metric (Figure A2 is measured with the identical data while simultaneously dropping the modes of nine classes of CIFAR- 10. Since the number of the images dropped in each step is identical, the trend of ground truth diversity should linearly decrease. Here, P&R metric captures the simultaneous mode dropping better than D&C because this time random drop of the modes has reduced the area of the estimated fake manifold. On the other hand, TopP& R best captures the true trend of decreasing diversity on average, consistent with the toy result in Figure] In addition, we perform the experiments on dataset with long-tailes distribution and find that TopP &R captures the trend well even when there are minority sets (Appendix G.2.. This again shows the reliability of TopP &R.
5.2.3 ROBUSTNESS TO PERTURBATIONS BY OUTLYING FEATURES
To demonstrate the robustness of our metric against the adversarial noise model of Assumption 2 we test both scatter-nowe noise and swap noise scenarios with real data. In the experiment, following Kynkäänniemi et al. (2019), we first classify inliers and outliers that are generated by StyleGAN (Karras et al., 2019). For scatter noise we add the outliers to the inliers and for swap noise we swap the real FFHQ images with generated images. Under these specific noise conditions, Preci sion
7

---

Under review as a conference paper at ICLR 2023
Fidelity
Diversity
Fidelity
Diversity
ou 03B
1.02
210
240
L.10
1.70
Trpit
Reialive rediverriio
0..88
0.91
0.96
0:00 DOG
0.10
0115 0,20 0,25 0.30
1.50
1.80
1,00
1.25
0003
006 0.10 015 0.20 025 0.30
0.56
0.00
1.00
000
0.55
Prodece
0.10
0.15
0.30 0.25 0.30
1,42
1,00
1.14
0:01
DO5 0I0 015 0.00 022 0.30
outlo (nf
Rrilo of outther inlees (N)
Pusso outfier Indiiis (50)
Radio of outteer inlles
@)) Non-IID Scatter noise
(b) Non-IID Swap noise
Figure 6: Comparison of evaluation metries on Non-III perturbations using FFHQ dataset. We replaced certain ratio of A' and y (a) with outliers and (b) by exchanging features.
(a) Caussaan noise
(b) Caussaa blur
(c) Black rectagle
Flaore
Tepht
DC 00 -m
DC (1-0)
DC 0-
HD
PED
0.0
0.0
1.0
20
30 0.0
10
2.0
30 0.0
1.0
20
30
Noise intenslyy
Noise intenill
Naise intensily
Figure 7: Verification of whether TopP& &R can make an accurate quantitative assessment of noisy image features. Gaussian Noise, gaussian blur, and black rectangle noise are added on the FFHQ imageset and embedded with T4096.
shows similar or even better robustness than Density (Figure 6 On the other hand, Coverage is more robust than Recall. In both cases, TopP & shows the best performance, resistant to noise.
5.2.4 SENSITIVENESS TO THE NOISE INTENSITY
One of the advantages of FID (Heusel al. 2017) is that it is good at estimating the degrees of distortion applied to the images. Similarly, we check whether the Fi-score based on TopP &R provides reasonable evaluation according to different noise levels. As illustrated in Figure 7 X and ) are sets of reference FFHQ features and noisy FFHQ features, respectively. The experimental results show that TopP&T &R actually reflects well the different degrees of distortion added to the images.
5.2.5 RANKING BETWEEN GENERATIVE MODELS
One of the major caveats with two-score metrics is that they make difficult to rank between different models; e.g., which model is better? High fidelity with low diversity or low fidelity with high diversity? In the case of traditional precision and Recall, this problem could be solved by using F1-score, which is the harmonic mean of fidelity and diversity. However, unlike the traditional ones, the F1 F1-score based on P P or D&C does not provide reliable or stable score due their inherent instability and unboundedness. Thanks to its stability and robustness to various perturbations, we find that TopP & R-based score offers consistent ranking with FID under various embedding networks
(Table To quantitatively compare between the similarity of rankings across varying embedders by different metrics, we have computed Hamming Distance (HD) (Appendix F.4 where lower HD indicates more similarity, TopP R, P&R, and D&C have HDs of 1.33, 2.66, and 3.0, respectively, From this, TopP provides the most consistent ranking across varying embedders (consistent to Section 5.2.1b.
RELATED WORKS
Persistent homology and deep learning. Topology shows various potentials in the field of deep learning by introducing the noisy information, and a technique for GAN evaluation, Chen et al. (2017) introduces a method for
perspective the support estimation, new distance function robust to
All GAN models used in the experiment follow the settings in StudioGANN PyTorch-Sture Torch-StudiGGAN is an open-source library under the MIT license (MTT), which under the NVIDIA source code license.
8

---

Under review as a conference paper at ICLR 2023
Model
StyleGAN2
ReACGAN
BigGAN
PDGAN
ACGAN
WGAN-GP
FID (L)
3.78 (1)
3.87
(3)
31.54 (4)
33.39 (5)
107.68 (6)
VGG16 Inception
TopP&R (t)
0,9769 (1)
0.8457 (2)
0.7751 (3)
0.7339 (4)
0.6951 (5)
0.0163 (6)
D&C (1)
0.9626 (2)
0,9409 (3)
1,1562 (1)
0.4383 (4)
0.3883 (5)
0.1913 (6)
P&R (1)
0.6232 (1)
0,3320 (2)
0.3278 (3)
0,1801 (4)
0.0986 (5)
0.0604 (6)
TopP&R (t)
0.9754 (1)
0.5727 (3)
0.7556 (2) 0,4021 (4)
0,3463 (5)
0.0011 (6)
D&C (T)
0,9831 (3)
1.0484 (1)
0,9701 (4)
0,9872 (2)
0.8971 (5)
0.6372
P&R (1)
0.6861 (1)
0.1915 (3)
0.3526 (2)
0.0379 (4)
0.0195 (5)
0.0001 (6)
TopP&R (f)
0,9093 (1)
0.3568 (3)
0,5578 (2)
0,1592 (4)
0.1065 (5)
0.0003 (6)
SWAV
D&C (T)
1.0732 (1)
0.9492 (3)
1.0419 (2)
0.6328 (4)
0.4565 (5)
0.0721 (6)
P&R (1)
0.5623 (1)
0.0901 (3)
0.1459 (2) 0.0025 (4) 0.0000 (6)
0.0002 (5)
Table 1: Generative models ranked by FID and Fi-scores based on TopP&R, D&C, and P&R, respectively. The X' and y are embedded with InceptionV3, VGGI6, and SwAV. The number inside the parenthesis denotes the rank based on each metric.
approximating the support of distribution using general density estimator and the Hausdorff distance and a new visualization method for support. Chazal et al. (201I) proposes distance-fo-meaaurr. robust Wasserstein distance function for perturbation, existing distance functions are not robust to outliers. For the evaluation, one of the recent metric called -TTo--Divergent (Barannikov al. (021) uses the summation (or in another word statistics) of the life-length of homology to score which manifold is containing more important topological signals. While MTop-Divergence directly use persistent homology to score the deep-learning models, we employ topology to estimate a robust and stable manifold.
alternative the characteristic that
Evaluation metries. Various evaluation metrics for generative models have been recently proposed (Salimans et al. 2016 Heusel et al.] 20177] (Sjjadi et al., 2018]] Kynkäänniemi et al.) 2019 Nacem al.) 22200] Borji 2022). One of the earliest methods is Inception Score (IS) (Szegedy) ct al. 2016), which measures the divergence of generated samples on the Inception embedding space. However, IS fails to capture the simultaneous mode drop and only considers the population distribution. Fréchet Inception Distance (FID) (Heusel et al. 2017) measures the difference in the means and variances of the real and fake features. Since FID assumes the Mullti-auussia distribution of the features, if the true feature distribution is not normally distributed, the estimation becomes highly unreliable. Unlike IS and FID, which give single score, some metrics separate the score into two components, the fidelity and diversity Sajjadi et al. (2018); |Kynkäänniemi et al. 22019;; Nacem ct al.) (2020). While Topological Precision and Recall (TopP &R) falls into this category, unlike the others, it does not assume strong regularity conditions.
CONCLUSIONS
Recently, many works have been proposed to score the fidelity and diversity of generative models. However, none of them has focused on an accurate estimation of supports even though this is one of the key components in the evaluation pipeline. this paper, we proposed topological precision and recall (TopP& R) that provides a systematical fix for robustly estimating the manifold by employing topological and statistical ideas. Our thcoretical and experimental results showed that TopP &R serves as robust and reliable evaluation metric under various embeddings and noisy conditions, including mode collapse, outliers, and Non-IID perturbations.
REFERENCES
Serguei Barannikov, Ilya Trofimov, Grigorii Sotnikov, Ekaterina Trimbach, Alexander Korotin, Alexander Filippov, and Evgeny Burnaev. Manifold topology divergence: a framework for comparing data manifolds. Advances in Neural Information Processing Systems, 34, 2021.

---

Under review as a conference paper at ICLR 2023
Ali Borji. Pros and cons of gan evaluation measures: New developments. Computer Vision and Image Understanding, 215:103329 2022.
Andrew Brock, Jeff Donahue, and Karen Simonyan, Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1rXiv: 1096, 2018.
Gunnar Carlsson. Topology and data. Bull. Amer. Math. Soc. (N.S.), (6(2))46622))0000 2009, ISSN 0273-0979. doi: 10. 01099//0027300999999-01222999999 50273-0979-09-012295
URL
Frédéric Chazal and Bertrand Michel. introduction to topological data analysis: Fundamental and practical aspects for data scientists. Frontiers Artif. Intell., 4:667963, 2021. doi: 10.338 667963. URL
frai. 2021
wtw.:aii..02...... 667963
Frédéric Chazal, David Cohen-Steiner, and Quentin Mérigot. Geometric inference for probability measures. Foundations of Computational Mathematics, 11(6):733-751, 2011.
Frédéric Chazal, Brittany Fasy, Fabrizio Lecci, Alessandro Rinaldo. Aarti Singh, and Larry Wasserman. On the bootstrap for persistence diagrams and landscapes. Modelirovanie i Analiz Informacionnyh Sistem, 20, 11 2013. doi: 10.18255/1818. 1115-203--6111-1222
Frédéric Chazal, Brittany Terese Fasy, Fabrizio Lecci, Alessandro Rinaldo, and Larry Wasserman. Stochastic convergence of persistence landscapes and silhouettes. J. Comput. Geom., 662):140-161, 2015.
Frédéric Chazal, Brittany Fasy, Fabrizio Lecci, Bertrand Michel, Alessandro Rinaldo, Alessandro Rinaldo, and Larry Wasserman. Robust topological inference: Distance to measure and kernel distance. The Journal of Machine Learning Research, 18(1)584455584, 2017.
Yen-Chi Chen, Christopher R Genovese, and Larry Wasserman. Density level sets: Asymptotics, inference, and visualization. Journal of the American Statistical Association, 1186520):684-1666 2017.
Herbert Edelsbrunner and John L.. Harer. Computational topology. American Mathematical Society, Providence, RI, 2010. ISBN 978-0-8218-49255 doi: 10. 10.1099//b,/069, URL https://dooi
org/10. 1090 mbk 069,
An introduction.
Brittany Terese Fasy, Fabrizio Lecci, Alessandro Rinaldo, Larry Wasserman, Sivaraman Balakrishnan. and Aarti Singh. Confidence sets for persistence diagrams. Ann. Statist., (2(6):2301-2399 2014. ISSN 0090-5364. doi: 10.1214/14AASS1222 14 AOS1252
Richard W Hamming, Error detecting and error correcting codes. The Bell system technical journal,
29(2))147-160, 1950.
Allen Hatcher. Algebraic topology. Cambridge University Press, Cambridge, 2002. ISBN 0-521- 79160-X; 0-521-79540-0.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30. 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
William Johnson, Joram Lindenstrauss, and Gidcon Schechtman. Extensions of lipschitz maps into banach spaces. Israel Journal of Mathematics, (492):129-1333 1986.
Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. Advances in Neural Information Processing Systems, 33:21357-21369 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEÉ/CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019.
10

---

Under review as a conference paper at ICLR 2023
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 8110-8119, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
Jisu Kim, Jaehyeok Shin, Alessandro Rinaldo, and Larry Wasserman. Uniform convergence rate of the kernel density estimator adaptive intrinsic volume dimension. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3398-3407, PMLR, 09-15 Jun 2019. URL https:::
bkttps://proceeiing htpppceeiiiing....p/ppp//. amlarngs.m..........mmm press/ v97 mikkimml 9e html
Diederik P Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
urX2vv1112.661 14, 2013.
Michael R. Kosorok. Introduction to empirical processes and semiparametric inference. Springer Series Statistics. Springer, New York, 2008. ISBN 978-0-337-74977-8, doi: 10.1007/ 978-0-387-74978-5 URL htRhttts:://dd
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. arXiv preprint arXiv:2203.06026, 2022.
Zhengwen Li, Runmin Wu, and Tao Gan. Study on image data cleaning method of early esophageal cancer based on vgg_nin nin neural network. Scientific Reports, 12(1):1-10, 2022.
Stanislav Morozov, Andrey Voynov, and Artem Babenko. On self-supervised image representations for gan evaluation. In International Conference on Learning Representations, 2020.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. In International Conference on Machine Learning, pp. 7176-7185. PMLR, 2020.
Michael H. Neumann. Strong approximation of density estimators from weakly dependent observations by density estimators from independent observations. Ann. Statist. 26(5):2014- 2048, 1998. ISSN 0090-5364. doi: 10.12144a0//1024691333 URL /ttps:://oi..rr//10 2114//05/102469133
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33: 17044-17000 17056, 2020.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucie, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in Neural Information Processing Systems, 31, 2018
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems, 34, 2021.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. arXiv preprint arXiv:2202.00273, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409 1556, 2014.

---

Under review as a conference paper at ICLR 2023
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer viston and pattern recognition, pp.
2016.
George Terrell and David W Scott. Variable kernel density estimation. The Annals of Statistics, pp. 1236-1265, 1992.
A.W. van der Vaart. Asymptotic Statistics. Asymptotic Statistics. Cambridge University Press, 2000. ISBN 9780521784504
google 2t/DbookkS..GUUUEEEEMSR jWcC
Hubert Wagner, Chao Chen, and Erald Vuçini. Efficient computation of persistent homology for cubical data. In Topological methods in data analysis and visualization pp. 91-106. Springer, 2012.
Larry Wasserman. Topological data analysis. Annu. Rev. Stat. Appl., 5:501-535, 2018. ISSN 2326-8298. doi: 10.1 oatp:/rwwwweaatiiitsoo000000--0- URL URL/attps::/doo..oo/// 1146 //annnnnnnvvaaeae stat istiissiiissss satatttsttss10111111111111111111 100045
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A MORE BACKGROUND ON TOPOLOGICAL DATA ANALYSIS
Topological data analysis (TDA) [Carlsson 2009) is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data. A key object in TDA persistent homology, which quantifies salient topological features of data by observing them in multi-resolutions.
Filtration. A filtration a collection of subspaces approximating the data points at different resolutions, formally F = [F8 C ] such that 61 S 62 implies that F6, C F&g Typically, filtration defined through function related to data. Given a function f: IRd - R, we consider its sublevel filtration (f- SER or superlevel filtration [ oc) )EER.
Persistent homology. Persistent homology multiscale approach to represent the topological features, and is represented in the persistence diagram. For filtration F and for each nonnegative k. we track when k-dimensional homological features (e.g.. O-dimensionn connected component, 1-dimension: loop, 2-dimension: cavity,... appear and disappear in the filtration. As increasing or decreasing 8 in the filtration [, if a homological feature appears at F6 and disappears at Fa, then say that it is born at b and dies at d. By considering these pairs {(b, d)) points in the plane (R U {+² we obtain a persistence diagram. From this, a homological feature with a longer life length, d b, can be treated shorter life length as a topological noise, which lies near the diagonal line { ó E RJ.
significant feature in the data set, and homological feature with
As diseussed above, homological feature with long life-length an important information in topology while the homology with short life-length can be treated as a non- non-significant information or noise. The confidence band estimator provides the confidence set from the features that only includes topologically and statistically significant (statistically considered as elements in the population set) under certain level of confidence. One way of constructing the confidence set uses the superlevel filtration of kernel density estimator and the bootstrap confidence band. Let X' = {Xi, X2, ..... X,, } as given points cloud, then the probability for the distribution points can estimated via KDE defined as following: (z) nhd
1 Σ K( K(= <(=xx=) where is the bandwidth and d as a dimension of the
space. We derive estimated likelihood of X' with KDE and likehood of ÿ with using bootstrapped samples X Now, given the significance level and h >> 0, confidence band qx be bootstrap bandwidth of a Gaussian Empirical Process (van der Vaart 2000 Kosorok| 2008), /nilph p% oloo. Then it satisfies P(Vvil fr - pallx < qxx 2 2I- at, as in Proposition 4 in Section C Then the ball of persistent homology centered at Pr, and radius cx = xx/ / V in the bottleneck distance da is BBg
valid confidence set as lim inf IP (PeBa (B, (x). 2 1 a. This confidence set has further
interpretation that in the persistence diagram, homological features that are above twice the radius 2cx from the diagonal are simultaneously statistically significant.
B DENOISING TOPOLOGICAL FEATURES FROM OUTLIERS
Using the bootstrap bandwidth cx as threshold is the key part of our estimators TopP &R for robustly estimating supp(P), (P). When the level set p7, [cxx, x)) is used, the homology of pr [cx, oe) consists of homological features whose (birth) 2 cx and (death) < cx. which are the homological features in skyblue area in Figure In this example, we conisder three types of homological noise, though there can be many more corresponding different homological dimensions.
There can be O-dimensional homological noise of (birth) < cx and (death) < cx,, which is the red point the persistence diagram of Figure Al This noise corresponds the orange connected component on the left. As in the figure, this type of homological noise usually corresponds to outliers.
There can be O-dimensional homological noise of (birth) > cx and (death) > cx, which is the green point in the persistence diagram of Figure Al This noise corresponds the connected component surrounded by green line on the left. As in the figure, this type of homological noise lies within the estimated support, not like the other two.
13

---

Under review as a conference paper at ICLR 2023
Data and KDE levels
Persistence Diagram
1.5
Birth
0-dim
0-dim
0-dim
1-dim
0
Ca
1.5
Death
Figure A1: To robustly estimate the support, we use the bootstrap bandwidth ca to filter out topological noise (orange) and keep topological signal (skyblue). Then TopP&R & is computed on this support.
There be 1 1-dimensional homological noise of (birth) Cx and (death) cx. which
is the purple point in the persistence diagram of Figure A1 This noise corresponds to the purple loop on the left.
These homological noises satisfy either their (birth) cx and (death) cx or their (birth) ca and (death) simultaneously with high probability, so those homological noises are removed
in the estimated support 1h [cxx,o) o0), which is the blue area in the left and the skyblue area in the
right in Figure A1
We would like to further emphasize that homological noises are not restricted to O-dimension lying outside the estimated support (red point in the persistence diagram of Figure A1) 0- dimensional homological noise inside the estimated support (green point in the persistence diagram
of Figure A1 I-dimensional homological noise can also arise, and the bootstrap bandwidth ca
allows to simultancously filter them.
C ASSUMPTIONS ON DISTRIBUTIONS AND KERNELS
For distributions, we assume that the order of probability volume decay P(B(x,r)) at least nd
Assumption A1. For all E Supp(P) and y E (Q))(Q),
inf P (B((x,)) r))
lim Q (B(,,r)) 0.
0,
pd
pdd
Remark 3. Assumption
analogous to Assumption 2 of Kim et al. 2019), but is weaker since the
condition is pointwise on each T. And this condition is much weaker than assuming density
on Rd. for example, distribution supported
low- www-dimensiona manifold satisfies Assumption A1
This provides framework suitable for high dimensional data, since many times high dimensional
data lies low dimensional structure hence its density R cannot exist. See Kim et al. (2019 for more detailed discussion.
For kernel functions, we assume the following regularity conditions:
Assumption A2. Let K Rd + be a nonnegative with |[]], = 1. |K]] ||]]| < oc, and satisfy the following:
14

---

Under review as a conference paper at ICLR 2023
(1) ((()) > O.
(2) K has compact support.
(3) is Lipschitz continuous and of second order.
Assumption
A2 allows to build a valid bootstrap confidence band for kernel density estimator (KDE). See Thcorem 12 of (Fasy et al. 2014) or Theorem 3.4 of (Neumann 1998)
Proposition (Theorem 3.4 of (Nuumann 1998)) Let X {XI..... ..)...,,,,)).. be IID from a distribution P. For h > 0, let Pr. P% be kernel density estimator for & and its bootstrap X respectrively, and for a E (0, 1), let ex be the a bootstrap quantile from Vnhi l1 Pilloc For hn +
(Vare) li - -hhn llao > ex xxx==at
log n nh?
4428
P
Assumption A1
A2 ensures that, when the bandwidth hn - average KDEs are bounded away
from O.
Lemma 5. Let P be a distribution satisfying Assumption [7]] Suppose is a nonnegative function satisfying ((0)= and continuous Suppose {hn Jnen with h, 2 and hn + 0 is given. Then for all x E SUpp(P),
lim inf Ph.. (x) O.
Proof. Since K(0)= > 0 and K continuous at there ro 0 such that for all y E B(0,
K(y) 2 K (0) > 0. And hence
(x(x)) = / hd h
--
dP())
K(0),
( x (x-y
E B(0,To)) dP(y)
2hd
h
((0)
2 2hd
P (B((,)roh))
Hence as hn + 0,
lim inf Ph. (x) 0.
D DETAILS AND PROOFS FOR SECTION
Let pr. be the KDE on X For finite set X, we use the notation cx for a-bootstrap quantile satisfying P (llix,,... Pxx ">x*,,,,,l. ll > -xr (x)
a, where X is the bootstrap sample from X. For a distribution P. we use the notation cP.o for a-quantile satisfying P (llp,, Ph., Il >> cP.a = IID samples
1-a, where Pr., is kernel density estimator of IID samples from P. Hence when
from P, the relation of Proposition 4 may not hold.
Lemma 6.
(1) Under Assumption
and A2
Pn. |||]
p1 Palloo <
hd
(2) Under Assumption 2 and A2
Cx 22 -0 Pn +
Pr. 1og(1/0) 1/)) nh?d
< Cx S 0x + ( Pn.
Pn log(1/8) nhhd
(3) Suppose Assumption
A2 hold, and suppose nh, AP2 - 0. Then with probability
a 26,
ll ph loc cx2 CPa-d-
15

---

Under review as a conference paper at ICLR 2023
Proof. (1)
First, note that
Ph
Ph
1
Σ
1
K
(
K
( (x-Xp")
x
X!
)
nha
h
h
Then under Assumption A2
T X;
I x
11
Pn
100
I nhd
K
h
h
1
n
|| oc I (X, + x!-
nhd
Then from Assumption 2
Ei=1 (X: & x npnn and hence
K 1lo Pn
Ppn 11 S
hd
(2)
Let Xn X3 be bootstrapped samples of X, X with the same sampling with replacement process.
Let ph. Ph be KDE of 24, and X, respectively. And, note that
pr - p - 11 - ph bol S I - Palloo + 11ph -hh llo
Let La be the number elements where X;, and xg differ, i.e., Lo = 1 12,00 = \ La Binomial (n, p.), and
then
|1 Lo nhd
11th pholo0
And hence,
pb ph 1l. - 1| - ph bll S || (mpn Lt)
nhd
Then by using subgaussian tail bound, with probability 1 - 5,
Pn log (1/8) nh.2d
pn. Pholoo - -1i0 - Fhil 500 On
Hence this implies
o
( Pn hd
log(1/1)) (1/5) nh2d
Pri
log(1/8) nh?dd
Cx 3.9
+
S cx S cx + +0
hd
(3)
Since X IID samples from P, with probability 1 a 28,
|| Phlloc < CP, S cx2 + O nhd
(v
log(1/a) /a) nied
Now, note that Cx = +
and hence
(
(Vogg(1/a)
(og(1/(a a + 0)) 2 Ω
log((a + 6)/a)
Cxe,a Cxo,a+ō =6
nhe
Vnhd
16

---

Under review as a conference paper at ICLR 2023
Then under Assumption
2 since nh,"p? =[1) and hn o(1),
p ph ||
x pin phll + +lph
5h llo
< cx9 +0
nh?
Pn
Pn (1g(1/0)
S cx0 - 0
+
ha
nh2d
S Cx,a
Pn
Pn log(1/0)
S ax- +0
hd
nh22d
S cP.a228
Corollary 7. Suppose Assumption 7
A2 hold.
(1) Suppose nh p + 0. Then with probability 1
25,
Ph. 120p, 25. 00) Icx 00) supp(Ph..)
(2) Suppose mhm Pm + 0. Then with probability - - 26,
4nm [2cQ a- 25, 00) C 9 le),, 00) C supp(Qh....
Proof. (1) From Lemma
6 || phll < <x S CP.a-28- This implies
Ph.. [2cp,a-26 00) C. Icx 0o) C supp(Ph..).
(2) can be proven similarly to (1).
Claim 8. For nonnegative measure j and sets A, B, #(An B) ((C D) < +
Proof.
-
= +
S +
From here, let Ps, and Qm be the empirical measures on X' and V, respectvelly i.e., Pa 1 Ei= 8x, and Q... = - Em= oy,- Lemma 9. Suppose Assumption 7 hold.
(1) Let AC Rd. Then with probability 1-
Pn - PI o())
(2) Let An C Rdd be sequence of set satisfying A, + 6. i.e lim sup, A, 0. Then (An) in probability.
17

---

Under review as a conference paper at ICLR 2023
Proof. Since P(An) I P(U A,) and Ui=1 A 0 as well, assume that A, 0, i.e. A₃ C An+1 for all and n==1 An 0.
(1)
Let pd be the empirical measure on X i.e., P - "' 5x. By using subgaussian tail bound, with probability
(V/
po - (A))=
71
And Pr -Pal (A) is expanded as
1 n
Pn P (A)
Σ - n
T(X? A)-
Under Assumption 2 Ei=1 I (X: x
npn, and hence
Pn - - (A)
Σ
1(X, A) - I(X) E A)
==1
Σ
1 (X; + x²) Pn-
i=1
Therefore, with probability 1- 8,
Pa P(A) On + log(L/0)
72
(2) Note that (A,,) can be bounded as
(A())) S P(An) + (A,)) P(An).
Fix 8 > 0 and 0. Since lim, P(An) 0, choose N1 such that P(AN,) e, and
we can choose N2 such that for all n 2 N2, P (IP, - PI (AN,) <<) 2 -- Then for all max (N1, N2), with probability 6,
Pr(An) & (AN))) & P(AN) + Pr (ANi) < 2e.
Hence PA(An) + 0 in probability.
Claim 10. Suppose Assumption 1 AI A2 hold.
(1)
Qm (Fx,, o0) n Till ley, o0)) Qm (supp(P) n supp(Q))) + 0 in probability.
Qm (9mnl loy, o0)) Qm (supp(Q))) + 0 in probability.
(3)
Qm (ph Iex, 20) n g lcy. oc)) Q (supp(P)) in probability.
(4)
Qm (en, o0)) 1 in probability.
18

---

Under review as a conference paper at ICLR 2023
Proof. (1)
From Lemmal 7 with high probability,
Qm (Ph,, [2cp, 0o) 7 1hm [20q. 00)) s Qm (P oc) n Iimm ley; o0))
I Qm (supp Pr...) n supp (Qhm.)
Then from the first inequality, combining with Claim gives
gives
Qm (Ph, [cx, 00) n 9 [cy, o0)) - Qm (supp n suup(()))
2 mm (Ph,, [2cr, xc) n Ihm 220 o0)) Qm (supp (P) Nsupp)()
2 (Qm (supp(P) P\P.. (2cr, 00)) + Qm (supp(Q) VT 220 x)))
And from the second inequality, combining with Claim 8
gives
Qm (P |ex, oo) T...
le,, ley, xo) Qm (supp(P) supp
< Qm (supp( Ph.,) ). n susup(Quun (hmp))))..)) Qm (supp P) supp(()))
s Qm (supp( Pr,.) (supp(P) + Qm ( suppp(())
And hence
10m (f,llcx o0) n Tnm loy, o0)) Qm (supp(P) nsupp(())
I max {Qm (supp PT [2pp, 20)) + Qm (supp(Q)\ (2cq,
Qm (supp( PA) Asupp P)) + Qm (supp(Qn.)) (I (upp))))))
Now, note that from Lemma 5 implies that for all æ E supp PP), lim inf, ph.. (x) 0. Ph., (z) 2cp for large enough n. And hence
supp(P)) Ph.. [2cp, 00) - 0. And similar argument holds for supp(Q)).. 1% [2cQ, 00). supp(Q) 9i... [2cq, 00) 0 as well.
Then from Lemma 9
Qm (supp( P) [2pp, o0)) o1)). Qm (supp(Q)) [cq
2oo. o0)) = (1()
Also, since K has compact support, for any x & supp P), & supp(Pn.) once Inn < d(x, supp( (P).
Hence supp(PM. (Ms,,,)ssppp((((C) (supp(P) 0, and similarly supp(Q (suppp)))))))
as well. Then again
with Lemma 9
Qm (supp(Pr..) (sspp(()) P)) or()),
Qm (supp(Qn...) (ssp:(kk....supQ)))) or).
And hence
Qm (Pm,) [cx, (cx, oo) n d (ey, x0)) + Qm (supp(P) (supp(Q)) in probability.
(2)
This can be done similarly to (1).
(3)
Lemma (1) gives that with probability 1 6,
IQm (Supp(P) supp((Q)) - Q (supp(P)) S o(1). Hence combining with (I) gives the desired result.
(4)
Lemma (1) gives that with probability 1- 8,
1Qm (supp (Q)) 11 s o(1). Hence combining with (2) gives the desired result.
Proof of Proposition
Now this is a combination of Claim 10 (1) (2).
Proof of Theorem
Now this is a combination of Claim 10 (3) (4).
19

---

Under review as a conference paper at ICLR 2023
E
PHILOSOPHY OF OUR METRIC & PRACTICAL SCENARIOS
E. PHILOSOPHY OF OUR METRIC
All evaluation metrics have different resolutions and properties. Here, we designed our proposed metric with the philosophy of evaluating the performance more conservatively based on (topologically and statistically) certain things. More specifically, in a real situation, there may be outliers in the data or samples we receive, noise may be present, and many other problems may arise due to various other unexpected causes. In these situations, two approaches can be used in the assessment. One is to accept ignorance and use all the data together, the other is to systematically select and exclude as much unreliable information as possible and only use reliable information. We chose the latter because we thought seeing a conservatively consistent result had its own merits (At least we think our approach is worth investigating, showing different aspects that have not been explored before).
E.2 PRACTICAL SCENARIOS
From this perspective, we present two examples of realistic situations where outliers exist in the data and filtering out them can have a significant impact on proper model analysis and evaluation. With real data, there are many cases where outliers are introduced into the data due to human error (Pleiss) jet al. 2020 Li et al. 2022). Taking the simplest MNIST as an example, suppose our task of interest generate 4. Since image number 7 is included in data set number 4 due to incorrect labeling (see Figure of (Pleiss et al. 20200), the support of the real data in the feature space can be overestimated by such outliers, leading to an unfair evaluation of generative models (as in Section 5.1.2 and 5.2.3): That is, the sample generated with weird noise may be in the overestimated support, and existing metrics without taking into account the reliability of the support could not penalize this, giving a good score to poorly performing generator.
similar but different example when noise or distortions in the captured data (unfortunately) behave adversely on the feature embedding network used by the current evaluation metrics (as in Section Sections5.1.2) 5.1.2 and 5.2.3p; e.g., visually it is the number 7, but it is mismapped near the feature space where there are usually 4 and becomes an outlier. Then the same problem as above may occur. Note that in these simple cases, where the definition of outliers is obvious with enough data, one could easily examine the data and exclude outliers priori to train generative model. In the case of more complicated problems such as the medical field (Li et al. 2022). however, it is often not clear how outliers are to be defined. Moreover, because data are often scarce, even outliers are very useful and valuable in practice for training models and extracting features, making it difficult to filter outliers in advance and decide not use them.
On the other hand, we also provide an example where it is very important to filter out outliers in the generator sample and then evaluate them. To evaluate the generator, samples are generated by sampling from the preset latent space (typically Gaussian). Even after training is complete and the generators' outputs are generally fine, there's latent area where generators aren't fully trained. Note that latent space sampling may contain samples from regions that the generator does not cover well during training ("unfortunate outlier"). When unfortunate outliers are included, the existing evaluation metrics may underestimate or overestimate the generator's is performance than its general performance. (To get around this, it is necessary to try this evaluation several times to statistically stabilize it, but this requires a lot of computation and becomes impractical, especially when the latent space dimension is high.)
Especially considering the evaluation scenario in the middle of training, the above situation is likely to occur due to frequent evaluation, which can interrupt training or lead to wrong conclusions. On the other hand, can expect that our metrie will be more robust against the above problem since it pays more attention to the core (samples that form topologically meaningful structures) generation performance of the model.
20

---

Under review as a conference paper at ICLR 2023
F EXPERIMENTAL DETAILS
F.1 IMPLEMENTATION DETAILS OF EMBEDDING
We summarize the detailed information of our embedding networks implemented for the experiments. In Figure 3 4 A2 and 6 P&R and D&C are computed from the features of ImageNet pre-trained VGGI6 (tc2 layer and TopP &R computed from features placed in R with additional random linear projection. In the experiment in Figure 5 the SwAV embedder additionally considered, We implement ImageNet pre-trained Inception V3 (fc layer), VGG16 (fc2 layer), and SwAV embedding networks with random linear projection to 32 dimensional feature space to compare the ranking of GANs Table [ The random projection characterized by preserving the information about distances and homological features defined in the higher dimensional spaces by Johnson Linenstrauss Lemma (Johnson al. 1986).
F.2 CHOICE OF CONFIDENCE LEVEL
For the confidence level a, we would like to point out that is not the usual hyperparameter to be tuned: It has statistical interpretation of the probability or the level of confidence to allow error, noise, etc. The most popular choices are = 0.1, 0.05, 0.01, leading to 90%, 95%, 99% confidence. We used a 0.1 throughout our experiments.
F.3 ESTIMATION OF BANDWIDTH PARAMETER
As we discussed in section since TopP & estimates the manifold through KDE with kernel bandwidth parameter h. we need to approximate it. The estimation techniques for f are as follows: (a) a method of selecting that maximizes the survival time (S(h) or the number significant homological features (N (h)) based information obtained about persistent homology using the filtration method, (b) a method using the median the k-nearest neighboring distances between features obtained by the balloon estimator (for more details, please refer to Chazal et al. (2017). Wagner et al. 20112) and Terrell & Scott (1922)) Note that, the bandwidth h for all the experiments in this paper are estimated via Balloon Bandwidth Estimator.
For (a), following the notation Section A the ith homological feature of persistent diagram be (b,, de), then we define its life length as 4
d, at kernel bandwidth h. With confidence band Ca (h), we select h that maximizes one of the following two quantities;
N(h) # (i: 4(h) > c. (h)), S(h)
Σ1((m) c. (h)]+-
Note that, we denote the confidence band c, as c, (h) considering the kernel bandwidth parameter of KDE Algorithm
For (b), the balloon bandwidth estimator is defined as bellow:
Algorithm 2 Balloon Bandwidth Estimator
1: # h: bandwidth; KND: kth nearest distance; idx: index 2: Given X' = XX,, }
3: for idx 1,2,...,111 do
4:
Compute 1.2 distance between Xidx and X
5:
d(Xido;X) X) X| sdw i.e. d(Xidz; X) = {..., 11, Xida), d( Xide Xida), , d(Xias Xide+1), .....
XII.
6:
7: Define the kth nearest neighbor distance by sorting in ascending order
8: KND sort(d((a d(Xida. {()
9: end for
10: Given kth nearest distance set: KND (KND), KND2, ..ND,,,,,.
KND,)
11: # Define the estimated bandwidth h
h median( KND)
21

---

Under review as a conference paper at ICLR 2023
F.4 MEAN HAMMING DISTANCE
Hamming distance (HD) Hamming 1950 counts the number of items with different ranks between and B, then measures how much proportion differs in the overall order, i.e. for A; E A and B, E B, HD(A, B) = > 1fk: A, + B, where n is the number of items in list or В, and the number of differently ordered elements. The mean HD is calculated as follows to measure the average distances three ordered lists: Given three ordered lists A, B, and C
HD (HD(A, B) HD(A, C) HD(B, C))/3
F.5 EXPLICIT VALUES OF BANDWIDTH PARAMETER
Since our metric adaptively reacts to the given samples of and Q, we have two hs per experiment. For example, the translation experiment (Figure 2), there are 13 steps in total, and each time we estimate h for and Q. resulting in a total of 26 hs. To show them all at glance, we have listed all values one place. We will also provide the code that can reproduce the results in our experiments upon acceptance.
f- shift
shift ->
real h
8.79
8.60
8.58
8.67
8.63
8.71
8.66
8.73
8.54
8.63
8.54
8.75
8.78
fake h
8.46
8.86
8.60
8.49
8.44
8.80
8.55
8.63
8.74
8.61
8.58
8.83
8.60
Table Al: Bandwidth parameters h of distribution shift in Section 5.1.1
Steps
2
3
4
5
6
real h 10.5
10.6
10.5
10.7
10.5
10.8
10.6
11.0
9.82
9.57
9.29
8.78
Table A2: Bandwidth parameters h of sequential mode drop in Section
5.1.2
Steps
5
7
8
9
10
real h
10.5
10.6
10.6
10.8
10.3
10.6
10.7
10.6
10.4
10.4
10.6
fake
10.6
10.5
10.7
10.3
10.4
10.0
9.88
9.31
9.19
9.07
8.79
Table A3: Bandwidth parameters h of simultaneous mode drop in Section 5.1.2
Steps
0
2
3
4
5
6
real
8.76
8.71
8.78
8.59
8.85
8.71
8.61
8.97
9.00
8.95
Scatter
fake h
8.61
8.76
8.63
8.93
8.46
8.72
8.67
8.68
9.07
8.88
real
8.65
8.44
8.76
8.62
8.55
8.75
8.96
8.59
8.53
8.74
Swap
fake h
8.52
8.55
8.68
8.97
8.87
8.94
8.84
9.05
8.94
8,90
Table A4: Bandwidth parameters of non-IID noise perturbation Section 5.1.3
V
V
+
1
real h
6.26
6.41
6.31
6.23
6.06
6.24
6.20
6.61
6.39
6.28
6.26
fake h
1.04
1.67
2.36
2.90
3.41
3.93
4.31
4.90
5.51
5.60
6.46
Table A5: Bandwidth parameters h of FFHQ truncation trick in Section
5.2.
22

---

Under review as a conference paper at ICLR 2023
Steps 0
1
2
3
4
5
6
real
7.09
6.95
6,96
6.92
7.02
6.86
6.78
6.67
fake h
6.87
6.75
6.88
6.64
6.40
6.64
6.41
6,44
6.62
6.04
Table A6: Bandwidth parameters of CIFARIO sequential mode drop Section 5.2.2
Steps
3
4
5
6
7
8
9
10
6.78
6.95
7.05
6.62
6.85
6.51
6.94
6.91
6.80
7.00
6.78
fake
6.75
6.69
7.14
6.68
6.75
6.84
6.61
6.68
6.50
6.32
6.17
Table A7: Bandwidth parameters of CIFAR10 simultaneous mode drop in Section 5.2.2
Steps
0
real h
7.24
7.03
7.01
7.45
7.38
7.66
7.46
7.36
7.79
7.66
Scatter
fake h
5.41
5.77
5.67
5.49
5.71
5.65
5.82
6.08
5.81
6.02
real
7.25
7.20
6.99
6.67
7.07
6.96
7.03
6.94
6.83
6.86
Swap
fake h
6.87
6.63
6.60
6.91
6.74
6.77
6.92
6.83
6.83
6.82
Table A8: Bandwidth parameters h of FFHQ non-IID noise perturbation in Section 5.2.3
Noise intensity
0
real
6.52
6.27
6.41
6.49
Gaussian noise
fake h
6.32
6.15
4.06
2.64
real
6.08
6.26
6,26
6.47
Gaussian blur
fake h
6.12
5.13
4.60
3.40
real
6.06
6.28
6.49
6.44
Black rectangle
fake h
6.36
6.66
6.31
5.93
Table A9: Bandwidth parameters of FFHQ noise addition in Section 5.2.4
StyleGAN2
ReACGAN
BigGAN
PDGAN
ACGAN
WGAN-GP
real h 2.605
2.629
2.581
2.587
2.615
2.606
Inception
fake
2.226
2.494
2.226
2.639
2.573
2.063
real h 7.732
8.021
8.175
7.792
7.845
8.036
VGG16
fake
7,880
6.839
7.178
6.174
6,839
3,615
real h 0.774
0.765
0.785
0.780
0.780
0.822
SwAV
fake
0.740
0.667
0.6746
0.619
0.627
0.502
Table A10: Bandwidth parameters h of GAN ranking in Section 5.2.5
G ADDITIONAL EXPERIMENTS
G.1 TRUCATION TRICK
a parameter for the truncation trick and is first introduced in Brock et (2018) and Karras et 2211 We followed the approach Brock al. (2018) and Karras al. (2019). GANs generate images using the noise input which follows the standard normal distribution N (0, I) or uniform distribution ( 1). Suppose GAN inadvertently samples noise outside of distribution, then less likely sample the image from the high density area the image distribution p(z) defined in the latent space of GAN, which leads generate image with artifacts. The truncation trick takes
23

---

Under review as a conference paper at ICLR 2023
this into account and uses the following truncated distribution. Let be the mapping from the input to
the latent space. Let t [(=), and = [=)( where is either from (O, 7) or UI( - 1, 1). Then
we use w' ti + ( 2)) as truncated latent vector. If the value of to increases, then the degree
of truncation decreases which makes images have greater diversity but possibly lower fidelity.
G.2 SURVIVABILITY OF MINORITY SETS IN THE LONG-TAILED DISTRIBUTION
Before mode drop
After mode drop
Proportion of survival (before/ after filtering)
100% + 59%
100% + 57%
TopP&R (Fidelity / Diversity)
0.99 / 0.96
0.99 / 0.87 (9 p-p .b)
P&R (Fidelity / Diversity)
0.73 / 0.73
0.74 / 0.70 (3 p-p +)
D&C (Fidelity/ / Diversity)
0.99/ / 0.96
1.02 /.92 (2 p.p
Table A11: Proportion of surviving minority samples in the long-tailed distribution after the noise exclusion with confidence band cx. The p.p. indicates the percentage points.
An important point to check in our proposed metric is the possibility that small part of the total data (i.e., minority sets), but containing important information, can be ignored by the confidence band. We emphasize that since our metric takes topological features into account, even minority sets are not filtered conditioned that they have topologically significant structures. We assume that signals or data that are minority sets have topological structures, but outliers exist far apart and lack a topological structure in general.
To test this, we experimented with CIFAR1O, which has 5,000 samples per class. We simulate dataset with the majority set of six classes (2,000 samples per class, 12,000 total) and the minority set of four classes (500 samples per class, 2,000 total), and an ideal generator that exactly mimics the full data distribution. As shown in the Table A1l the samples in the minority set remained after the filtering process, mcaning that the samples were sufficient to form a significant structure, Both D&C and TopP &R successfully evaluate the distribution for the ideal generator, To check whether our metric reacts to the change in the distribution even with this harsh setting, we also carried out the mode decay experiment. We dropped the samples of the minority set from 500 to 100 per class, which can interpreted as an 11.3% decrease diversity relative to the full distribution (Given (I) ratio of the number of samples between majority and minority sets = 12, 000 2, 000
and (2) 80% decrease in samples per minority class, the true decay the diversity calculated as (1+6) x 0.8 11.3% with respect to the enitre samples). Here, recall and coverage react somewha less sensitively with their reduced diversities as 3 p.p. and 2 p.p., respectively, while TopR reacted most similarly (9 p.p.) to the ideal value. In summary, T TopP& &R shows much more sensitiveness to the changes in data distribution like mode decay. Thus, once the minority set has survived the filtering process, our metric is likely to be much more responsive than existing methods.
G.3 SEQUENTIAL AND SIMULTANEOUS MODE DROPPING WITH CIFAR-10
Diversity
Diversity
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
Tuxk
Tark
0.2
Becall
0.2
Rocal
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.3
1.0
Visualization on danaset
Concentration on the first mode
Concentration on the first mode
CIFARIO
(a) Sequential mode dropping
(b) Simullameous mode dropping
Figure A2: Comparison of evaluation metrics under sequential and simultaneous mode dropping scenario with CIFAR-10.
24


Reviews:
[{"review_id": "lw40dALZie4", "reviewer": "Reviewer_xs7v", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper contributes a topology-based variant of the precision and recall measures, originally introduced by [Sajjadi et al.](https://proceedings.neurips.cc/paper/2018/file/f7696a9b362ac5a51c3dc8f098b73923-Paper.pdf). The new method makes use of a confidence band analysis, inspired by methods from topological data analysis, to obtain more robust estimates of the underlying support. The proposed measures are thus less affected in the presence of certain outliers and specific types of perturbations. A suite of experiments comparing the new measure with existing methods shows that the proposed method can be gainfully employed in practice.\n", "summary_of_the_review": "While I appreciate the direction and subject of this paper, I cannot endorse it for publication in its current form, mainly due to a lack of clarity and intuition, as well as a missing delineation to previous work, which in turns makes it hard to assess the novelty  of the approach. I do believe that filtering outliers using the proposed method is a potential step in the right direction, but in its present form, the paper does not make a compelling case for this. This might change if the topological considerations were to be expanded in order to better discuss their potential benefits.\n", "strength_and_weaknesses": "I see the main **strength** of the paper in making the existing measures more robust towards certain sources of noise and perturbation. Employing a method from topological data analysis is a useful way forward to better characterise the underlying support manifold. The recent years have shown that a better understanding of generative models necessitates also a better understanding of the respective spaces involved; the method at hand provides a useful step in this direction.\n\nWhile I appreciate the research direction and consider this to be a timely, relevant contribution, the current version of the paper needs to be improved in the following areas:\n\n1. **Clarity (and intuition)**. Even though the method is described as topology-driven, there are almost no explanations of the underlying concepts of topology in the main text. While I am familiar with the cited literature, in particular concerning the reliable estimation of topological features, the paper does not provide sufficient explanations for its approach:\n    - Concepts from persistent homology are being used throughout the text but not explained. This will make the paper inaccessible to readers without prior exposure to the theory. I see two ways around this, the first involving a more detailed discussing of topological concepts, the second aiming to rephrase statements in a \"less topological\" way. For instance, \"homological features\" could probably be rephrased to \"connected components\" in this context.\n    - It seems to me that only *connected components* are being used to assess the topology. This needs to be clarified, in particular since the persistence diagram shown in Figure 1 works for higher dimensions as well.\n    - Likewise, there is no exploitation of multi-resolution structure in the data as far as I can tell. A single threshold is defined for extracting the superlevel set; this is more or less like a \"noise suppression technique\"; the differences in scale between modes/connected components is not exploited in any way.\n    - Given that the main feature of the new method is its purported robustness, more details about this calculation need to be provided. Section 3.2 is too terse for this.\n    - Fidelity and diversity as concepts need to be introduced at least briefly.\n\n2. **Novelty and delineation to existing concepts.** Reading the definitions in Section 3.1, my impression is that these are minor modifications of the existing measures, but potentially being calculated in a somewhat simplified space, i.e. the superlevel set of the data with low-persistence (irrelevant) connected components being removed. It is therefore important to provide a better delineation to the existing terms. Moreover, it appears that the main contribution of this paper consists in applying the confidence band calculation method to derive a parameter for removing outliers form an underlying space. If this is the case, would it not be possible to apply this method *as-is* for the \"ordinary\" definition of precision and recall? In some sense, all the experiments crucially depend on the outlier removal process, so I wonder what would happen if the outlier removal and the calculation of `TopP` and `TopR` would be disentangled.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "huljrDjxdhl", "reviewer": "Reviewer_YyJA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper claims that the previous metrics for evaluating generative models (e.g., FID, IS, Precision, and Recall) may not be reliable as they estimate support manifolds based on sample features, and sample features may contain outlier or noisy features. Thus, it proposes a new method for robustly estimating support manifolds using Kernel Density Estimation under topological conditions. The theoretical results show that the proposed metrics are 1) robust to outliers and Non-IID noise and 2) consistent. Finally, the experiments demonstrate relative robustness to outliers and Non-IID perturbations of features for the metrics. ", "summary_of_the_review": "Based on the points above, I think the paper should be enhanced in several aspects:\nHow such a metric will be useful to develop generative models considering the concerns that I mentioned about the metric may not reveal the problems of the model regarding fairness or the data itself?\n\nSubstantial missing details regarding experimental implementations and hyperparameter values should be provided.\n\nThe paper should also compare the relationship between different metrics when using self-supervised learned representations as they have been shown to better rank the generative models.\n", "strength_and_weaknesses": "<Strengths> \n\n1) The paper addresses an important topic as several metrics have been proposed by the community for evaluating generative models, but there is no global consensus on which metric can better rank the models. \n\n2) The theoretical analysis shows the robustness of the proposed metrics in the presence of outliers / noisy samples. Also, it shows that the proposed metrics are consistent.\n\n3) The experiments demonstrate some degree of noise/outlier robustness in both synthetic and real scenarios.\n\n<Weaknesses>\n\nAlthough the theoretical insights and the goal of the paper to “develop a metric robust to noise/outliers” are interesting, I think they may have a limited application due to the following points:\nIn practice, there is neither ground truth about a data point being an outlier nor about the number of modes. One uses the training dataset (real data denoted by X in the paper) to train the generative model and then uses the metrics to measure different aspects of the model like fidelity, diversity, etc. The proposed metric does not provide a metric that can identify noisy/outlier samples before training. Rather, it attempts to filter noisy/outlier samples in (X) and generated images (Y) when quantifying the quality of the trained model.  Now I have the following questions in this regard:\nThe paper mentions in section 3.1 that: “Using the superlevel set at C_x allows to filter out noise whose KDE values are likely to be small.” However, doing so may result in unwanted consequences. For example, in practical settings, datasets have a long-tailed and imbalanced distribution w.r.t different factors such as age, gender, race, etc. The minority sets in data are not necessarily outliers, but they are likely to have a low KDE value considering that real-world data usually lies on a low-dimensional manifold with disconnected parts in the high-dimensional space [1]. Thus, the proposed metrics may simply discard them and hide the potential problems of the generative model regarding fairness.\n\nI appreciate the experiment in Figs. 2 and 3 that shows precision and recall may not accurately reflect fidelity and diversity in the presence of outliers. However, as I mentioned above, there is no ground truth about a point in data being an outlier in practice, and the generative model gets trained on both inlier and outlier/noisy data. Thus, even if the proposed metric can filter noisy samples in X and Y, its predicted quantities may not reflect the true behavior of the model and its problems.\n\nBased on parts (a) and (b) above, it seems the theorems provided in the paper may also have limited usefulness in practice.\nAs far as I know, the truncation trick is proposed because the training dataset may not cover all of the latent space of the style GAN. Thus, randomly generating samples from some areas of the latent space of the trained model may result in low-quality/noisy samples. When increasing the diversity in Fig. 5, inevitably, the model will generate low-quality/noisy samples as well which should result in lower precision values. However, TopP remains constant, and the precision score decreases. Does this observation indicate that TopP may be inferior to the regular Precision score in some scenarios?\n\nThe proposed method needs several parameters that should be tuned while limited details and no code implementations have been provided. The paper only mentions references about how the values of ‘h’ are calculated for different experiments, but it does not state what those values have been. Also, different choices of ‘alpha’ in Alg. 1 will result in different C_x, which also determines the samples to keep/discard when calculating TopP/TopR. I believe that exploring how the proposed metrics change with different values of alpha/h and suggestions about how to choose them should be provided in the paper.\n\nI think the paper should also consider using self-supervised learned (SSL) representations beyond ImageNet pretrained classifiers to check the relationship between the metrics. It has been shown that SSL representations produce a more reasonable ranking than the latter [3].\n\n[1] Disconnected Manifold Learning for Generative Adversarial Networks, NeurIPS 2018.\n[2] On Self-Supervised Image Representations for GAN Evaluation, ICLR 2021.\n\nI would be happy to raise my score if the authors can address my concerns.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dXhoM1QY0q", "reviewer": "Reviewer_sgKE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a pair of robust metric for the evaluation of generative models, dubbed Topological Precision and Recall (TopP&R). The proposed metric is proved to achieve consistency with robustness. The effectiveness is validated through experiments on both synthetic and real data. ", "summary_of_the_review": "Due to the comments in \"Clarity, Quality, Novelty And Reproducibility\", I suggest \"marginally below the acceptance threshold\".\n\n\n-------------------------After rebuttal---------------------\nMany thanks for the authors rebuttal. I read the authors' feedback and other reviewers' comments. Although some of my concerns have been addressed, I am still sorry to suggest \"5 borderline reject\" due to the \"Clarity, Quality, Novelty And Reproducibility\".", "strength_and_weaknesses": "Strength: \n\nS1:  A new pair of robust metric for the evaluation of generative models is proposed.\n\nS2: Consistency with robustness is proved for the new metric. \n\nWeakness:\n\nW1: This paper has some parts not clearly written. There are some examples:\n\n(W1.1): The persistence diagram is strange. The x-label and y-label seem incorrect.\n\n(W1.2):In defining precision$_P(\\mathcal{Y})$ in Section 3.1, is $Q(supp(Q))$ really needed? \n\n(W1.3): What is $\\mathcal{P}$ in \"lim inf$_{n\\rightarrow \\infty}\\mathbb{P}\\big( \\mathcal{P}\\in\\mathcal{B} \\big)$\" in Page 13?\n\n(W1.4): How to understand \"homological features whose (birth) ≥ $c_X$ and (death) ≤ $c_X$\" in Page 13?\n\n(W1.5): In Assumption A1, it says \"(3) $K$ is Lipschitz continuous and of second order..\". Is this sentence complete? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject