Title: TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: GAN

Evaluation

Support Estimation
Abstract: We propose a robust and reliable evaluation metric for generative models called Topological Precision and Recall (TopP&R, pronounced “topper”), which systematically estimates supports by retaining only topologically and statistically significant features with a certain level of confidence. Existing metrics, such as Inception Score (IS), Frechet Inception Distance (FID), and various Precision and Recall (P&R) variants, rely heavily on support estimates derived from sample features. However, the reliability of these estimates has been overlooked, even though the quality of the evaluation hinges entirely on their accuracy. In this paper, we demonstrate that current methods not only fail to accurately assess sample quality when support estimation is unreliable, but also yield inconsistent results. In contrast, TopP&R reliably evaluates the sample quality and ensures statistical consistency in its results. Our theoretical and experimental findings reveal that TopP&R provides a robust evaluation, accurately capturing the true trend of change in samples, even in the presence of outliers and non-independent and identically distributed (Non-IID) perturbations where other methods result in inaccurate support estimations. To our knowledge, TopP&R is the first evaluation metric specifically focused on the robust estimation of supports, offering statistical consistency under noise conditions.

TopP&R: Robust Support Estimation Approach for Evaluating Fidelity and Diversity in Generative Models

Abstract

We propose a robust and reliable evaluation metric for generative models called Topological Precision and Recall ( TopP&R , pronounced 'topper'), which systematically estimates supports by retaining only topologically and statistically significant features with a certain level of confidence. Existing metrics, such as Inception Score (IS), Fréchet Inception Distance (FID), and various Precision and Recall ( P&R ) variants, rely heavily on support estimates derived from sample features. However, the reliability of these estimates has been overlooked, even though the quality of the evaluation hinges entirely on their accuracy. In this paper, we demonstrate that current methods not only fail to accurately assess sample quality when support estimation is unreliable, but also yield inconsistent results. In contrast, TopP&R reliably evaluates the sample quality and ensures statistical consistency in its results. Our theoretical and experimental findings reveal that TopP&R provides a robust evaluation, accurately capturing the true trend of change in samples, even in the presence of outliers and non-independent and identically distributed (Non-IID) perturbations where other methods result in inaccurate support estimations. To our knowledge, TopP&R is the first evaluation metric specifically focused on the robust estimation of supports, offering statistical consistency under noise conditions.

1 Introduction

In keeping with the remarkable improvements of deep generative models [1, 2, 3, 4, 5, 6, 7, 8, 9], evaluation metrics that can well measure the performance of generative models have also been continuously developed [10, 11, 12, 13, 14]. For instance, Inception Score (IS) [10] measures the Kullback-Leibler divergence between the real and fake sample distributions. Fréchet Inception Score (FID) [11] calculates the distance between the real and fake supports using the estimated mean and variance under the multi-Gaussian assumption. The original Precision and Recall [12] and its variants [13, 14] measure fidelity and diversity separately, where fidelity is about how closely the generated samples resemble real samples, while diversity is about whether a generative model can generate samples that are as diverse as real samples.

Considering the eminent progress of deep generative models based on these existing metrics, some may question why we need another evaluation study. In this paper, we argue that we need more reliable evaluation metrics now precisely, because deep generative models have reached sufficient maturity and evaluation metrics are saturated (Table 8 in [15]). Even more, it has been recently reported that even the most widely used evaluation metric, FID, sometimes doesn't match with the expected perceptual quality, fidelity, and diversity, which means the metrics are not always working properly [16].

Feature

embedder

(a)

Probability support estimation

via KDE and bootstrap

(b) Confidence band

estimation

(c) TopP and TopR

evaluation

Given

!

𝑠𝑢𝑝𝑝 𝑃

for real features

𝒳

and

!

𝑠𝑢𝑝𝑝 𝑄

for generated features

𝒴,

(c)

Real

images

Generated

images

𝜃

!

=

̂ 𝑝

"

- ̂ 𝑝

"

(!)

%

(a)

Kernel Density Estimator

for features

.

.

.

̂ 𝑝

"

Bootstrap Sampling

.

.

.

̂ 𝑝

"

(&)

̂ 𝑝

"

(!)

̂ 𝑝

"

(')

𝜃

&

=

̂ 𝑝

"

- ̂ 𝑝

"

(&)

%

𝜃

'

=

̂ 𝑝

"

- ̂ 𝑝

"

(')

%

𝑇𝑜𝑝𝑃

𝒳

(𝒴) ≔

∑

)*&

+

1(𝑌

)

∈ !

𝑠𝑢𝑝𝑝 𝑃 ∩ !

𝑠𝑢𝑝𝑝 𝑄 )

∑

)*&

+

1(𝑌

)

∈ !

𝑠𝑢𝑝𝑝 𝑄 )

𝑇𝑜𝑝𝑅

𝒴

(𝒳) ≔

∑

-*&

.

1(𝑋

-

∈ !

𝑠𝑢𝑝𝑝 𝑄 ∩ !

𝑠𝑢𝑝𝑝 𝑃 )

∑

.

1(𝑋

-

∈ !

𝑠𝑢𝑝𝑝 𝑃 )

𝒳

𝒴

Topological

signal

Topological

noise

-𝑐

!

+𝑐

!

̂ 𝑝

"

/&

[𝑐

0

, ∞)

Given confidence level

1 - 𝛼,

b

ootstrap conFidence band 𝑐

0

computed from

𝜃

&

, ⋯ , 𝜃

'

satisfies

.→%

ℙ ||

̂ 𝑝

"

-𝑝

"

||

%

< 𝑐

0

≥ 1 - 𝛼

(b)

𝑐

!

𝑐

!

𝐷𝑒𝑎𝑡ℎ

𝐵𝑖𝑟𝑡ℎ

!

supp P

!

supp Q

-*&

In addition, existing metrics are vulnerable to the existence of noise because all of them rely on the assumption that real data is clean. However, in practice, real data often contain lots of artifacts, such as mislabeled data and adversarial examples [17, 18], which can cause the overestimation of the data distribution in the evaluation pipeline. This error seriously perturbs the scores, leading to a false impression of improvement when developing generative models. See Appendix G.2 for possible scenarios. Thus, to provide more comprehensive ideas for improvements and to illuminate a new direction in the generative field, we need a more robust and reliable evaluation metric.

An ideal evaluation metric should effectively capture significant patterns (signal) within the data, while being robust against insignificant or accidental patterns (noise) that may arise from factors such as imperfect embedding functions, mislabeled data in the real samples, and other sources of error. Note that there is an inherent tension in developing metrics that meet these goals. On one hand, the metric should be sensitive enough so that it can capture real signals lurking in data. On the other hand, it must ignore noises that hide the signal. However, sensitive metrics are inevitably susceptible to noise to some extent. To address this, one needs a systematic way to answer the following three questions: 1) What is signal and what is noise? 2) How do we draw a line between them? 3) How confident are we on the result?

One solution can be to use the idea of Topological Data Analysis (TDA) [19] and statistical inference. TDA is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data. A key object in TDA is persistent homology, which observes how long each topological feature would survive over varying resolutions and provides a measure to quantify its significance; i.e. , if some features persist longer than others over varying resolutions, we consider them as topological signals and vice versa as noise.

In this paper, we combine these ideas to estimate supports more robustly and overcome various issues of previous metrics such as unboundedness, inconsistency, etc. Our main contributions are as follows: we establish 1) a systematic approach to estimate supports via Kernel Density Estimator (KDE) derived under topological conditions; 2) a new metric that is robust to outliers while reliably detecting the change of distributions on various scenarios; 3) a consistency guarantee with robustness under very weak assumptions that are suitable for high dimensional data; 4) a combination of a noise framework and a statistical inference in TDA. Our code is available at TopP&R -NeurIPS 2023.

2 Background

To lay the foundation for our method and theoretical analysis, we first explain the previous evaluation method Precision and Recall ( P&R ). Then, we introduce the main idea of persistent homology and its confidence estimation techniques that bring the benefit of using topological and statistical tools for addressing uncertainty in samples. For the sake of space constraints and to streamline the discussion of our main idea, we only provide a brief overview of the concepts that are relevant to this work and refer the reader to Appendix A or [20, 21, 22, 23] for further details on TDA.

Notation. For any x and r > 0 , we use the notation B d ( x, r ) = { y : d ( y, x ) < r } be the open ball in distance d of radius r . We also write B ( x, r ) when d is understood from the context. For a distribution P on R d , we let supp( P ) := { x ∈ R d : P ( B ( x, r )) > 0 for all r > 0 } be the support of P . Throughout the paper, we refer to supp(P) as support of P , or simply support, or manifold, but we don't necessarily require the (geometrical) manifold structure on supp(P) . Note that, when the support is high dimensional, what we can recover at most through estimation is the partial support. For a kernel function K : R d → R , a dataset X = { X 1 , . . . , X n } ⊂ R d and bandwidth h > 0 , we let the kernel density estimator (KDE) as ˆ p h ( x ) := 1 nh d ∑ n i =1 K ( x -X i h ) , and we let the average KDE as p h := E [ˆ p h ] . We denote by P , Q the probability distributions in R d of real data and generated samples, respectively. And we use X = { X 1 , . . . , X n } ⊂ R d and Y = { Y 1 , . . . , Y m } ⊂ R d for real data and generated samples possibly with noise, respectively.

Precision and Recall. There exist two aspects of generative samples' quality; fidelity and diversity. Fidelity refers to the degree to which the generated samples resemble the real ones. Diversity, on the other hand, measures whether the generated samples cover the full variability of the real samples. Sajjadi et al. [12] was the first to propose assessing these two aspects separately via Precision and Recall ( P&R ). In the ideal case where we have full access to the probability distributions P and Q , precision P ( Q ) := Q (supp( P )) , recall Q ( P ) := P (supp( Q )) , which correspond to the max Precision and max Recall in [12], respectively.

Persistent homology and diagram. Persistent homology is a tool in computational topology that measures the topological invariants (homological features) of data that persist across multiple scales, and is represented in the persistence diagram. Formally, let filtration be a collection of subspaces F = {F δ ⊂ R d } δ ∈ R with δ 1 ≤ δ 2 implying F δ 1 ⊂ F δ 2 . Typically, filtration is defined through a function f related to data. Given a function f : R d → R , we consider its sublevel filtration { f -1 ( -∞ , δ ] } δ ∈ R or superlevel filtration { f -1 [ δ, ∞ ) } δ ∈ R . For a filtration F and for each nonnegative k , we track when k -dimensional homological features ( e.g. , 0 -dimension: connected component, 1 -dimension: loop, 2 -dimension: cavity, . . . ) appear and disappear. As δ increases or decreases in the filtration {F δ } , if a homological feature appears at F b and disappears at F d , we say that it is born at b and dies at d . By considering these pairs { ( b, d ) } as points in the plane ( R ∪ {±∞} ) 2 , we obtain a persistence diagram . From this, a homological feature with a longer life length, d -b , can be treated as a significant feature, and a homological feature with a shorter life length as a topological noise, which lies near the diagonal line { ( δ, δ ) : δ ∈ R } (Figure 1 (b)).

Confidence band estimation. Statistical inference has recently been developed for TDA [24, 25, 26]. TDA consists of features reflecting topological characteristics of data, and it is of question to systematically distinguish features that are indeed from geometrical structures and features that are insignificant or due to noise. To statistically separate topologically significant features from topological noise, we employ confidence band estimation. Given the significance level α , let confidence band c α be the bootstrap bandwidth of ∥ ˆ p h -p h ∥ ∞ , computed as in Algorithm 1 (see Appendix H.2). Then it satisfies lim inf n →∞ P ( ∥ ˆ p h -p h ∥ ∞ < c α ) ≥ 1 -α , as in Proposition D.2 (see Appendix D). This confidence band can simultaneously determine significant topological features while filtering out noise features. We use c X and c Y to denote the confidence band defined under significance level α according to the datasets X and Y . In later sections, we use these tools to provide a more rigorous way of scoring samples based on the confidence level we set.

3 Robust support estimation for reliable evaluation

Current evaluation metrics for generative models typically rely on strong regularity conditions. For example, they assume samples are well-curated without outliers or adversarial perturbation, real or generative models have bounded densities, etc. However, practical scenarios are wild: both real and generated samples can be corrupted with noise from various sources, and the real data can be very sparsely distributed without density. In this work, we consider more general and practical situations, wherein both real and generated samples can have noises coming from the sampling procedure, remained uncertainty due to data or model, etc. See Appendix G.2 for more on practical scenarios.

Overview of our metric We design our metric to evaluate the performance very conservatively. Our metric is based on topologically significant data structures with statistical confidence above a certain level. Toward this, we apply KDE as a function f to define a filtration, which allows

us to approximate the support with data through { f -1 [ δ, ∞ ) } δ ∈ R . Since the significance of data comprising the support is determined by the life length of homological features, we calculate c α that enables us to systematically separate short/long lifetimes of homological features. We then estimate the supports with topologically significant data structure via superlevel set f -1 [ c α , ∞ ) and finally, we evaluate fidelity and diversity with the estimated supports. We have collectively named this process TopP&R . By its nature, TopP&R is bounded and yields consistent performance under various conditions such as noisy data points, outliers, and even with long-tailed data distribution.

3.1 Topological precision and recall

To facilitate our discussion, we rewrite the precision in Section 2 as precision P ( Y ) = Q (supp( P ) ∩ supp( Q )) /Q (supp( Q )) and define the precision of data points as

which is just replacing the distribution Q with the empirical distribution 1 m ∑ m j =1 δ Y j of Y in the precision. Similarly,

In practice, supp( P ) and supp( Q ) are not known a priori and need to be estimated, and these estimates should be robust to noise since we allow it now. For this, we use the KDE ˆ p h n ( x ) := 1 nh d n ∑ n i =1 K ( x -X i h n ) of X and the bootstrap bandwidth c X of ∥ ˆ p h n -p h n ∥ ∞ , where h n > 0 and a significance level α ∈ (0 , 1) (Section 2). Then, we estimate the support of P by the superlevel set at c X 1 as ˆ supp( P ) = ˆ p -1 h n [ c X , ∞ ) , which allows to filter out noise whose KDE values are likely to be small. Similarly, the support of Q is estimated: ˆ supp( Q ) = ˆ q -1 h m [ c Y , ∞ ) , where ˆ q h m ( x ) := 1 mh d m ∑ m j =1 K ( x -Y j h m ) is the KDE of Y and c Y is the bootstrap bandwidth of ∥ ˆ q h m -q h m ∥ ∞

For the robust estimates of the precision, we apply the support estimates to precision P ( Y ) and recall Q ( X ) and define the topological precision and recall ( TopP&R ) as

The kernel bandwidths h n and h m are hyperparameters, and we provide guidelines to select the optimal bandwidths h n and h m in practice (See Appendix H.4).

3.2 Bandwidth estimation using bootstrapping

Using the bootstrap bandwidth c X as the threshold is the key part of our estimator ( TopP&R ) for robustly estimating supp( P ) . As we have seen in Section 2, the bootstrap bandwidth c X filters out the topological noise in topological data analysis. Analogously, using c X allows to robustly estimate supp( P ) . When X i is an outlier, its KDE value ˆ p h ( X i ) is likely to be small as well as the values at the connected component generated by X i . So those components from outliers are likely to be removed in the estimated support ˆ p -1 h [ c X , ∞ ) . Higher dimensional homological noises are also removed. Hence, the estimated support denoises topological noise and robustly estimates supp( P ) . See Appendix C for a more detailed explanation.

Now that we are only left with topological features of high confidence, this allows us to draw analogies to confidence intervals in statistical analysis, where the uncertainty of the samples is treated

by setting the level of confidence. In the next section, we show that TopP&R not only gives a more reliable evaluation score for generated samples but also has good theoretical properties.

3.3 Addressing the curse of dimensionality

As discussed in Section 3.2, getting the bootstrap bandwidth c X with a theoretical guarantee plays a key role in our metric, and the choice of KDE as a filtration function is inevitable, as in Remark 4.4. However, computing the support of a high-dimensional feature with KDE demands significant computation, and the accuracy is low due to low density values. This hinders an efficient and correct evaluation in practice. To address this issue, we apply a random projection into a low-dimensional space by leveraging the Johnson-Lindenstrauss Lemma (Lemma B.1). This lemma posits that using a random projection effectively preserves information regarding distances and homological features, composed of high-dimensional features, in a low-dimensional representation. Furthermore, we have shown that random projection does not substantially reduce the influence of noise, nor does it affect the performance of TopP&R under various conditions with complex data (Section 5, I.7, and I.8).

4 Consistency with robustness of TopP&R

The key property of TopP&R is consistency with robustness. The consistency ensures that, the precision and the recall we compute from the data approaches the precision and the recall from the distribution as we have more samples. The consistency allows to investigate the precision and recall of full distributions only with access to finite sampled data. TopP&R achieves consistency with robustness, that is, the consistency holds with the data possibly corrupted by noise. This is due to the robust estimation of supports with KDE with confidence bands.

We demonstrate the statistical model for both data and noise. Let P , Q , X , Y be as in Section 2, and let X 0 , Y 0 be real data and generated data without noise. X , Y , X 0 , Y 0 are understood as multisets, i.e. , elements can be repeated. We first assume that the uncorrupted data are IID.

Assumption 1. The data X 0 = { X 0 1 , . . . , X 0 n } and Y 0 = { Y 0 1 , . . . , Y 0 m } are IID from P and Q , respectively.

In practice, the data is often corrupted with noise. We consider the adversarial noise, where some fraction of data are replaced with arbitrary point cloud data.

Assumption 2. Let { ρ k } k ∈ N be a sequence of nonnegative real numbers. Then the observed data X and Y satisfies ∣ ∣ X\X 0 ∣ ∣ = nρ n and ∣ ∣ Y\Y 0 ∣ ∣ = mρ m .

In the adversarial model, we control the level of noise by the fraction ρ , but do not assume other conditions such as IID or boundedness, to make our noise model very general and challenging.

For distributions and kernel functions, we assume weak conditions, detailed in Assumption A1 and A2 in Appendix D. Under the data and the noise models, TopP&R achieves consistency with robustness. That is, the estimated precision and recall are asymptotically correct with high probability even if up to a portion of 1 / √ n or 1 / √ m are replaced by adversarial noise. This is due to the robust estimation of the support with the kernel density estimator with the confidence band of the persistent homology.

Proposition 4.1. Suppose Assumption 1, 2, A1, A2 hold. Suppose α → 0 , h n → 0 , nh d n → ∞ , nh -d n ρ 2 n → 0 , and similar relations hold for h m , ρ m . Then,

for fixed sequences of sets { A n,m } n,m ∈ N , { B n,m } n,m ∈ N with P ( A n,m ) → 0 and Q ( B n,m ) → 0 as n, m →∞ .

Theorem 4.2. Under the same condition as in Proposition 4.1,

Since P ( A n,m ) → 0 and Q ( B n,m ) → 0 , these imply consistencies of TopP&R . In fact, additionally under minor probabilistic and geometrical assumptions, P ( A n,m ) and Q ( B n,m ) are of order h m + h n .

Lemma 4.3. Under the same condition as in Proposition 4.1 and additionally under Assumption A3, A4, P ( A n,m ) = O ( h n + h m ) and Q ( B n,m ) = O ( h n + h m ) .

Remark 4.4 . Consistency guarantees from Proposition 4.1 and Theorem 4.2 are in principle due to the uniform convergence of KDE over varying bandwidth h n (Proposition D.2). Once we replace estimating the support with KDE by k-NN or something else, we wouldn't have consistency guarantees. Hence, using the KDE is an essential part for the theoretical guarantees of TopP&R .

Our theoretical results in Proposition 4.1 and Theorem 4.2 are novel and important in several perspectives. These results are among the first theoretical guarantees for evaluation metrics for generative models as far as we are aware of. Also, as in Remark D.1, assumptions are very weak and suitable for high dimensional data. Also, robustness to adversarial noise is provably guaranteed.

5 Experiments

A good evaluation metric should not only possess desirable theoretical properties but also effectively capture the changes in the underlying data distribution. To examine the performance of evaluation metrics, we carefully select a set of experiments for sanity checks. With toy and real image data, we check 1) how well the metric captures the true trend of underlying data distributions and 2) how well the metric resists perturbations applied to samples.

Wecompare TopP&R with several representative evaluation metrics that include Improved Precision and Recall ( P&R ) [13], Density and Coverage ( D&C ) [14], Geometric Component Analysis (GCA) [27], and Manifold Topology Divergence (MTD) [28] (Appendix F). Both GCA and MTD are the recent evaluation metrics that utilize topological features to some extent; GCA defines precision and recall based on connected components of P and Q , and MTD measures the distance between two distributions using the sum of lifetimes of homological features. For all the experiments, linear random projection to 32 dimensions is additionally used for TopP&R , and the shaded area of all figures denotes the ± 1 standard deviation for ten trials. For a fair comparison with existing metrics, we have utilized 10k real and fake samples for all experiments. For more details, please refer to Appendix H.1.

5.1 Sanity checks with toy data

Following [14], we first examine how well the metric reflects the trend of Y moving away from X and whether it is suitable for finding mode-drop phenomena. In addition to these, we newly design several experiments that can highlight TopP&R 's favorable theoretical properties of consistency with robustness in various scenarios.

5.1.1 Shifting the generated feature manifold

We generate samples from X ∼ N ( 0 , I ) and Y ∼ N ( µ 1 , I ) in R 64 , where 1 is a vector of ones and I is an identity matrix. We examine how each metric responds to shifting Y with µ ∈ [ -1 , 1] while there are outliers at 3 ∈ R 64 for both X and Y (Figure 2). We discovered that GCA struggles to detect changes using its default hyperparameter configuration, and this issue persists even after performing an exhaustive hyperparameter sweep. Since empirical tuning of the hyperparameters is required for each dataset, utilizing GCA in practical applications proves to be challenging (Appendix F). Both improved P&R and D&C behave pathologically since these methods estimate the support via the k-nearest neighbor algorithm, which inevitably overestimate the underlying support when there are outliers. For example, when µ < 0 . 5 , Recall returns a high-diversity score, even though the true supports of X and Y are actually far apart. In addition, P&R does not reach 1 in high dimensions even when X = Y . D&C [14] yields better results than P&R because it consistently uses X (the real data distribution) as a reference point, which typically has fewer outliers than Y (the fake data distribution). However, there is no guarantee that this will always be the case in practice [17, 18]. If an outlier is present in X , D&C also returns an incorrect high-fidelity score at µ > 0 . 5 . On the other hand, TopP&R shows a stable trend unaffected by the outlier, demonstrating its robustness.

5.1.2 Dropping modes

We simulate mode-drop phenomena by gradually dropping all but one mode from the fake distribution Y that is initially identical to X (Figure 3). Here, we consider the mixture of Gaussians with seven

Fidelity

1.0

0.0

0.5

0.2

0.7

1.0

0.0

0.5

0.2

0.7

Diversity

-1.0

0.0

1.0

-0.5

0.5

-1.0

0.0

1.0

-0.5

0.5

𝜇:

Fidelity

1.0

0.0

0.5

0.2

0.7

-1.0

0.0

1.0

-0.5

0.5

Center of the fake distribution

𝜇

Center of the fake distribution

𝜇

TopP

Precision (k = 3)

Density (k = 5)

TopR

Recall (k = 3)

Coverage (k = 5)

TopP

Precision (k = 3)

Density (k = 5)

GCA precision

TopR

Recall (k = 3)

Coverage (k = 5)

GCA recall

-1

Diversity

Fidelity

Fidelity

Diversity

1.0

1.4

1.2

0.8

0.6

0.5

1.0

0.7

0.3

0.0

0.5

1.0

0.7

0.3

0.0

1.0

1.4

1.2

0.8

0.6

0.0

1.0

0.2

0.4

0.6

0.8

0.0

1.0

0.2

0.4

0.6

0.8

0.0

1.0

0.2

0.4

0.6

0.8

0.0

1.0

0.2

0.4

0.6

0.8

Concentration on the first mode

Concentration on the first mode

Concentration on the first mode

Concentration on the first mode

(a) Sequential mode dropping

(b) Simultaneous mode dropping

Precision (k = 3)

TopP

Density (k = 5)

GCA precision

Precision (k = 3)

TopP

Density (k = 5)

GCA precision

Recall (k = 3)

TopR

Coverage (k = 5)

GCA recall

Recall (k = 3)

TopR

Coverage (k = 5)

GCA recall

Real distribution

Realdistribution

Fake distribution

Fakedistribution

+

+

(a) Non-IID scatter noise

(b) Non-IID swap noise

Ratio of outlier in the data (%)

Fidelity

Diversity

Fidelity

Diversity

0.8

0.0

0.4

0.6

0.2

1.00

0.00

0.50

0.75

0.25

0.6

0.4

0.2

0.0

0.8

Ratio of outlier in the data (%)

Ratio of outlier in the data (%)

Ratio of outlier in the data (%)

0.04

0.08

0.12

0.15

0.00

0.04

0.08

0.12

0.15

0.00

0.04

0.08

0.12

0.15

0.00

0.04

0.08

0.12

0.15

0.00

1.00

0.00

0.50

0.25

0.75

GCA precision

TopP

Precision (k = 3)

Density (k = 5)

GCA recall

TopR

Recall (k = 3)

Coverage (k = 5)

二二

modes in R 64 . We keep the number of samples in X constant so that the same amount of decreased samples are supplemented to the first mode which leads fidelity to be fixed to 1. We observe that the values of Precision fail to saturate, i.e. , mainly smaller than 1, and the Density fluctuates to a value greater than 1, showing their instability and unboundedness. Recall and GCA do not respond to the simultaneous mode drop, and Coverage decays slowly compared to the reference line. In contrast, TopP performs well, being held at the upper bound of 1 in sequential mode drop, and TopR also decreases closest to the reference line in simultaneous mode drop.

5.1.3 Tolerance to Non-IID perturbations

Robustness to perturbations is another important aspect we should consider when designing a metric. Here, we test whether TopP&R behaves stably under two variants of noise cases (see Section G.3); 1) scatter noise : replacing X i and Y j with uniformly distributed noise and 2) swap noise : swapping the position between X i and Y j . These two cases all correspond to the adversarial noise model of Assumption 2. We set X ∼ N ( µ = 0 , I ) ∈ R 64 and Y ∼ N ( µ = 1 , I ) ∈ R 64 where µ = 1 , and thus an ideal evaluation metric must return zero for both fidelity and diversity. In the result, while the GCA precision is relatively robust to the scatter noise, GCA recall tends to be sensitive to the swap noise. In both cases, we find that P&R and D&C are more sensitive while TopP&R remains relatively stable until the noise ratio reaches 15% of the total data, which is a clear example of the weakness of existing metrics to perturbation (Figure 4).

(a) Non-IID Scatter noise

Relative ratio to the

no outlier case

2.40

2.10

1.80

1.50

1.25

1.00

Ratio of outlier to inliers (%)

TopR

Recall (k = 3)

Coverage (k = 5)

(b) Non-IID Swap noise

1.70

1.56

1.42

1.28

1.14

1.00

Ratio of outlier to inliers (%)

TopR

Recall (k = 3)

Coverage (k = 5)

1.02

1.00

0.98

0.96

0.94

Ratio of outlier to inliers (%)

TopP

Precision (k = 3)

Density (k = 5)

1.10

1.05

1.00

0.95

0.90

0.85

Ratio of outlier to inliers (%)

TopP

Precision (k = 3)

Density (k = 5)

Fidelity

Diversity

Fidelity

Diversity

0.00

0.30

0.05

0.20

0.10

0.15

0.25

0.00

0.30

0.05

0.20

0.10

0.15

0.25

0.00

0.30

0.05

0.20

0.10

0.15

0.25

0.00

0.30

0.05

0.20

0.10

0.15

0.25

(a) Gaussian noise

F1-score

Noise intensity

0.0

1.0

0.5

0.0

1.0

2.0

3.0 0.0

1.0

2.0

3.0 0.0

1.0

2.0

3.0 0.0

1.0

2.0

3.0

Noise intensity

Noise intensity

Noise intensity

(c) Salt and pepper

(d) Black rectangle

TopPR

PR (k = 3)

DC (k = 5)

FID

MTD

TopPR

PR (k = 3)

DC (k = 5)

FID

MTD

TopPR

PR (k = 3)

DC (k = 5)

FID

MTD

TopPR

PR (k = 3)

DC (k = 5)

FID

MTD

5.2 Sanity check with Real data

Now that we have verified the metrics on toy data, we move on to real data. Just like in the toy experiments, we concentrate on how the metrics behave in extreme situations, such as outliers, mode-drop phenomena, perceptual distortions, etc. We also test different image embedders, including pretrained VGG16 [29], InceptionV3 [30], and SwAV [31].

5.2.1 Dropping modes in Baby ImageNet

We have conducted an additional experiment using Baby ImageNet [15] to investigate the sensitivity of TopP&R to mode-drop in real-world data. The performance of each metric (Figure A4) is measured with the identical data while simultaneously dropping the modes of nine classes of Baby ImageNet, in total of ten classes. Since our experiment involves gradually reducing the fixed number of fake samples until nine modes of the fake distribution vanish, the ground truth diversity should decrease linearly. From the experimental results, consistent to the toy result of Figure 3, both D&C and P&R still struggle to respond to simultaneous mode dropping. In contrast, TopP&R consistently exhibit a high level of sensitivity to subtle distribution changes. This notable capability of TopP&R can be attributed to its direct approximation of the underlying distribution, distinguishing it from other metrics. In addition, we perform the experiments on a dataset with long-tailed distribution and find that TopP&R captures the trend well even when there are minority sets (Appendix I.1). This again shows the reliability of TopP&R .

5.2.2 Robustness to perturbations

To test the robustness of our metric against the adversarial noise model of Assumption 2, we test both scatter-noise and swap noise scenarios with real data (see Section G.3). In the experiment, following Kynkäänniemi et al. [13], we first classify inliers and outliers that are generated by StyleGAN [1]. For scatter noise we add the outliers to the inliers and for swap noise we swap the real FFHQ images with generated images. Under these specific noise conditions, Precision shows similar or even better robustness than Density (Figure 5). On the other hand, Coverage is more robust than Recall . In both cases, TopP&R shows the best performance, resistant to noise.

5.2.3 Sensitiveness to the noise intensity

One of the advantages of FID [11] is that it is good at estimating the degrees of distortion applied to the images. Similarly, we check whether the F1-score based on TopP&R provides a reasonable evaluation according to different noise levels. As illustrated in Figure 6 and A5, X and Y are sets

of reference and noisy features, respectively. The experimental results show that TopP&R actually reflects well the different degrees of distortion added to the images while a similar topology-based method MTD shows inconsistent behavior to the distortions.

5.2.4 Ranking between generative models

The alignment between FID (or KID) and perceptual evaluation has been well-established in prior research, and these scores are widely used as a primary metric in the development of generative models due to its close correspondence with human perception. Consequently, generative models have evolved to align with FID's macroscopic perspective. Therefore, we believed that the order determined by FID at a high level captures to some extent the true performance hierarchy among models, even if it may not perfectly reflect it. In other words, if the development of generative models based on FID leads to genuine improvements in generative performance and if there is a meaningful correlation, similar rankings should be maintained even when the representation or embedding model changes. From this standpoint, while other metrics exhibit fluctuating rankings, TopP&R consistently provides the most stable and consistent results similar to both FID and KID. To quantitatively compare the similarity of rankings across varying embedders by different metrics, we have computed mean Hamming Distance (MHD) (Appendix H.6) where lower value indicates more similarity. TopP&R , P&R , D&C , and MTD have MHDs of 1.33, 2.66, 3.0, and 3.33, respectively.

6 Conclusions

Many works have been proposed recently to assess the fidelity and diversity of generative models. However, none of them has focused on the accurate estimation of support even though it is one of the key components in the entire evaluation pipeline. In this paper, we proposed topological precision and recall ( TopP&R ) that provides a systematical fix by robustly estimating the support with both topological and statistical treatments. To the best of our knowledge, TopP&R is the first evaluation metric that offers statistical consistency under noisy conditions, which may arise in real practice. Our theoretical and experimental results showed that TopP&R serves as a robust and reliable evaluation metric under various embeddings and noise conditions, including mode drop, outliers, and Non-IID perturbations. Last but not least, TopP&R provides the most consistent ranking among different generative models across different embeddings via calculating its F1-score.

References

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8110-8119, 2020.

Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems , 34, 2021.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. arXiv preprint arXiv:2202.00273 , 2022.

Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected gans converge faster. Advances in Neural Information Processing Systems , 34, 2021.

Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. Advances in Neural Information Processing Systems , 33:21357-21369, 2020.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in Neural Information Processing Systems , 31, 2018.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems , 32, 2019.

Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning , pages 7176-7185. PMLR, 2020.

Minguk Kang, Joonghyuk Shin, and Jaesik Park. Studiogan: A taxonomy and benchmark of gans for image synthesis. arXiv preprint arXiv:2206.09479 , 2022.

Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. arXiv preprint arXiv:2203.06026 , 2022.

Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. Advances in Neural Information Processing Systems , 33:17044-17056, 2020.

Zhengwen Li, Runmin Wu, and Tao Gan. Study on image data cleaning method of early esophageal cancer based on vgg_nin neural network. Scientific Reports , 12(1):1-10, 2022.

Gunnar Carlsson. Topology and data. Bull. Amer. Math. Soc. (N.S.) , 46(2):255-308, 2009. ISSN 0273-0979. doi: 10.1090/S0273-0979-09-01249-X. URL https://doi.org/10.1090/ S0273-0979-09-01249-X .

Herbert Edelsbrunner and John L. Harer. Computational topology . American Mathematical Society, Providence, RI, 2010. ISBN 978-0-8218-4925-5. doi: 10.1090/mbk/069. URL https://doi.org/10. 1090/mbk/069 . An introduction.

Frédéric Chazal and Bertrand Michel. An introduction to topological data analysis: Fundamental and practical aspects for data scientists. Frontiers Artif. Intell. , 4:667963, 2021. doi: 10.3389/frai.2021.667963. URL https://doi.org/10.3389/frai.2021.667963 .

Larry Wasserman. Topological data analysis. Annu. Rev. Stat. Appl. , 5:501-535, 2018. ISSN 2326-8298. doi: 10.1146/annurev-statistics-031017-100045. URL https://doi.org/10.1146/ annurev-statistics-031017-100045 .

Allen Hatcher. Algebraic topology . Cambridge University Press, Cambridge, 2002. ISBN 0-521-79160-X; 0-521-79540-0.

Frédéric Chazal, Brittany Fasy, Fabrizio Lecci, Alessandro Rinaldo, Aarti Singh, and Larry Wasserman. On the bootstrap for persistence diagrams and landscapes. Modelirovanie i Analiz Informacionnyh Sistem , 20, 11 2013. doi: 10.18255/1818-1015-2013-6-111-120.

Frédéric Chazal, Brittany Terese Fasy, Fabrizio Lecci, Alessandro Rinaldo, and Larry Wasserman. Stochastic convergence of persistence landscapes and silhouettes. J. Comput. Geom. , 6(2):140-161, 2015.

Brittany Terese Fasy, Fabrizio Lecci, Alessandro Rinaldo, Larry Wasserman, Sivaraman Balakrishnan, and Aarti Singh. Confidence sets for persistence diagrams. Ann. Statist. , 42(6):2301-2339, 2014. ISSN 0090-5364. doi: 10.1214/14-AOS1252. URL https://doi.org/10.1214/14-AOS1252 .

Petra Poklukar, Anastasiia Varava, and Danica Kragic. Geomca: Geometric evaluation of data representations. In International Conference on Machine Learning , pages 8588-8598. PMLR, 2021.

Serguei Barannikov, Ilya Trofimov, Grigorii Sotnikov, Ekaterina Trimbach, Alexander Korotin, Alexander Filippov, and Evgeny Burnaev. Manifold topology divergence: a framework for comparing data manifolds. Advances in Neural Information Processing Systems , 34, 2021.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Stanislav Morozov, Andrey Voynov, and Artem Babenko. On self-supervised image representations for gan evaluation. In International Conference on Learning Representations , 2020.

A.W. van der Vaart. Asymptotic Statistics . Asymptotic Statistics. Cambridge University Press, 2000. ISBN 9780521784504. URL https://books.google.fr/books?id=UEuQEM5RjWgC .

Michael R. Kosorok. Introduction to empirical processes and semiparametric inference . Springer Series in Statistics. Springer, New York, 2008. ISBN 978-0-387-74977-8. doi: 10.1007/978-0-387-74978-5. URL https://doi.org/10.1007/978-0-387-74978-5 .

William B. Johnson and Joram Lindenstrauss. Extensions of Lipschitz mappings into a Hilbert space. In Conference in modern analysis and probability (New Haven, Conn., 1982) , volume 26 of Contemp. Math. , pages 189-206. Amer. Math. Soc., Providence, RI, 1984. doi: 10.1090/conm/026/737400. URL https://doi.org/10.1090/conm/026/737400 .

Kasper Green Larsen and Jelani Nelson. The Johnson-Lindenstrauss lemma is optimal for linear dimensionality reduction. In 43rd International Colloquium on Automata, Languages, and Programming , volume 55 of LIPIcs. Leibniz Int. Proc. Inform. , pages Art. No. 82, 11. Schloss Dagstuhl. Leibniz-Zent. Inform., Wadern, 2016.

Kasper Green Larsen and Jelani Nelson. Optimality of the Johnson-Lindenstrauss lemma. In 58th Annual IEEE Symposium on Foundations of Computer Science-FOCS 2017 , pages 633-638. IEEE Computer Soc., Los Alamitos, CA, 2017.

Jisu Kim, Jaehyeok Shin, Alessandro Rinaldo, and Larry Wasserman. Uniform convergence rate of the kernel density estimator adaptive to intrinsic volume dimension. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 3398-3407. PMLR, 09-15 Jun 2019. URL https: //proceedings.mlr.press/v97/kim19e.html .

Michael H. Neumann. Strong approximation of density estimators from weakly dependent observations by density estimators from independent observations. Ann. Statist. , 26(5):2014-2048, 1998. ISSN 0090-5364. doi: 10.1214/aos/1024691367. URL https://doi.org/10.1214/aos/1024691367 .

Herbert Federer. Curvature measures. Transactions of the American Mathematical Society , 93:418-491, 1959. ISSN 0002-9947. doi: 10.2307/1993504. URL https://doi.org/10.2307/1993504 .

Christoph Thäle. 50 years sets with positive reach-a survey. Surv. Math. Appl. , 3:123-165, 2008. ISSN 1843-7265. doi: 10.1007/s11590-008-0097-2. URL https://doi.org/10.1007/ s11590-008-0097-2 .

Erik Carlsson, Gunnar Carlsson, and Vin De Silva. An algebraic topological method for feature identification. International Journal of Computational Geometry & Applications , 16(04):291-314, 2006.

Frédéric Chazal, Brittany Fasy, Fabrizio Lecci, Bertrand Michel, Alessandro Rinaldo, Alessandro Rinaldo, and Larry Wasserman. Robust topological inference: Distance to a measure and kernel distance. The Journal of Machine Learning Research , 18(1):5845-5884, 2017.

Hubert Wagner, Chao Chen, and Erald Vuçini. Efficient computation of persistent homology for cubical data. In Topological methods in data analysis and visualization II , pages 91-106. Springer, 2012.

George R Terrell and David W Scott. Variable kernel density estimation. The Annals of Statistics , pages 1236-1265, 1992.

Richard W Hamming. Error detecting and error correcting codes. The Bell system technical journal , 29(2): 147-160, 1950.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data , pages 93-104, 2000.

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining , pages 413-422. IEEE, 2008.

Jiyeon Han, Hwanil Choi, Yunjey Choi, Junho Kim, Jung-Woo Ha, and Jaesik Choi. Rarity score: A new metric to evaluate the uncommonness of synthesized images. arXiv preprint arXiv:2206.08549 , 2022.

Appendix

A More Background on Topological Data Analysis

Topological data analysis (TDA) [19] is a recent and emerging field of data science that relies on topological tools to infer relevant features for possibly complex data. A key object in TDA is persistent homology, which quantifies salient topological features of data by observing them in multi-resolutions.

A.1 Persistent Homology

Persistent homnology. Persistent homology is a multiscale approach to represent the topological features. For a filtration F and for each k ∈ N 0 = N ∪ { 0 } , the associated k -th persistent homology PH k F is a collection of k -th dimensional homologies { H k F δ } δ ∈ R equipped with homomorphisms { ı a,b k : H k F a → H k F b } a ≤ b induced by the inclusion F a ⊂ F b .

Persistence diagram. For the k -th persistent homology PH k F , the set of filtration levels at which a specific homology appears is always an interval [ b, d ) ⊂ [ -∞ , ∞ ] . The corresponding k -th persistence diagram is a multiset of points ( R ∪ {∞} ) 2 , consisting of all pairs ( b, d ) where [ b, d ) is the interval of filtration values for which a specific homology appears in PH k F .

A.2 Statistical Inference of Persistent Homology

As discussed above, a homological feature with a long life-length is important information in topology while the homology with a short life-length can be treated as non-significant information or noise. The confidence band estimator provides the confidence set from the features that only include topologically and statistically significant (statistically considered as elements in the population set) under a certain level of confidence. And to build a confidence set, we first need to endow a metric on the space of persistence diagrams.

Bottleneck distance. The most fundamental metric to measure the distance between two persistence diagrams is the bottleneck distance .

Definition A.1. The bottleneck distance between two persistence diagrams D 1 and D 2 is defined by

where the set Γ consists of all the bijections γ : D 1 ∪ Diag → D 2 ∪ Diag , and Diag is the diagonal { ( x, x ) : x ∈ R } ⊂ R 2 with infinite multiplicity.

One way of constructing the confidence set uses the superlevel filtration of the kernel density estimator and the bootstrap confidence band. Let X = { X 1 , X 2 , ..., X n } as given points cloud, then the probability for the distribution of points can be estimated via KDE defined as following: ˆ p h ( x ) := 1 nh d ∑ n i =1 K ( x -X i h ) where h is the bandwidth and d as a dimension of the space. We compute ˆ p h and ˆ p ∗ h , which are the KDE of X and the KDE of bootstrapped samples X ∗ , respectively. Now, given the significance level α and h > 0 , let confidence band q X be bootstrap bandwidth of a Gaussian Empirical Process [32, 33], √ n || ˆ p h -ˆ p ∗ h || ∞ . Then it satisfies P ( √ n || ˆ p h -p h || ∞ < q X ) ≥ 1 -α , as in Proposition D.2 in Section D. Then B d B ( ˆ P h , c X ) , the ball of persistent homology centered at ˆ P h and radius c X = q X / √ n in the bottleneck distance d B , is a valid confidence set as lim inf n →∞ P ( P ∈ B d B ( ˆ P h , c X ) ) ≥ 1 -α . This confidence set has further interpretation that in the persistence diagram, homological features that are above twice the radius 2 c X from the diagonal are simultaneously statistically significant.

B Johnson-Lindenstrauss Lemma

Johnson-Lindenstrass Lemma is stated as follows:

Lemma B.1 (Johnson-Lindenstrauss Lemma) . [34, Lemma 1]

Let 0 < ϵ < 1 and X ⊂ R D be a set of points with size n . Then for d = O ( min { n, D, ϵ -2 log n }) , there exists a linear map f : R D → R d such that for all x, y ∈ X ,

Remark B.2 . This Johnson-Lindenstrass Lemma is optimal for linear maps, and nearly-optimal even if we allow non-linear maps: there exists a set of points X ⊂ R D with size n , such that d should satisfy: (a) d = Ω ( ϵ -2 log n ) for a linear map f : R D → R d satisfying (5) to exist [35, Theorem 3], and (b) d = Ω ( ϵ -2 log( ϵ 2 n ) ) for a map f : R D → R d satisfying (5) to exist [36, Theorem 1].

C Denoising topological features from outliers

Using the bootstrap bandwidth c X as the threshold is the key part of our estimators TopP&R for robustly estimating supp( P ) . When the level set ˆ p -1 h [ c X , ∞ ) is used, the homology of ˆ p -1 h [ c X , ∞ ) consists of homological features whose (birth) ≥ c X and ( death ) ≤ c X , which are the homological features in skyblue area in Figure A1. In this example, we consider three types of homological noise, though there can be many more corresponding to different homological dimensions.

There can be a 0 -dimensional homological noise of (birth) < c X and (death) < c X , which is the red point in the persistence diagram of Figure A1. This noise corresponds to the orange connected component on the left. As in the figure, this type of homological noise usually corresponds to outliers.

There can be a 0 -dimensional homological noise of (birth) > c X and (death) > c X , which is the green point in the persistence diagram of Figure A1. This noise corresponds to the connected component surrounded by the green line on the left. As in the figure, this type of homological noise lies within the estimated support, not like the other two.

There can be a 1 -dimensional homological noise of (birth) < c X and (death) < c X , which is the purple point in the persistence diagram of Figure A1. This noise corresponds to the purple loop on the left.

These homological noises satisfy either their (birth) < c X and (death) < c X or their (birth) > c X and (death) > c X simultaneously with high probability, so those homological noises are removed in the estimated support ˆ p -1 h [ c X , ∞ ) , which is the blue area in the left and the skyblue area in the right in Figure A1.

We would like to further emphasize that homological noises are not restricted to 0 -dimension lying outside the estimated support (red point in the persistence diagram of Figure A1). 0 -dimensional homological noise inside the estimated support (green point in the persistence diagram of Figure A1), 1 -dimensional homological noise can also arise, and the bootstrap bandwidth c X allows to simultaneously filter them.

D Assumptions on distributions and kernels

For distributions, we assume that the order of probability volume decay P ( B ( x, r )) is at least r d .

Assumption A1. For all x ∈ supp( P ) and y ∈ supp( Q ) ,

Remark D.1 . Assumption A1 is analogous to Assumption 2 of Kim et al. [37], but is weaker since the condition is pointwise on each x ∈ R d . And this condition is much weaker than assuming a density on R d : for example, a distribution supported on a low-dimensional manifold satisfies Assumption A1. This provides a framework suitable for high dimensional data, since many times high dimensional data lies on a low dimensional structure hence its density on R d cannot exist. See Kim et al. [37] for a more detailed discussion.

For kernel functions, we assume the following regularity conditions:

Assumption A2. Let K : R d → R be a nonnegative function with ∥ K ∥ 1 = 1 , ∥ K ∥ ∞ , ∥ K ∥ 2 < ∞ , and satisfy the following:

Data and KDE levels

Death

Birth

0

c

a

1

1.5

0

c

a

1

1.5

0-dim

0-dim

0-dim

1-dim

(i) K (0) > 0 .

(ii) K has a compact support.

(iii) K is Lipschitz continuous and of second order.

Assumption A2 allows to build a valid bootstrap confidence band for kernel density estimator (KDE). See Theorem 12 of [26] or Theorem 3.4 of [38]

Proposition D.2 (Theorem 3.4 of [38]) . Let X = { X 1 , . . . , X n } be IID from a distribution P . For h > 0 , let ˆ p h , ˆ p ∗ h be kernel density estimator for X and its bootstrap X ∗ , respectively, and for α ∈ (0 , 1) , let c X be the α bootstrap quantile from √ nh d ∥ ˆ p h -ˆ p ∗ h ∥ ∞ . For h n → 0 ,

Assumption A1, A2 ensures that, when the bandwidth h n → 0 , average KDEs are bounded away from 0 .

Lemma D.3. Let P be a distribution satisfying Assumption A1. Suppose K is a nonnegative function satisfying K (0) > 0 and continuous at 0 . Suppose { h n } n ∈ N is given with h n ≥ 0 and h n → 0 . Then for all x ∈ supp( P ) ,

Proof. Since K (0) > 0 and K is continuous at 0 , there is r 0 > 0 such that for all y ∈ B (0 , r 0 ) , K ( y ) ≥ 1 2 K (0) > 0 . And hence

Hence as h n → 0 ,

Before specifying the rate of convergence, we introduce the concept of reach. First introduced by [39], the reach is a quantity expressing the degree of geometric regularity of a set. Given a closed subset A ⊂ R d , the medial axis of A , denoted by Med( A ) , is the subset of R d consisting of all the points that have at least two nearest neighbors on A .

̸

where d ( x, A ) = inf q ∈ A || q -x || denotes the distance from a generic point x ∈ R d to A , The reach of A is then defined as the minimal distance from A to Med( A ) .

Definition D.4. The reach of a closed subset A ⊂ R d is defined as

Now, for specifying the rate of convergence, we first assume that distributions have densities away from 0 and ∞ .

Assumption A3. P and Q have Lebesgue densities p and q that, there exists 0 < p min ≤ p max < ∞ , 0 < q min ≤ q max < ∞ with for all x ∈ supp( P ) and y ∈ supp( Q ) ,

We also assume weak geometric assumptions on the support of the distributions supp( P ) and supp( Q ) , being bounded and having positive reach.

Assumption A4. We assume supp( P ) and supp( Q ) are bounded. And the support of P and Q have positive reach, i.e. reach(supp( P )) > 0 and reach(supp( Q )) > 0 .

Sets with positive reach ensure that the volume of its tubular neighborhood grows in polynomial order, which is Theorem 26 from [40] and originally from [39].

Proposition D.5. Let A be a set with reach( A ) > 0 . Let A r := { x ∈ R d : d ( x, A ) ≤ r } . Then for r < reach( A ) , there exists a 0 , . . . , a d ∈ R ∪ {∞} that satisfies

where λ d is the usual Lebesgue measure of R d .

E Details and Proofs for Section 4

For a random variable X and α ∈ (0 , 1) , let q X,α be upper α -quantile of X , i.e., P ( X ≥ q α ) = α , or equivalently, P ( X < q α ) = 1 -α . Let ˜ p h be the KDE on X 0 . For a finite set X , we use the notation c X ,α for α -bootstrap quantile satisfying P (∥ ∥ ˆ p X ,h n -ˆ p X b ,h n ∥ ∥ ∞ < c X ,α |X ) = 1 -α , where X b is the bootstrap sample from X . For a distribution P , we use the notation c P,α for α -quantile satisfying P ( ∥ ˆ p h n -p h n ∥ ∞ < c P,α ) = 1 -α , where ˆ p h n is kernel density estimator of IID samples from P . Hence when X is not IID samples from P , the relation of Proposition D.2 may not hold.

Claim E.1 . Let Z ∼ N (0 , 1) be a sample from standard normal distribution, then for α ∈ (0 , 1) ,

Proof. Let Z ∼ N (0 , 1) , then for all x > 2 ,

Then x = C √ log(1 /α ) gives

And for 0 ≤ δ < α < 1 ,

And hence and in particular,

Now for 0 ≤ δ < α < 1 ,

And hence

and

Then from δ < α ,

Claim E.2 . Let X,Y be random variables, and 0 ≤ δ < α < 1 . Suppose there exists c > 0 satisfying P ( | X -Y | > c ) ≤ δ. (6)

Then,

And hence

Then changing the role of X and Y gives

Lemma E.3. (i) Under Assumption 1, 2 and A2,

(ii) Under Assumption 1, 2 and A2, with probability 1 -δ ,

Proof. (i)

First, note that

̸

̸

Then from Assumption 2, ∑ n i =1 I ( X i = X 0 i ) ≤ nρ n , and hence

(ii)

Let X b , X 0 b be bootstrapped samples of X , X 0 with the same sampling with replacement process. Let ˆ p b h , ˜ p b h be KDE of X b and X 0 b , respectively. And, note that

Let L b be the number of elements where X b and X 0 b differ, i.e., L b = ∣ ∣ X b \X 0 b ∣ ∣ = ∣ ∣ X 0 b \X b ∣ ∣ , then L b ∼ Binomial( n, ρ n ) , and

And hence,

Then by Hoeffding's inequality, with probability 1 -δ ,

by applying this to (7), with probability 1 -δ ,

Hence applying Claim E.2 'implies that, with probability 1 -δ ,

Let ˜ δ n := O ( ( log n nh d n ) 4+ d 4+2 d ) be from RHS of Proposition D.2, then for large enough n , δ n ≥ ˜ δ n . Note that Proposition D.2 implies that for all α ∈ (0 , 1) ,

Now, nh d n →∞ implies that √ nh d n (˜ p h n -p h n ) converges to a Gaussian process, and then Claim E.2 implies that c P,α = Θ (√ log(1 /α ) nh d n ) and

Then under Assumption 2, since nh -d n ρ 2 n δ -2 n = o (1) and nρ n ≥ 1 implies h -d n ρ n δ -2 n = o (1) , and then

Then (8), (9), (10) implies that

and

Now, from the definition of c P,α +3 δ n , with probability 1 -α -3 δ n ,

And (ii) implies that, with probability 1 -δ n ,

Hence by combining (11), (12), (13), (14), with probability 1 -α -4 δ n ,

Corollary E.4. Suppose Assumption 1, 2, A2 hold, and let α ∈ (0 , 1) . Suppose δ -1 n = O ( ( log n nh d n ) 4+ d 4+2 d ) and nh -d n ρ 2 n δ -2 n → 0 . Then with probability 1 -α -4 δ n ,

Proof. (i)

Lemma E.3 (iii) implies that with probability 1 -α -4 δ n , ∥ ˆ p h -p h ∥ < c X ,α ≤ c P,α -3 δ n . This implies

(ii)

This can be proven similarly to (i).

Claim E.5 . For a nonnegative measure µ and sets A,B,C,D,

Proof.

From here, let P n and Q m be the empirical measures on X and Y , respectively, i.e., P n = 1 n ∑ n i =1 δ X i and Q m = 1 m ∑ m j =1 δ Y j .

Lemma E.6. Suppose Assumption 1, 2 hold. Let A ⊂ R d . Then with probability 1 -δ ,

Proof. Let P 0 n be the empirical measure on X 0 , i.e., P 0 n = 1 n ∑ n i =1 δ X 0 i . By using Hoeffding's inequality,

and hence with probability 1 -δ , and in particular,

And ∣ ∣ P n -P 0 n ∣ ∣ ( A ) is expanded as

̸

Under Assumption 2 , ∑ n i =1 I ( X i = X 0 i ) ≤ nρ n , and hence

̸

Therefore, with probability 1 -δ ,

Claim E.7 . Suppose Assumption 1, 2, A1, A2 hold, and let δ n , δ m ∈ (0 , 1) . Suppose nh d n → ∞ , mh d m → ∞ , δ -1 n = O ( ( log n nh d n ) 4+ d 4+2 d ) , δ -1 m = O ( ( log m mh d m ) 4+ d 4+2 d ) , nh -d n ρ 2 n δ -2 n → 0 , and nh -d m ρ 2 m δ -2 m → 0 . Let

(i) With probability 1 -2 α -4 δ n -8 δ m ,

(ii) With probability 1 -2 α -8 δ m ,

(iii) With probability 1 -2 α -4 δ n -9 δ m ,

(iv) With probability 1 -2 α -9 δ m ,

(v) As n, m →∞ , B n,m →∅ . And in particular,

Proof. (i)

From Corollary E.4 (i) and (ii), with probability 1 -2 α -4( δ n + δ m ) ,

Then from the first inequality of (16), combining with Claim E.5 gives

And from the second inequality of (16), combining with Claim E.5 gives

And hence combining (17) and (18) gives that, with probability 1 -2 α -4( δ n + δ m ) ,

Now we further bound the upper bound of (19). From Lemma E.6, with probability 1 -δ m ,

And similarly, with probability 1 -δ m ,

And similarly, with probability 1 -δ m ,

And similarly, with probability 1 -δ m ,

Hence by applying (20), (21), (22), (23) to (19), with probability 1 -2 α -4 δ n -8 δ m ,

where B n,m is from (15).

(ii)

This can be done similarly to (i).

(iii)

Lemma E.6 gives that with probability 1 -δ m ,

Hence combining this with (i) gives the desired result.

(iv)

Lemma E.6 gives that with probability 1 -δ m ,

Hence combining this with (ii) gives the desired result.

(v)

Note that Lemma D.3 implies that for all x ∈ supp( P ) , lim inf n →∞ p h n ( x ) > 0 , so p h n ( x ) > 2 c P for large enough n . And hence as n →∞ ,

And similar argument holds for supp( Q ) \ q -1 h m [2 c Q , ∞ ) , so as m →∞ ,

Also, since K has compact support, for any x / ∈ supp( P ) , x / ∈ supp( P h n ) once h n < d ( x, supp( P )) . Hence as n →∞ ,

Hence by applying (24), (25), (26), (27) to the definition of B n,m in (15) gives that as n, m →∞ ,

Below we state a more formal version of Proposition 4.1.

Proposition E.8. Suppose Assumption 1,2,A1,A2 hold. Suppose α → 0 , δ n → 0 , δ -1 n = O ( ( log n nh d n ) 4+ d 4+2 d ) , h n → 0 , nh d n → ∞ , and nh -d n ρ 2 n δ -2 n → 0 , and similar relations hold for h m , ρ m . Then there exists some constant C > 0 not depending on anything else such that

where

And as n, m →∞ , P ( A n,m ) → 0 and Q ( B n,m ) → 0 hold.

Proof of Proposition E.8. Now this is an application of Claim E.7 (i) (ii) (v) to the definitions of precision P ( Y ) and recall Q ( X ) in (1) and (2) and the definitions of TopP X ( Y ) and TopR Y ( X ) in (3) and (4).

Similarly, we state a more formal version of Theorem 4.2.

Theorem E.9. Suppose Assumption 1,2,A1,A2 hold. Suppose α → 0 , δ n → 0 , δ -1 n = O ( ( log n nh d n ) 4+ d 4+2 d ) , h n → 0 , nh d n → ∞ , and nh -d n ρ 2 n δ -2 n → 0 , and similar relations hold for h m , ρ m . Then there exists some constant C > 0 not depending on anything else such that

where A n,m and B n,m are the same as in Proposition E.8. Again as n, m →∞ , P ( A n,m ) → 0 and Q ( B n,m ) → 0 hold.

And similarly, as m →∞ ,

And in particular,

Proof of Theorem E.9. Now this is an application of Claim E.7 (iii) (iv) (v) to the definitions of precision P ( Q ) and recall Q ( P ) and the definitions of TopP X ( Y ) and TopR Y ( X ) in (3) and (4).

Proof of Lemma 4.3. Recall that B n,m is defined in (15) as

and hence Q ( B n,m ) can be upper bounded as

For the first term of RHS of (28), suppose supp( K ) ⊂ B R d (0 , 1) for convenience, and let A -h n := { x ∈ supp( P ) : d ( x, R d \ supp( P )) ≥ h n } . Then from Assumption A3, for all x ∈ A -h n , p h n ( x ) ≥ p min holds. Hence for large enough N such that for n ≥ N , c P < p min , then p h n ( x ) ≥ p min for all x ∈ A -h n , and hence

Also, supp(P) being bounded implies that all the coefficients a 0 , . . . , a d -1 in Proposition D.5 for supp( P ) are in fact finite. Then q ≤ q max from Assumption A3 and Proposition D.5 implies

And hence combining (29) and (30) gives that

For the second term of RHS of (28), with a similar argument,

Finally, for the third term of RHS of (28), Proposition D.5 implies

Hence applying (31), (32), (33) to (28) gives that

F Related Work

Improved Precision & Recall ( P&R ). Existing metrics such as IS and FID assess the performance of generative models with a single score while showing usefulness in determining performance rankings between models and are still widely used. These evaluation metrics have problems that they cannot provide detailed interpretations of the evaluation in terms of fidelity and diversity. Sajjadi et al. [12] tried to solve this problem by introducing the original Precision and Recall, which however has a limitation as it is inaccurate due to the simultaneous approximation of real support and fake support through ' k -means clustering'. For example, if we evaluate the fake images having high fidelity and large value of k , many fake features can be classified in a small cluster with no real features, resulting in a low fidelity score ( precision P := Q ( supp ( P )) ). Kynkäänniemi et al. [13] focuses on the limitation from the support estimation and presents an P&R that allows us to more accurately assess fidelity and diversity by approximating real support and fake support separately:

Here, supp ( P ) is defined as the union of spheres centered on each real feature X i and whose radius is the distance between k th nearest real features of X i

Density & Coverage ( D&C ). Naeem et al. [14] has reported that P&R cannot stably provide accurate fidelity and diversity when real or fake features are present through experiment. This limitation of P&R is that it has a problem with approximating overestimated supports by outliers due to the use of " k -nearest neighborhood" algorithm. For a metric that is robust to outlying features, Naeem et al. [14] proposes a new evaluation metric that only relies on the real support, based on the fact that a generative model often generates artifacts which possibly results in outlying features in the embedding space:

In the equation, NND k ( X i ) means the k th-nearest neighborhood distance of X i and B ( X i , NND k ( X i )) denotes the hyper-sphere centered at X i with radius NND k ( X i ) . However, D&C is only a partial solution because it still gives an inaccurate evaluation when a real outlying feature exists. Unlike coverage, density is not upper-bounded which makes it unclear exactly which ideal score a generative model should achieve.

Geometric Evaluation of Data Representations (GCA) . Poklukar et al. [27] proposes a metric called GCM that assesses the fidelity and diversity of fake images. GCM uses the geometric and topological properties of connected components formed by vertex V (feature) and edge E (connection). Briefly, when the pairwise distance between vertices is less than a certain threshold ϵ , a connected component is formed by connecting two vertices with an edge. This set of connected components can be thought of as a graph G = ( V , E ) , and each connected component separated from each other in G is defined as a subgraph G i (i.e., G = ∪ i G i ). GCA uses the consistency c ( G i ) and the quality q ( G i ) to select the important subgraph G i that is formed on both real vertices and fake vertices ( V = X ∪ Y ). c ( G i ) evaluates the ratio of the number of real vertices and fake vertices and q ( G i ) computes the ratio of the number of edges connecting real vertices and fake vertices to the total number of edges in G i . Given the consistency threshold η c and quality threshold η q , the set of important subgraphs is defined as S ( η c , η q ) = ∪ q ( G i ) >η q , c ( G i ) >η c G i . Based on this, GCA precision and recall are defined as follows:

In above, S Y and G Y represent a subset ( V = Y , E ) of each graph S and G , respectively. One of the drawbacks is that the new hyper-parameters ϵ , η c , and η q need to be set arbitrarily according to the image dataset in order to evaluate fake images well. In addition, even if we use the hyper-parameters that seem most appropriate, it is difficult to verify that the actual evaluation results are correct because they do not approximate the underlying feature distribution.

Manifold Topology Divergence (MTD) . Barannikov et al. [28] proposes a new metric that uses the life-length of the k -dimensional homology of connected components between real features and fake features formed through Vietoris-Rips filtration [41]. MTD is simply defined as the sum of the lifelength of homologies and is characterized by an evaluation trend consistent with the FID. Specifically, MTD constructs the graph G by connecting edges between features with a distance smaller than the threshold ϵ . Then the birth and death of the k -dimensional homologies in G are recorded by adjusting the threshold ϵ from 0 to ∞ . The life-length l i ( h ) is obtained through death i -birth i of any k -dimensional homology h , and let life-length set of all homologies as L ( h ) . MTD repeats the process of obtaining L ( h ) on randomly sampled subsets X ′ ∈ X and Y ′ ∈ Y at each iteration, and defined as the following:

However, MTD has a limitation of only considering a fixed dimensional homology for evaluation. In addition, MTD lacks interpretability as it evaluates the model with a single assessment score.

G Philosophy of our metric & Practical Scenarios

G.1 Philosophy of our metric

All evaluation metrics have different resolutions and properties. The philosophy of our metric is to propose a reliable evaluation metric based on statistically and topologically certain things. In a real situation, the data often contain outliers or noise, and these data points come from a variety of sources (e.g. human error, feature embedding network). These errors may play as outliers, which leads to an overestimation of the data distribution, which in turn leads to a false impression of improvement when developing generative models. As discussed in Section 2, P&R and its variants have different ways to estimate the supports, which overlook the possible presence and effect of noise. Naeem et al. [14] revealed that previous support estimation approaches may inaccurately estimate the true support under noisy conditions, and partially solved this problem by proposing to only use the real support. However, this change goes beyond the natural definition of precision and recall, and results in losing some beneficial properties like boundedness. Moreover, this is still a temporary solution since real data can also contain outliers. We propose a solution to the existing problem by minimizing the effects of noise, using topologically significant data points and retaining the natural definition of precision and recall.

G.2 Practical Scenarios

From this perspective, we present two examples of realistic situations where outliers exist in the data and filtering out them can have a significant impact on proper model analysis and evaluation. With real data, there are many cases where outliers are introduced into the data due to human error [17, 18]. Taking the simplest MNIST as an example, suppose our task of interest is to generate 4. Since image number 7 is included in data set number 4 due to incorrect labeling (see Figure 1 of [17]), the support of the real data in the feature space can be overestimated by such outliers, leading to an unfair evaluation of generative models (as in Section 5.1.2 and 5.2.2); That is, the sample generated with weird noise may be in the overestimated support, and existing metrics without taking into account the reliability of the support could not penalize this, giving a good score to a poorly performing generator.

A similar but different example is when noise or distortions in the captured data (unfortunately) behave adversely on the feature embedding network used by the current evaluation metrics (as in Section 5.1.2 and 5.2.2); e.g., visually it is the number 7, but it is mismapped near the feature space where there are usually 4 and becomes an outlier. Then the same problem as above may occur. Note that in these simple cases, where the definition of outliers is obvious with enough data, one could easily examine the data and exclude outliers a priori to train a generative model. In the case of more complicated problems such as the medical field [18], however, it is often not clear how outliers are to be defined. Moreover, because data are often scarce, even outliers are very useful and valuable in practice for training models and extracting features, making it difficult to filter outliers in advance and decide not to use them.

On the other hand, we also provide an example where it is very important to filter out outliers in the generator sample and then evaluate them. To evaluate the generator, samples are generated by sampling from the preset latent space (typically Gaussian). Even after training is complete and the generators' outputs are generally fine, there's a latent area where generators aren't fully trained. Note that latent space sampling may contain samples from regions that the generator does not cover well during training ("unfortunate outlier"). When unfortunate outliers are included, the existing evaluation metrics may underestimate or overestimate the generator's performance than its general performance. (To get around this, it is necessary to try this evaluation several times to statistically stabilize it, but this requires a lot of computation and becomes impractical, especially when the latent space dimension is high.)

Especially considering the evaluation scenario in the middle of training, the above situation is likely to occur due to frequent evaluation, which can interrupt training or lead to wrong conclusions. On the other hand, we can expect that our metric will be more robust against the above problem since it pays more attention to the core (samples that form topologically meaningful structures) generation performance of the model.

G.3 Details of the noise framework in the experiments

We have assessed the robustness of TopP&R against two types of non-IID noise perturbation through toy data experiments (in Section 5.1.3) and real data experiments (in Section 5.2.2). The scatter noise we employed consists of randomly extracted noise from a uniform distribution. This noise, even when extensively added to our data, does not form a data structure with any significant signal (topological feature). In other words, from the perspective of TopP&R , the formation of topologically significant data structures implies that data samples should possess meaningful probability values in the feature space. However, noise following a uniform distribution across all feature dimensions holds very small probability values, and thus, it does not constitute a topologically significant data structure. Consequently, such noise is filtered out by the bootstrap bands c X or c Y that we approximate (see Section 2).

The alternate noise we utilized in our experiments, swap noise, possesses distinct characteristics from scatter noise. Initially, given the real and fake data that constitute important data structures, we introduce swap noise by randomly selecting real and fake samples and exchanging their positions. In this process, the swapped fake samples follow the distribution of real data, and conversely, the real samples adhere to the distribution of fake data. This implies the addition of noise that follows the actual data distribution, thereby generating significant data structures. When the number of samples undergoing such positional swaps remains small, meaningful probability values cannot be established within the distribution. Statistically, an extremely small number of samples cannot form a probability distribution itself. As a result, our metric operates robustly in the presence of noise. However, as the count of these samples increases and statistically significant data structures emerge, TopP&R estimates the precise support of such noise as part of their distribution. By examining Figure 4 and 5, as well as Table A12, it becomes evident that conventional metrics struggle with accurate support estimation due to vulnerability to noise. This limitation results in an inability to appropriately evaluate aspects involving minority sets or data forming long distributions.

G.4 Limitations

Since the KDE filtration requires extensive computations in high dimensions, a random projection that preserves high-dimensional distance and topological properties is inevitable. Based on the topological structure of the features present in the embedded low dimensional space, we have shown that TopP&R theoretically and experimentally has several good properties such as robustness to the noise from various sources and sensitivity to small changes in distribution. However, matching the distortion from the random projection to the noise level allowed by the confidence band is practically

1

infeasible: The bootstrap confidence band is of order ( nh d n ) -2 , and hence for the distortion from the random projection to match this confidence band, the embedded dimension d should be of order Ω ( nh d n log n ) from Remark (B.2). However, computing the KDE filtration in dimension Ω ( nh d n log n ) is practically infeasible. Hence in practice, the topological distortion from the random projection is not guaranteed to be filtered out by the confidence band, and there is a possibility of obtaining a less accurate evaluation score compared to calculations in the original dimensions.

Exploring the avenue of localizing uncertainty at individual data points separately presents an intriguing direction of research. Such an approach could potentially be more sample-efficient and disregard less data points. However, considering our emphasis on preserving topological signals, achieving localized uncertainty estimation in this manner is challenging within the current state of the art. For instance, localizing uncertainty in the kernel density estimate ˆ p is feasible, as functional variability is inherently local. To control uncertainty at a specific point x , we primarily need to analyze the function value ˆ p ( x ) at that point. In contrast, localizing uncertainty for homological features situated at point x requires the analysis of all points connected by the homological feature. Consequently, even if our intention is to confine uncertainty to a local point, it demands estimating the uncertainty at the global structural level. This makes localizing the uncertaintly for topological features difficult given the tools in topology and statistics we currently have.

H Experimental Details

H.1 Implementation details of embedding

We summarize the detailed information of our embedding networks implemented for the experiments. In Figure 2, 3, 4, A10, A4, and 5, P&R and D&C are computed from the features of ImageNet pretrained VGG16 (fc2 layer), and TopP&R is computed from features placed in R 32 with additional random linear projection. In the experiment in Figure A10, the SwAV embedder is additionally considered. We implement ImageNet pre-trained InceptionV3 (fc layer), VGG16 (fc2 layer), and SwAV as embedding networks with random linear projection to 32 dimensional feature space to compare the ranking of GANs in Table 1.

H.2 Implementation details of confidence band estimator

Algorithm 1 Confidence Band Estimator

H.3 Choice of confidence level

For the confidence level α , we would like to point out that α is not the usual hyperparameter to be tuned: It has a statistical interpretation of the probability or the level of confidence to allow error, noise, etc. The most popular choices are α = 0 . 1 , 0 . 05 , 0 . 01 , leading to 90%, 95%, 99% confidence. We used α = 0 . 1 throughout our experiments.

H.4 Estimation of Bandwidth parameter

As we discussed in Section 2, since TopP&R estimates the manifold through KDE with kernel bandwidth parameter h , we need to approximate it. The estimation techniques for h are as follows: ( a ) a method of selecting h that maximizes the survival time ( S ( h ) ) or the number of significant homological features ( N ( h ) ) based on information obtained about persistent homology using the filtration method, ( b ) a method using the median of the k-nearest neighboring distances between features obtained by the balloon estimator (for more details, please refer to [42], [43], and [44]). Note that, the bandwidths h for all the experiments in this paper are estimated via Balloon Bandwidth Estimator.

For (a) , following the notation in Section A, let the i th homological feature of persistent diagram be ( b i , d i ) , then we define its life length as l i ( h ) = d i -b i at kernel bandwidth h . With confidence band c α ( h ) , we select h that maximizes one of the following two quantities:

Note that, we denote the confidence band c α as c α ( h ) considering the kernel bandwidth parameter h of KDE in Algorithm 1.

For (b) , the balloon bandwidth estimator is defined as below:

#

h

: kernel bandwidth

# KND: kth nearest distance

# idx: index

# sort: sort in ascending order

Given

X

=

{

X

1

, X

2

, . . . , X

n

}

for

idx

= 1

,

2

, . . . , n

do

# Compute L2 distance between

X

idx

and

X

d

(

X

idx

,

X

) =

||

X

idx

-X||

2

2

,

i.e.,

{

d

(

X

idx

, X

1

)

, d

(

X

idx

, X

2

)

, . . .

}

# Calculate kth nearest neighbor distance

KND

idx

=

sort

(

d

(

X

idx

,

X

)

)[k]

end for

Given

KND

=

{

KND

1

,

KND

2

, . . . ,

KND

n

}

# Define the estimated bandwidth

ˆ

h

ˆ

h

=

median

(

KND

)

H.5 Computational complexity

TopP&R is computed using the confidence band c α and KDE bandwidth h . The computational cost of the confidence band calculation (Algorithm 1) and the balloon bandwidth calculation (Algorithm 2) are O ( k ∗ n 2 ∗ d ) and O ( n 2 ∗ d ) , respectively, where n represents the data size, d denotes the data dimension, and k stands for the number of repeats ( k = 10 in our implementation). The resulting computational cost of TopP&R is O ( k ∗ n 2 ∗ d ) . In our experiments based on real data, calculating TopP&R once using real and fake features, each consisting of 10k samples in a 4096-dimensional space, takes approximately 3-4 minutes. This computation speed is notably comparable to that of P&R and D&C , utilizing CPU-based computations. Note that, P&R and D&C that we primarily compare in our experiments are algorithms based on k -nearest neighborhood for estimating data distribution, and due to the computation of pairwise distances, the computational cost of these algorithm is approximately O ( n 2 ∗ d ) .

H.6 Mean hamming distance

̸

Hamming distance (HD) [45] counts the number of items with different ranks between A and B, then measures how much proportion differs in the overall order, i.e. for A i ∈ A and B i ∈ B , HD ( A,B ) := ∑ n i =1 1 { k : A i = B i } where k is the number of differently ordered elements. The mean HD is calculated as follows to measure the average distances of three ordered lists: Given three ordered lists A,B, and C , ¯ HD = ( HD ( A,B ) + HD ( A,C ) + HD ( B,C )) / 3 .

H.7 Explicit values of bandwidth parameter

Since our metric adaptively reacts to the given samples of P and Q , we have two h 's per experiment. For example, in the translation experiment (Figure 2), there are 13 steps in total, and each time we estimate h for P and Q , resulting in a total of 26 h 's. To show them all at a glance, we have listed all values in one place. We will also provide the code that can reproduce the results in our experiments upon acceptance.

I Additional Experiments

I.1 Survivability of minority sets in the long-tailed distribution

An important point to check in our proposed metric is the possibility that a small part of the total data (i.e., minority sets), but containing important information, can be ignored by the confidence band. We emphasize that since our metric takes topological features into account, even minority sets are not filtered conditioned that they have topologically significant structures. We assume that signals or data

that are minority sets have topological structures, but outliers exist far apart and lack a topological structure in general.

To test this, we experimented with CIFAR10 [46], which has 5,000 samples per class. We simulate a dataset with the majority set of six classes (2,000 samples per class, 12,000 total) and the minority set of four classes (500 samples per class, 2,000 total), and an ideal generator that exactly mimics the full data distribution. As shown in the Table A12, the samples in the minority set remained after the filtering process, meaning that the samples were sufficient to form a significant structure. Both D&C and TopP&R successfully evaluate the distribution for the ideal generator. To check whether our metric reacts to the change in the distribution even with this harsh setting, we also carried out the mode decay experiment. We dropped the samples of the minority set from 500 to 100 per class, which can be interpreted as an 11.3% decrease in diversity relative to the full distribution (Given (a) ratio of the number of samples between majority and minority sets = 12 , 000 : 2 , 000 = 6 : 1 and (b) 80% decrease in samples per minority class, the true decay in the diversity is calculated as 1 (1+6) × 0 . 8 = 11 . 3% with respect to the enitre samples). Here, recall and coverage react somewhat less sensitively with their reduced diversities as 3 p.p. and 2 p.p., respectively, while TopR reacted most similarly (9 p.p.) to the ideal value. In summary, TopP&R shows much more sensitiveness to the changes in data distribution like mode decay. Thus, once the minority set has survived the filtering process, our metric is likely to be much more responsive than existing methods.

I.2 Experiment on Non-IID perturbation with outlier removal methods

We compared the performance of metrics on a denoised dataset using Local Outlier Factor (LOF) [47] and Isolation Forest (IF) [48]. The experimental setup is identical to that of Figure 4 (see Section G.3 for details of our noise framework). In this experiment, we applied the outlier removal method before calculating P&R and D&C . From the result, P&R and D&C still do not provide a stable evaluation, while TopP&R shows the most consistent evaluation for the two types of noise without changing its trend.

Diversity

Ratio of outlier in the data (%)

(a) Non-IID scatter noise

Diversity

Fidelity

Fidelity

Ratio of outlier in the data (%)

Ratio of outlier in the data (%)

0.00

0.15

0.12

0.08

0.04

0.00

0.15

0.12

0.08

0.04

0.00

0.15

0.12

0.08

0.04

0.00

0.15

0.12

0.08

0.04

Ratio of outlier in the data (%)

(b) Non-IID swap noise

0.0

0.8

0.2

0.4

0.6

0.0

0.8

0.2

0.4

0.6

1.0

0.6

0.4

0.2

1.0

0.0

0.8

0.6

0.0

0.2

0.4

0.8

Clean precision (k = 3)

Clean recall (k = 3)

Clean precision (k = 3)

Clean recall (k = 3)

Clean density (k = 5)

Precision (k = 3)

Recall (k = 3)

Clean coverage (k = 5)

Precision (k = 3)

Clean density (k = 5)

Clean coverage (k = 5)

Recall (k = 3)

Density (k = 5)

Coverage (k = 5)

Density (k = 5)

Coverage (k = 5)

Clean TopP

Clean TopR

Clean TopP

Clean TopR

Diversity

Ratio of outlier in the data (%)

(a) Non-IID scatter noise

Diversity

Fidelity

Fidelity

Ratio of outlier in the data (%)

Ratio of outlier in the data (%)

0.00

0.15

0.12

0.08

0.04

Ratio of outlier in the data (%)

(b) Non-IID swap noise

0.00

0.15

0.12

0.08

0.04

0.00

0.15

0.12

0.08

0.04

0.00

0.15

0.12

0.08

0.04

0.0

0.2

0.4

0.6

0.8

1.0

0.8

0.6

0.4

0.2

0.0

1.0

0.8

0.6

0.4

0.2

0.0

0.2

0.0

0.4

0.6

Density (k = 5)

Clean TopP

Clean precision (k = 3)

Precision (k = 3)

Clean density (k = 5)

Clean recall (k = 3)

Clean coverage (k = 5)

Coverage (k = 5)

Clean TopR

Recall (k = 3)

Clean TopP

Clean precision (k = 3)

Precision (k = 3)

Clean density (k- 5)

Density (k = 5)

Clean TopR

Clean recall (k = 3)

Clean coverage (k = 5)

Coverage (k = 5)

Recall (k = 3)

Since TopP&R can discriminate topologically significant signals by estimating the confidence band (using KDE filtration function), there is no need to arbitrarily set cutoff thresholds for each dataset. On the other hand, in order to distinguish inliers from specific datasets using LOF and IF methods, a threshold specific to the dataset must be arbitrarily set each time, so there is a clear constraint that different results are expected each time for each parameter setting (which is set by the user). Through this, it is confirmed that our evaluation metric guarantees the most consistent scoring based on the topologically significant signals. Since the removal of outliers in the TopP&R is part of the support estimation process, it is not appropriate to replace our unified process with the LOF or IF. In detail, TopP&R removes outliers through a clear threshold called confidence band which is defined by KDE, and this process simultaneously defines KDE's super-level set as the estimated support. If this process is separated, the topological properties and interpretations of estimated support also disappear. Therefore, it is not practical to remove outliers by other methods when calculating TopP&R .

I.3 Sequential and simultaneous mode dropping with real dataset

Concentration on the first mode

(a) Sequential mode dropping

(b) Simultaneous mode dropping

1.0

0.2

0.6

0.8

0.4

1.0

0.2

0.6

0.8

0.4

Diversity

1.0

0.8

0.0

0.2

0.4

0.6

1.0

0.8

0.0

0.2

0.4

0.6

Concentration on the first mode

TopR

Recall (k = 3)

Coverage (k = 5)

Baby ImageNet

Visualization on Baby ImageNet dataset

TopR

Recall (k = 3)

Coverage (k = 5)

Diversity

I.4 Sensitiveness to noise intensity

The purpose of this experiment (Figure A5) is to closely observe the noise sensitivity of metrics in Figure 6. As the noise intensity is incrementally increased until all the metrics converge to 0, the most ideal outcome for the metrics is to exhibit a linear trend to capture these changes most effectively. From the results, it is able to observe that both TopP&R and P&R exhibit the most linear trend in their evaluation outcomes, while D&C demonstrates that least capability to reflect differences based on noise intensity.

(a) Gaussian noise

Noise intensity

1.0

0.0

0.5

0.3

0.7

(c) Salt and pepper

(d) Black rectangle

Noise intensity

Noise intensity

Noise intensity

0.0

1.0

0.2

0.4

0.6

0.8

0.0

1.0

0.2

0.4

0.6

0.8

0.0

1.0

0.2

0.4

0.6

0.8

0.0

1.0

0.2

0.4

0.6

0.8

F1-score

F1-score

F1-score

F1-score

P&R

P&R

P&R

P&R

D&C

D&C

D&C

D&C

I.5 Evaluating state-of-the-art generative models on the ImageNet

We conducted our evaluation using the ImageNet dataset, which covers a wide range of classes. We employed commonly accepted metrics in the community, including FID and KID, to rank the considered generative models effectively. The purpose of this experiment is not only to establish the applicability of our ranking metric for generative models, akin to existing single-score metrics, but also to emphasize its interpretability. In Table A13, we calculated the HD between P&R variants and the more reliable KID metric. This comparison aimed to assess the alignment of our metric with established single-score metrics. The results consistenly showed that TopP&R 's evaluation results are the most consistent with single-score metrics across various embedding networks. Conversely, P&R exhibited an inconsistent evaluation trend compared to KID, which struggled to effectively differentiate between ReACGAN and BigGAN. Note that, Kang et al. [15] previously demonstrated that BigGAN performs worse than ReACGAN. However, surprisingly, P&R consistently rated BigGAN as better than ReACGAN across all embedding networks. These findings imply that the P&R 's unreliable support estimation under noisy conditions might hinder its ability to accurately distinguish between generative models.

I.6 Verification of random projection effect in generative model ranking

We also applied the same random projection used by TopP&R to P&R and D&C in Section I.6. From the results of the Table A14, it is evident that the application of random projection to P&R leads to different rankings than its original evaluation tendencies. Similarly, the rankings of D&C with random projection still exhibit significant variations based on the embedding. Thus, we have demonstrated that random projection does not provide consistent evaluation tendencies across different embeddings, while also highlighting that this characteristic is unique to TopP&R . To quantify these differences among metrics, we computed the mean Hamming Distance (MHD) between the results in the embedding spaces of each metric. The calculated MHD scores were 1.33 for TopP&R , 2.67 for P&R , and 3.33 for D&C , respectively.

I.7 Verifying the effect of random projection to the noisy data

(a) All metrics with random projection to 32-dimensional space

Fidelity

Center of the fake distribution

𝜇

Diversity

Diversity

Fidelity

1.0

0.2

0.5

0.0

0.7

1.0

0.2

0.5

0.0

0.7

1.0

0.2

0.5

0.0

0.7

1.0

0.2

0.5

0.0

0.7

-1.0

-0.5

0.0

0.5

1.0

-1.0

-0.5

0.0

0.5

1.0

-1.0

-0.5

0.0

0.5

1.0

-1.0

-0.5

0.0

0.5

1.0

Center of the fake distribution

𝜇

Center of the fake distribution

𝜇

Center of the fake distribution

𝜇

(b) All metrics without random projection

TopP

Precision (k = 3)

Density (k = 5)

TopR

Recall (k = 3)

Coverage (k = 5)

TopP

Precision (k = 3)

Density (k = 5)

TopR

Recall (k = 3)

Coverage (k = 5)

Figure A6: Behaviors of evaluation metrics for outliers on real and fake distribution. For both real and fake data, the outliers are fixed at 3 ∈ R d where d is a dimension, and the parameter µ is shifted from -1 to 1. (a) depicts the comparison of all metrics using a random projection from 64 dimensions to 32 dimensions. (b) illustrates the difference between metrics without using random projection, where real and fake data are considered in 32 dimensions.

We experiment to ascertain whether utilizing a random projection address the drawbacks of existing metrics that are susceptible to noise, and we also conduct tests to validate whether TopP&R possesses noise robustness without using random projection. Through Figure A6 (a), it is evident that P&R and D&C still retain vulnerability to noisy features, and it is apparent that random projection itself does not diminish the impact of noise. Furthermore, Figure A6 (b) reveals that TopP&R , even without utilizing a random projection, exhibits robustness to noise through approximating the data support based on statistically and topologically significant features.

Building upon the insight from the toy data experiment in Figure A6 that random projection does not confer noise robustness to P&R and D&C , we aim to further demonstrate this fact through a real data experiment. In the experiment depicted in Figure A7, we follow the same setup as the previous experiment in Figure 5, applying the same random projection to P&R and D&C only. The results of

(a) Non-IID scatter noise

Fidelity

Ratio of outlier in the data (%)

1.05

0.90

1.00

0.95

0.0

0.04

0.08

0.12

1.5

Diversity

Diversity

Fidelity

TopP

Precision (k = 3)

Density (k = 5)

TopR

Recall (k = 3)

Coverage (k = 5)

TopR

Recall (k = 3)

Coverage (k = 5)

TopP

Precision (k = 3)

Density (k = 5)

0.0

0.04

0.08

0.12

1.5

0.0

0.04

0.08

0.12

1.5

0.0

0.04

0.08

0.12

1.5

Ratio of outlier in the data (%)

Ratio of outlier in the data (%)

Ratio of outlier in the data (%)

(b) Non-IID swap noise

1.6

1.4

1.0

1.2

1.6

1.4

1.0

1.2

1.05

0.90

1.00

0.95

this experiment affirm that random projection does not provide additional robustness against noise for P&R and D&C , while TopP&R continues to exhibit the most robust performance.

I.8 Robustness of TopP&R with respect to random projection dimension

Sequential mode drop

Concentration on the first mode

1.0

0.0

0.4

0.6

0.2

0.8

1.0

0.0

0.4

0.6

0.2

0.8

1.0

0.0

0.4

0.6

0.2

0.8

1.0

0.0

0.4

0.6

0.2

0.8

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Concentration on the first mode

Concentration on the first mode

Concentration on the first mode

Simultaneous mode drop

Simultaneous mode drop

Sequential mode drop

TopR

Recall (k = 3)

Coverage (k = 5)

TopR

Recall (k = 3)

Coverage (k = 5)

TopR

Recall (k = 3)

Coverage (k = 5)

TopR

Recall (k = 3)

Coverage (k = 5)

(a) Random projection to 64-dimensional space

(b) Random projection to 128-dimensional space

Figure A8: Comparison of evaluation trends of TopP&R with respect to random projection dimension in both sequential and simultaneous mode drop scenarios. Baby ImageNet dataset is used for the comparison, and TopP&R is computed using random projections to (a) 64 dimensions and (b) 128 dimensions.

To investigate whether the dimension of random projection influences the evaluation trend of TopP&R , we conduct mode dropping experiment using the Baby ImageNet dataset, similar to the experiment in Section I.3. Results in (a) and (b) of Figure A8 respectively compare the performance of TopP&R using random projection in dimensions of 64 and 128 with other metrics. From the experimental results, we observe that TopP&R is the most sensitive to the distributional changes, akin to the tendencies observed in the previous toy mode dropping experiment. Furthermore, upon examining the differences across random projection dimensions, TopP&R shows consistent evaluation trends regardless of dimensions.

I.9 Trucation trick

ψ is a parameter for the truncation trick and is first introduced in [4] and [1]. We followed the approach in [4] and [1]. GANs generate images using the noise input z , which follows the standard normal distribution N (0 , I ) or uniform distribution U ( -1 , 1) . Suppose GAN inadvertently samples noise outside of distribution, then it is less likely to sample the image from the high density area of the image distribution p ( z ) defined in the latent space of GAN, which leads to generate an image with artifacts. The truncation trick takes this into account and uses the following truncated distribution. Let f be the mapping from the input to the latent space. Let w = f ( z ) , and ¯ w = E [ f ( z )] , where z is either from N (0 , I ) or U ( -1 , 1) . Then we use w ′ = ¯ w + ψ ( w -¯ w ) as a truncated latent vector. If the value of ψ increases, then the degree of truncation decreases which makes images have greater diversity but possibly lower fidelity.

I.10 Toy experiment of trade-off between fidelity and diversity

We have designed a new toy experiment to mimic the truncation trick. With data distributions of 10k samples, X ∼ N ( µ = 0 , I ) ∈ R 32 and Y ∼ N ( µ = 0 . 6 , σ 2 ) ∈ R 32 , we measure the fidelity and diversity while incrementally increasing σ from 0.7 to 1.3. For smaller σ values, the evaluation metric should exhibit higher fidelity and lower diversity, while increasing σ should demonstrate decreasing

Standard deviation of

𝒴

(

𝜎

)

0.70

1.00

1.15

1.30

0.85

Standard deviation of

𝒴

(

𝜎

)

0.00

0.50

0.75

1.00

1.25

0.25

1.50

1.75

0.0

1.0

0.8

0.6

0.4

0.2

TopP

Precision (k = 3)

Density (k = 5)

TopR

Recall (k = 3)

Coverage (k = 5)

:

𝒳~𝒩 𝜇,𝜎 2 = 0

:

𝒴 ~ 𝒩(𝜇, 𝜎 2

= 0.8)

:

𝒴 ~ 𝒩(𝜇, 𝜎 2

= 1.1)

:

𝒴 ~ 𝒩(𝜇, 𝜎 2

= 1.3)

0.70

1.00

1.15

1.30

0.85

fidelity and higher diversity. From the results in Figure A9, both TopP&R and P&R exhibit a trade-off between fidelity and diversity.

I.11 Resolving fidelity and diversity

Decreased

ψ

Increased

ψ

Generated Images

VGG16

SwAV

Truncation Psi

ψ

Density

Coverage

Precision

Recall

GCA precision

GCA recall

TopP

TopR

0.0

1.0

0.2

0.4

0.6

0.8

Truncation Psi

ψ

0.0

1.0

0.2

0.4

0.6

0.8

5.0

0.0

1.0

2.0

3.0

4.0

0.0

1.0

0.5

1.5

2.0

To test whether TopP&R responds appropriately to the change in the underlying distributions in real scenarios, we test the metric on the generated images of StyleGAN2 [2] using the truncation trick [1]. StyleGAN2, trained on FFHQ as shown by Han et al. [49], tends to generate samples mainly belonging to the majority of the real data distribution, struggling to produce rare samples effectively. In other words, while StyleGAN2 excels in generating high-fidelity images, it lacks diversity. Therefore, the evaluation trend in this experiment should reflect a steady high fidelity score while gradually increasing diversity. Han et al. [49] demonstrates that the generated image quality of StyleGAN2 that lies within the support region exhibits sufficiently realistic visual quality. Therefore, particularly in the case of TopP&R , which evaluates fidelity excluding noise, the fidelity value should remain close to 1. As shown in Figure A10, every time the distribution is transformed by ψ , TopP&R responds well and shows consistent behavior across different embedders with bounded scores in [0 , 1] , which are important virtues as an evaluation metric. On the other hand, Density gives unbounded scores (fidelity > 1) and shows inconsistent trend depending on the embedder. Because Density is not capped in value, it is difficult to interpret the score and know exactly which value denotes the best performance ( e.g. , in our case, the best performance is when TopP&R = 1 ).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper observes that the existing metrics for evaluating fidelity and diversity of generative models can be unreliable when outliers or non-IID perturbations are present. Thus, the authors propose a new metric based on Kernel Density Estimation under topological conditions that is more robust in the earlier scenarios. The main contributions of this paper are the metric that is robust to outliers and theoretical guarantees for robustness.\n\nSTRENGTHS:\nThe paper assesses a very important topic of evaluating generative models and proposes a metric that provides robust estimates of sample fidelity and diversity in several toy/synthetic examples. Additionally, bringing concepts from topological data analysis (TDA) to evaluation of generative models might open new future directions in metric research.\n\nWEAKNESSES:\nThe paper is difficult to follow for readers that are not very familiar with TDA and it could be improved significantly by adding intuition and a visual introduction to the most important concepts TDA and precision and recall. Furthermore, the concept of persistent homology, persistence diagram and confidence band estimation are discussed on a general level in Sec. 2 but the discussion seems disconnected how these are related to the proposed metric – adding concrete and intuitive examples would make this section more approachable. Unfortunately, I believe that in the current state paper is not very accessible to a wider audience. \n\nThe novelty of the work is challenging to evaluate and potentially limited. In particular, the proposed metric seems to rely on the removal of outliers before evaluating the Top P&R, and the Top P&R part of the metric seems quite similar to previous work. Ideally, I would hope to see real world use cases with state-of-the-art models, where the metric provides more reliable evaluation (now Tab. 1 Top P&R yields very similar results to improved P&R).\n\nQUESTIONS:\n1. The authors claim that robustness is required in practical evaluations of models since these cases are “wild”, i.e., might corrupt/outlier samples or the distributions might be sparse without density. I am curious as to why the proposed metric seems to perform similarly as improved PR with real generative models in Tab. 1? Also, Tab. 1 does not tell which dataset these models were trained and evaluated on.\nSince the proposed metric is composed of an outlier removal and the Top P&R evaluation, how much of the performance in toy/synthetic datasets can be explained by the outlier removal part? Would adding a similar outlier removal to existing improved PR and DC improve their performance in the toy experiments?\n\n2. I think detecting outliers might prove to be difficult in practice, compared to the toy experiments presented in the paper. Real datasets do not ground-truth “outlier labels” and in some datasets, e.g. in medical imaging, the distribution has a very long tail as there might be more data from patients without a certain disease than with the disease. In this case, modeling the patients with the disease might be more interesting than modeling patients without the disease. However, when evaluating Top P&R the contributions of the patients with the diseases might be suppressed as they are deemed as outliers.\n\n3. How does the dimensionality of the random projection affect the performance of the metric?\n\n4. Is the bandwidth parameter $h$ different for each model in Tab. 1? If so, is it possible that some models obtain artificially better scores because their outlier removal is different than for the other models? Are Top P&R scores with different bandwidth parameters directly comparable?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel evaluation metric called Topological Precision and Recall (TopP&R) for generative models. Existing metrics often suffer from unreliable support estimation and yield inconsistent results. In contrast, TopP&R systematically estimates supports by retaining topologically and statistically significant features with confidence. It provides robust evaluation, accurately capturing changes in samples even in the presence of outliers and non-independent and identically distributed perturbations. The metric combines ideas from Topological Data Analysis and statistical inference, offering a reliable approach to support estimation. The paper includes theoretical and experimental evidence to demonstrate the effectiveness and robustness of TopP&R, making it a valuable contribution to the evaluation of generative models.\n\nSTRENGTHS:\nThe paper introduces a novel evaluation metric, Topological Precision and Recall (TopP&R), specifically designed for generative models. This metric addresses the limitations of existing evaluation metrics by focusing on robust support estimation, which is crucial for accurately assessing sample quality.\n\nTopP&R provides a robust evaluation that is resilient to outliers and non-independent and identically distributed perturbations. The metric systematically estimates supports by retaining topologically and statistically significant features, ensuring reliable evaluation results even in challenging scenarios.\n\nThe paper demonstrates that TopP&R offers statistical consistency in its results. By incorporating concepts from Topological Data Analysis and statistical inference, the metric provides a systematic approach to distinguishing signal from noise and effectively capturing relevant features in the data.\n\nThe paper provides both theoretical and experimental evidence to support the effectiveness of TopP&R. The theoretical analysis establishes the robustness and consistency guarantees of the metric, while the experimental results demonstrate its superior performance compared to existing evaluation metrics.\n\nTopP&R is a versatile evaluation metric that can be applied to various generative models. Its systematic approach to support estimation makes it suitable for evaluating the performance of models across different domains and datasets.\n\nThe authors provide code implementation for TopP&R, making it readily accessible for researchers and practitioners. This facilitates the adoption and reproducibility of the metric, contributing to the transparency and advancement of the field.\n\nOverall, the paper's strengths lie in its introduction of a novel evaluation metric, its robustness and reliability in assessing sample quality, and its theoretical and experimental validation, demonstrating its effectiveness and broad applicability in the evaluation of generative models.\n\nWEAKNESSES:\nThe paper mentions Algorithm 1 in line 107 but does not provide the algorithm or any further explanation in the paper. This lack of clarity leaves readers confused and makes it difficult to understand the specific steps and procedures involved.\n\nLine 139 mentions the bootstrap bandwidth, but the paper does not clearly explain how this bandwidth is computed. It is important to provide details on the methodology used for computing the bandwidth to ensure transparency and reproducibility.\n\nThe paper lacks a clear discussion on the dimensionality of the problem that the proposed method can handle. It does not explain how the authors choose the dimensionality of the space for random projections or how they ensure that these projections capture relevant underlying features. Additionally, the paper does not provide theoretical results supporting the choices made in Section 3.4 regarding dimensionality and random projections.\n\nIn the experimental section, the authors choose a projection dimension of 32 without providing a clear justification or explanation for this specific choice. It is important to provide reasoning or empirical evidence to support the selection of this dimension and explain its impact on the performance of the method.\n\nWhile the paper includes simulations to demonstrate the robustness and effectiveness of the proposed method, it lacks a thorough evaluation on real-world data where changes are not simulated. It would be valuable to see the performance of the method in real data scenarios to assess its practical applicability and validate its effectiveness in capturing genuine changes.\n\nAddressing these weaknesses would strengthen the paper by providing clarity, offering detailed explanations for methodological choices, and conducting comprehensive evaluations on real-world data.\n\nQUESTIONS:\nNo additional questions in addition to the weaknesses listed above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TopP&R, a comprehensive evaluation metric for generative models that enhances the accuracy of sample quality assessment in comparison to existing metrics. TopP&R incorporates topological data analysis and statistical inference, effectively estimating supports through the application of Kernel Density Estimator (KDE) under topological conditions. This metric demonstrates remarkable robustness towards outliers, successfully detects changes in distributions, and provides consistency guarantees and robustness for high-dimensional data with minimal assumptions. Notably, TopP&R is the pioneering metric that prioritizes robust support estimation, ensuring statistical consistency even when confronted with noise. The paper presents compelling theoretical findings and experimental evidence that strongly validate the effectiveness of TopP&R.\n\nSTRENGTHS:\nThe paper introduces a novel evaluation metric, Topological Precision and Recall (TopP&R), for generative models. It combines topological data analysis and statistical inference to estimate support robustly and detect distributional changes accurately.\n\nThe paper provides theoretical and experimental evidence to demonstrate the reliability and consistency of the TopP&R metric. It accurately assesses sample quality, even in the presence of outliers, non-independent and identically distributed perturbations, and other noise conditions. The paper also presents a systematic approach to estimate support using Kernel Density Estimator (KDE) under topological conditions.\n\nThe paper clearly explains the motivation, limitations of existing metrics, and the design and implementation of the TopP&R metric. It includes visualizations and examples to enhance clarity and uses clear language to facilitate understanding.\n\nThe paper addresses the need for more reliable evaluation metrics for generative models. The proposed TopP&R metric fills this gap by providing a robust evaluation of sample quality and ensuring statistical consistency. Its findings have implications for improving the evaluation and development of generative models.\n\nWEAKNESSES:\nLimited accessibility: The paper may be challenging for readers unfamiliar with persistent homology and topological data analysis to grasp the methodology and results. Providing additional background information and explaining concepts in more detail would improve clarity.\nLimited experimentation: The experiments are confined to a few datasets and models. Conducting experiments on a broader range of datasets and models would demonstrate the effectiveness of TopP&R in diverse scenarios.\n\nInsufficient discussion of limitations: Although the authors mention some limitations, such as the requirement for a large number of samples to estimate the confidence band, a more comprehensive discussion of the limitations would help readers understand the scope and applicability of TopP&R.\n\nRecommendations for improvement:\n\nCritique existing metrics: Highlight limitations of commonly used metrics, such as Inception Score and Fréchet Inception Distance, due to unreliable support estimates from sample features.\n\nAddress noise vulnerability: Explain how TopP&R overcomes vulnerability to noise by considering topologically and statistically significant features with confidence.\n\nPresent experimental validation: Provide comprehensive experimental results comparing TopP&R to existing metrics, demonstrating its superior accuracy and consistency.\n\nDiscuss real-world applicability: Explore practical implications and applications of TopP&R in various domains, showcasing its relevance beyond theory.\n\nShare open-source implementation: Make TopP&R's implementation publicly available, ideally on platforms like GitHub, to promote reproducibility, validation, and further advancements.\n\nQUESTIONS:\n1. How does TopP&R handle the trade-off between precision and recall in evaluating generative models?\n\n2. Can you provide more details on the noise framework and statistical inference used in TopP&R?\n\n3. Have you considered comparing TopP&R to other evaluation metrics based on topological features, such as Persistence Landscape?\n\n4. Can you provide more information on the implementation of TopP&R and its applicability to different types of generative models?\n\n5. How does the choice of threshold for retaining topologically and statistically significant features affect the performance of TopP&R?\n\n6. Have you explored other techniques, like Gaussian mixture models or non-parametric methods, for estimating the density function in TopP&R?\n\nLIMITATIONS:\nBased on my understanding, the limitations of the TopP&R method mentioned in this paper can be summarized as follows: \n\n1. The method requires a large number of samples to estimate the confidence band accurately, which can be a challenge for datasets with limited samples. \n\n2. The reliance on the choice of kernel function and bandwidth can affect the performance of TopP&R, and the need for a theoretical guarantee on the choice of bandwidth can be difficult to obtain in practice. \n\n3. The method assumes a smooth density function and independence between samples, which may not hold for all types of data, and the reliance on topological features may not capture all aspects of the data distribution.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
The paper discusses robust estimation of precision and recall in high dimensional and potentially noisy data. The approach relies on using kernel density estimation to approximate the underlying distributions and using a bootstrap estimation of confidence interval to ignore support with low probability mass. The authors show the effectiveness of this approach extensively using simulated and real data.

STRENGTHS:
The paper is very well written and discusses the motivation and approach clearly. The problem addressed in the paper is very relevant as robust estimation of performance metrics is crucial for better comparing different approaches. The paper presents several interesting simulated and real examples to elaborate the method.

WEAKNESSES:
Although the estimation of confidence interval to remove 'noisy' samples is interesting, it would be great to see a discussion on estimating this uncertainty at individual points separately than working with the worst uncertainty, i.e., the inf norm over the kernel density.

QUESTIONS:
- Figure 1b: In the persistence plot, what does the shaded area imply? Can more blue points be added from the example on the right showing where in the persistence plot they land? Should the birth in this case always be at zero?
- Line 128: while c_X and c_Y are referred to as bootstrap bandwidth, is there a link between them and h_n and h_m, i.e., bandwidth of the kernel density estimate?
- Line 261: what does "swapping the position between Xi and Yj" imply?
- Line 275: sensitiveness or sensitivity?

LIMITATIONS:
Limitation of the method has not been discussed in details. More discussion on the computational complexity will be helpful.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the need for more reliable evaluation metrics for deep generative models. The existence of noise in real data, such as mislabeled data and adversarial examples, affects the reliability of existing metrics and may lead to false impressions of improvement when developing generative models. To overcome these issues, the paper proposes a new evaluation metric based on the idea of Topological Data Analysis (TDA) and statistical inference. The proposed metric uses persistent homology from TDA and bootstrap bandwidth with confidence level to distinguish between significant patterns (signal) and noise in the data, providing a more robust and reliable measure of generative model performance.\n\nSTRENGTHS:\nGiven the recent substantial advancements in generative models, having a metric that is robust and reliable under various real-world noises holds significant importance. The authors present an approach that aims to overcome the limitations of previous metrics, using straightforward yet well-founded methods. Additionally, the paper introduces theoretical properties that show the metric's consistency in the presence of adversarial noises. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. If I understand correctly, the authors use random linear projection only for the proposed metric, which might be unfair since it is unclear whether the improvements in robustness to outliers or noises stem from the low-dimensional space or the proposed methods. A comparison with previous methods operating in the same low-dimensional space would be more comprehensive. \n\n2. The proposed metric's algorithm relies on multiple iterations for KDE filtration, raising questions about its practicality. It would be beneficial to include a comparison of the computational cost and overall runtime between the proposed metric and previous ones. Gaining insights into its efficiency compared to existing metrics will be valuable for both potential users and researchers. Additionally, it is essential to consider whether the metric can efficiently scale to handle a large number of datasets. However, the paper does not seem to provide details about the number of datasets used in the experiments, which leaves uncertainty about its scalability.\n\n3. In section 5.1.2, the authors assert that precision values smaller than 1 indicate a problem. However, this lacks sufficient justification especially with a small number of samples. Additionally, the behavior of TopP for simultaneous mode dropping in Figure 3 exhibits worse instability compared to other metrics. \n\n4. In the experimental results for mode dropping experiments with CIFAR-10, TopR does not exhibit consistent behavior with the toy experiments in the main paper (Figure 4) and sometimes performs worse than Recall and Coverage.\n\n5. In section 5.2.3, the authors argue that the proposed metric is more sensitive to different levels of diverse noise, similar to FID. However, the results in Figure 6 do not appear to provide compelling evidence for this claim, except for the Gaussian noise case. Notably, in the salt and pepper noise case, TopPR, PR, and DC exhibit equal behavior, and in the black rectangle scenario, TopPR performs worse than PR. To strengthen the authors' claim about the proposed metric's sensitivity to diverse noise levels, additional explanation and experiments are necessary.\n\n6. The sole reliance on FID as the ground truth ranking for models in section 5.2.4 might be insufficient and unconvincing. It is not clear how much relying on FID as GT is reliable. Including more widely used metrics, such as KID or human evaluations, would enhance the credibility of the result. However, even with the FID, the results do not seem to show the clear superiority of the proposed metric. Additionally, the results with HD across different embeddings also raise questions about the effect of the random projection layer, as other metrics do not utilize such a layer.\n\n7. Some statements in the paper are unfounded. For instance, the rationale behind the ideal metric returning zero (line 263) is not clear. Moreover, the assumption in line 278 (\"ground truth diversity should linearly decrease\") is not correct because the supp(Q) remains the same with simultaneous mode dropping. In that sense, TopR works the worst of all.\n\nQUESTIONS:\n1. This paper employs the median of pairwise k-nearest distances between samples to estimate fixed-width kernels for KDE. I am curious whether this method is as effective as using sample-adaptive varying kernels, especially for datasets with multi-modal or long-tailed distributions. Adaptive kernels that adjust to local data characteristics might offer advantages in capturing complex and diverse patterns.\n\n2. Concerning the experiments presented in Table 1, which dataset was used for these experiments? Can the proposed metric demonstrate effectiveness when applied to large-scale datasets, such as ImageNet?\n\n3. There are some recommended experiments that are missing in the paper but would strengthen the effectiveness of the proposed metric:\n- Evaluating metrics on Diffusion models such as ADM or EDM, which are the leading and prominent advancements in generative models, would be beneficial to demonstrate the practicality of the proposed metrics.\n- Experiment for being robust to outliers in real-world dataset as in D&C paper.\n- Ablation study regarding the hyperparameters alpha and h for real-world datasets.\n\n4. To enhance the readability of the paper, the captions of figures should be more detailed.\n\nLIMITATIONS:\nLimitations are discussed in the supplement. Potential negative social impact is unlikely for a paper comparing distributions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}