SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} Q _ {m} \left(\hat {p} _ {h _ {n}} ^ {- 1} \left[ c _ {\mathcal {X}}, \infty\right) \cap \hat {q} _ {h _ {m}} ^ {- 1} \left[ c _ {\mathcal {Y}}, \infty\right)\right) - Q _ {m} \left(\operatorname {s u p p} (P) \cap \operatorname {s u p p} (Q)\right) \\ \leq Q _ {m} \left(\operatorname {s u p p} \left(P _ {h _ {n}}\right) \cap \operatorname {s u p p} \left(Q _ {h _ {m}}\right)\right) - Q _ {m} \left(\operatorname {s u p p} (P) \cap \operatorname {s u p p} (Q)\right) \\ \leq Q _ {m} \left(\operatorname {s u p p} \left(P _ {h _ {n}}\right) \backslash \operatorname {s u p p} (P)\right) + Q _ {m} \left(\operatorname {s u p p} \left(Q _ {h _ {m}}\right) \backslash \operatorname {s u p p} (Q)\right). \tag {18} \\ \end{array}
$$
And hence combining (17) and (18) gives that, with probability  $1 - 2\alpha - 4(\delta_n + \delta_m)$ ,
$$
\begin{array}{l} \left| Q _ {m} \left(\hat {p} _ {h _ {n}} ^ {- 1} [ c _ {\mathcal {X}}, \infty) \cap \hat {q} _ {h _ {m}} ^ {- 1} [ c _ {\mathcal {Y}}, \infty)\right) - Q _ {m} \left(\operatorname {s u p p} (P) \cap \operatorname {s u p p} (Q)\right) \right| \\ \leq \max  \left\{Q _ {m} \left(\operatorname {s u p p} (P) \backslash p _ {h _ {n}} ^ {- 1} [ 2 c _ {P}, \infty)\right) + Q _ {m} \left(\operatorname {s u p p} (Q) \backslash q _ {h _ {m}} ^ {- 1} [ 2 c _ {Q}, \infty)\right) \right. \\ , Q _ {m} \left(\operatorname {s u p p} \left(P _ {h _ {n}}\right) \backslash \operatorname {s u p p} (P)\right) + Q _ {m} \left(\operatorname {s u p p} \left(Q _ {h _ {m}}\right) \backslash \operatorname {s u p p} (Q)\right) \}. \tag {19} \\ \end{array}
$$
Now we further bound the upper bound of (19). From Lemma E.6, with probability  $1 - \delta_{m}$ ,
$$
Q _ {m} \left(\operatorname {s u p p} (P) \backslash p _ {h _ {n}} ^ {- 1} [ 2 c _ {P}, \infty)\right) \leq Q \left(\operatorname {s u p p} (P) \backslash p _ {h _ {n}} ^ {- 1} [ 2 c _ {P}, \infty)\right) + \rho_ {m} + \sqrt {\frac {\log (2 / \delta)}{2 m}}. \tag {20}
$$
And similarly, with probability  $1 - \delta_{m}$
$$
\begin{array}{l} Q _ {m} \left(\operatorname {s u p p} (Q) \backslash q _ {h _ {m}} ^ {- 1} [ 2 c _ {Q}, \infty)\right) \leq Q \left(\operatorname {s u p p} (Q) \backslash q _ {h _ {m}} ^ {- 1} [ 2 c _ {Q}, \infty)\right) + \rho_ {m} + \sqrt {\frac {\log (2 / \delta)}{2 m}} \\ = Q \left(q _ {h _ {m}} ^ {- 1} (0, \infty)\right) + \rho_ {m} + \sqrt {\frac {\log (2 / \delta)}{2 m}}. \tag {21} \\ \end{array}
$$
$$
Q _ {m} \left(\operatorname {s u p p} \left(P _ {h _ {n}}\right) \backslash \operatorname {s u p p} (P)\right) \leq Q \left(\operatorname {s u p p} \left(P _ {h _ {n}}\right) \backslash \operatorname {s u p p} (P)\right) + \rho_ {m} + \sqrt {\frac {\log (2 / \delta)}{2 m}}. \tag {22}
$$
$$
\begin{array}{l} Q _ {m} \left(\operatorname {s u p p} \left(Q _ {h _ {m}}\right) \backslash \operatorname {s u p p} (Q)\right) \leq Q \left(\operatorname {s u p p} \left(Q _ {h _ {m}}\right) \backslash \operatorname {s u p p} (Q)\right) + \rho_ {m} + \sqrt {\frac {\log (2 / \delta)}{2 m}} \\ = \rho_ {m} + \sqrt {\frac {\log (2 / \delta)}{2 m}}. \tag {23} \\ \end{array}
$$
Hence by applying (20), (21), (22), (23) to (19), with probability  $1 - 2\alpha - 4\delta_{n} - 8\delta_{m}$ ,
$$
\begin{array}{l} \left| Q _ {m} \left(\hat {p} _ {h _ {n}} ^ {- 1} [ c _ {\mathcal {X}}, \infty) \cap \hat {q} _ {h _ {m}} ^ {- 1} [ c _ {\mathcal {Y}}, \infty)\right) - Q _ {m} \left(\operatorname {s u p p} (P) \cap \operatorname {s u p p} (Q)\right) \right| \\ \leq C \left(Q \left(\left(\operatorname {s u p p} (P) \backslash p _ {h _ {n}} ^ {- 1} [ 2 c _ {P}, \infty)\right) \cup q _ {h _ {m}} ^ {- 1} (0, 2 c _ {Q}) \cup \operatorname {s u p p} \left(P _ {h _ {m}}\right) \backslash \operatorname {s u p p} (P)\right) \right. \\ \left. + \rho_ {m} + \sqrt {\frac {\log (1 / \delta)}{m}}\right) \\ = C \left(Q (B _ {n, m}) + \rho_ {m} + \sqrt {\frac {\log (1 / \delta)}{m}}\right), \\ \end{array}
$$
where  $B_{n,m}$  is from (15).
This can be done similarly to (i).
Lemma E.6 gives that with probability  $1 - \delta_{m}$
$$
\left| Q _ {m} \left(\operatorname {s u p p} (P) \cap \operatorname {s u p p} (Q)\right) - Q \left(\operatorname {s u p p} (P)\right) \right| \leq \rho_ {m} + \sqrt {\frac {\log (2 / \delta)}{2 m}}.
$$
Hence combining this with (i) gives the desired result.
(iv)
$$
| Q _ {m} (\operatorname {s u p p} (Q)) - 1 | \leq \rho_ {m} + \sqrt {\frac {\log (2 / \delta)}{2 m}}.
$$
Hence combining this with (ii) gives the desired result.
Note that Lemma D.3 implies that for all  $x \in \operatorname{supp}(P)$ ,  $\lim \inf_{n \to \infty} p_{h_n}(x) > 0$ , so  $p_{h_n}(x) > 2c_P$  for large enough  $n$ . And hence as  $n \to \infty$ ,
$$
\operatorname {s u p p} (P) \backslash p _ {h _ {n}} ^ {- 1} [ 2 c _ {P}, \infty) \rightarrow \emptyset . \tag {24}
$$
And similar argument holds for  $\mathrm{supp}(Q)\backslash q_{h_m}^{-1}[2c_Q,\infty)$ , so as  $m\to \infty$
$$
\operatorname {s u p p} (Q) \backslash q _ {h _ {m}} ^ {- 1} [ 2 c _ {Q}, \infty) \rightarrow \emptyset . \tag {25}
$$
Also, since  $K$  has compact support, for any  $x \notin \operatorname{supp}(P)$ ,  $x \notin \operatorname{supp}(P_{h_n})$  once  $h_n < d(x, \operatorname{supp}(P))$ . Hence as  $n \to \infty$ ,
$$
\operatorname {s u p p} \left(P _ {h _ {n}}\right) \backslash \operatorname {s u p p} (P) \rightarrow \emptyset . \tag {26}
$$
And similarly, as  $m\to \infty$
$$
\operatorname {s u p p} \left(Q _ {h _ {m}}\right) \backslash \operatorname {s u p p} (Q) \rightarrow \varnothing . \tag {27}
$$
Hence by applying (24), (25), (26), (27) to the definition of  $B_{n,m}$  in (15) gives that as  $n, m \to \infty$ ,
$$
B _ {n, m} \to \emptyset .
$$
And in particular,
Below we state a more formal version of Proposition 4.1.
Proposition E.8. Suppose Assumption 1,2,A1,A2 hold. Suppose  $\alpha \to 0$ ,  $\delta_{n} \to 0$ ,  $\delta_{n}^{-1} = O\left(\left(\frac{\log n}{nh_{n}^{d}}\right)^{\frac{4 + d}{4 + 2d}}\right)$ ,  $h_{n} \to 0$ ,  $nh_{n}^{d} \to \infty$ , and  $nh_{n}^{-d}\rho_{n}^{2}\delta_{n}^{-2} \to 0$ , and similar relations hold for  $h_{m}$ ,  $\rho_{m}$ . Then there exists some constant  $C > 0$  not depending on anything else such that
$$
| \mathrm {T o p P} _ {\mathcal {X}} (\mathcal {Y}) - \operatorname {p r e c i s i o n} _ {P} (\mathcal {Y}) | \leq C \left(Q (B _ {n, m}) + \rho_ {m} + \sqrt {\frac {\log (1 / \delta)}{m}}\right),
$$
$$
\left| \operatorname {T o p R} _ {\mathcal {Y}} (\mathcal {X}) - \operatorname {r e c a l l} _ {Q} (\mathcal {X}) \right| \leq C \left(P (A _ {n, m}) + \rho_ {n} + \sqrt {\frac {\log (1 / \delta)}{n}}\right),
$$
$$
A _ {n, m} := \left(\operatorname {s u p p} (Q) \backslash q _ {h _ {m}} ^ {- 1} [ 2 c _ {Q}, \infty)\right) \cup p _ {h _ {n}} ^ {- 1} (0, 2 c _ {P}) \cup \operatorname {s u p p} \left(Q _ {h _ {m}}\right) \backslash \operatorname {s u p p} (Q),
$$
$$
B _ {n, m} := \left(\operatorname {s u p p} (P) \backslash p _ {h _ {n}} ^ {- 1} [ 2 c _ {P}, \infty)\right) \cup q _ {h _ {m}} ^ {- 1} (0, 2 c _ {Q}) \cup \operatorname {s u p p} (P _ {h _ {n}}) \backslash \operatorname {s u p p} (P).
$$
And as  $n,m\to \infty$ $P(A_{n,m})\rightarrow 0$  and  $Q(B_{n,m})\rightarrow 0$  hold.
Proof of Proposition E.8. Now this is an application of Claim E.7 (i) (ii) (v) to the definitions of precision  $P(\mathcal{Y})$  and recall  $_Q(\mathcal{X})$  in (1) and (2) and the definitions of  $\mathrm{TopP}_{\mathcal{X}}(\mathcal{Y})$  and  $\mathrm{TopR}_{\mathcal{Y}}(\mathcal{X})$  in (3) and (4).
Similarly, we state a more formal version of Theorem 4.2.
Theorem E.9. Suppose Assumption 1,2,A1,A2 hold. Suppose  $\alpha \to 0$ ,  $\delta_{n} \to 0$ ,  $\delta_{n}^{-1} = O\left(\left(\frac{\log n}{nh_{n}^{d}}\right)^{\frac{4 + d}{4 + 2d}}\right)$ ,  $h_{n} \to 0$ ,  $nh_{n}^{d} \to \infty$ , and  $nh_{n}^{-d}\rho_{n}^{2}\delta_{n}^{-2} \to 0$ , and similar relations hold for  $h_{m}$ ,  $\rho_{m}$ . Then there exists some constant  $C > 0$  not depending on anything else such that
$$
| \mathrm {T o p P} _ {\mathcal {X}} (\mathcal {Y}) - \operatorname {p r e c i s i o n} _ {P} (Q) | \leq C \left(Q (B _ {n, m}) + \rho_ {m} + \sqrt {\frac {\log (1 / \delta)}{m}}\right),
$$
$$
\left| \operatorname {T o p R} _ {\mathcal {Y}} (\mathcal {X}) - \operatorname {r e c a l l} _ {Q} (P) \right| \leq C \left(P \left(A _ {n, m}\right) + \rho_ {n} + \sqrt {\frac {\log (1 / \delta)}{n}}\right),
$$
where  $A_{n,m}$  and  $B_{n,m}$  are the same as in Proposition E.8. Again as  $n, m \to \infty$ ,  $P(A_{n,m}) \to 0$  and  $Q(B_{n,m}) \to 0$  hold.
Proof of Theorem E.9. Now this is an application of Claim E.7 (iii) (iv) (v) to the definitions of precision  $P(Q)$  and recall  $_Q(P)$  and the definitions of  $\mathrm{TopP}_{\mathcal{X}}(\mathcal{Y})$  and  $\mathrm{TopR}_{\mathcal{Y}}(\mathcal{X})$  in (3) and (4).
Proof of Lemma 4.3. Recall that  $B_{n,m}$  is defined in (15) as
$$
B _ {n, m} = \left(\operatorname {s u p p} (P) \backslash p _ {h _ {n}} ^ {- 1} [ 2 c _ {P}, \infty)\right) \cup q _ {h _ {m}} ^ {- 1} (0, 2 c _ {Q}) \cup \operatorname {s u p p} \left(P _ {h _ {n}}\right) \backslash \operatorname {s u p p} (P),
$$
and hence  $Q(B_{n,m})$  can be upper bounded as
$$
Q \left(B _ {n, m}\right) \leq Q \left(\operatorname {s u p p} (P) \backslash p _ {h _ {n}} ^ {- 1} [ 2 c _ {P}, \infty)\right) + Q \left(q _ {h _ {m}} ^ {- 1} (0, 2 c _ {Q})\right) + Q \left(\operatorname {s u p p} \left(P _ {h _ {n}}\right) \backslash \operatorname {s u p p} (P)\right). \tag {28}
$$
For the first term of RHS of (28), suppose  $\operatorname{supp}(K) \subset \mathcal{B}_{\mathbb{R}^d}(0,1)$  for convenience, and let  $A_{-h_n} \coloneqq \{x \in \operatorname{supp}(P) : d(x, \mathbb{R}^d \backslash \operatorname{supp}(P)) \geq h_n\}$ . Then from Assumption A3, for all  $x \in A_{-h_n}$ ,  $p_{h_n}(x) \geq p_{\min}$  holds. Hence for large enough  $N$  such that for  $n \geq N$ ,  $c_P < p_{\min}$ , then  $p_{h_n}(x) \geq p_{\min}$  for all  $x \in A_{-h_n}$ , and hence
$$
Q \left(\operatorname {s u p p} (P) \backslash p _ {h _ {n}} ^ {- 1} [ 2 c _ {P}, \infty)\right) \leq Q \left(\operatorname {s u p p} (P) \backslash A _ {- h _ {n}}\right). \tag {29}
$$
Also,  $\operatorname{supp}(\mathrm{P})$  being bounded implies that all the coefficients  $a_0,\ldots ,a_{d - 1}$  in Proposition D.5 for  $\operatorname {supp}(P)$  are in fact finite. Then  $q\le q_{\max}$  from Assumption A3 and Proposition D.5 implies
$$
Q \left(\operatorname {s u p p} (P) \backslash A _ {- h _ {n}}\right) = O \left(h _ {n}\right). \tag {30}
$$
And hence combining (29) and (30) gives that
$$
Q \left(\operatorname {s u p p} (P) \backslash p _ {h _ {n}} ^ {- 1} [ 2 c _ {P}, \infty)\right) = O \left(h _ {n}\right). \tag {31}
$$
For the second term of RHS of (28), with a similar argument,
$$
Q \left(q _ {h _ {m}} ^ {- 1} \left(0, 2 c _ {Q}\right)\right) = O \left(h _ {m}\right). \tag {32}
$$
Finally, for the third term of RHS of (28), Proposition D.5 implies
$$
Q \left(\operatorname {s u p p} \left(P _ {h _ {n}}\right) \backslash \operatorname {s u p p} (P)\right) = O \left(h _ {n}\right). \tag {33}
$$
Hence applying (31), (32), (33) to (28) gives that
$$
Q (B _ {n, m}) = O (h _ {n} + h _ {m}).
$$
Improved Precision & Recall (P&R). Existing metrics such as IS and FID assess the performance of generative models with a single score while showing usefulness in determining performance rankings between models and are still widely used. These evaluation metrics have problems that they cannot provide detailed interpretations of the evaluation in terms of fidelity and diversity. Sajjadi et al. [12] tried to solve this problem by introducing the original Precision and Recall, which however has a limitation as it is inaccurate due to the simultaneous approximation of real support and fake support through "  $k$ -means clustering". For example, if we evaluate the fake images having high fidelity and large value of  $k$ , many fake features can be classified in a small cluster with no real features, resulting in a low fidelity score ( $precision_P := Q(supp(P))$ ). Kynkänniemi et al. [13] focuses on the limitation from the support estimation and presents an P&R that allows us to more accurately assess fidelity and diversity by approximating real support and fake support separately:
$$
p r e c i s i o n (\mathcal {X}, \mathcal {Y}) := \frac {1}{M} \sum_ {i} ^ {M} f (Y _ {i}, \mathcal {X}), r e c a l l (\mathcal {X}, \mathcal {Y}) := \frac {1}{N} \sum_ {j} ^ {N} f (X _ {j}, \mathcal {Y}),
$$
$$
\text {w h e r e} f (Y _ {i}, \mathcal {X}) = \left\{ \begin{array}{l l} 1, & \text {i f} Y _ {i} \in s u p p (P), \\ 0, & \text {o t h e r w i s e}. \end{array} \right.
$$
Here,  $supp(P)$  is defined as the union of spheres centered on each real feature  $X_{i}$  and whose radius is the distance between  $k$ th nearest real features of  $X_{i}$
Density & Coverage (D&C). Naeem et al. [14] has reported that P&R cannot stably provide accurate fidelity and diversity when real or fake features are present through experiment. This limitation of P&R is that it has a problem with approximating overestimated supports by outliers due to the use of "k-nearest neighborhood" algorithm. For a metric that is robust to outlying features, Naeem et al. [14] proposes a new evaluation metric that only relies on the real support, based on the fact that a generative model often generates artifacts which possibly results in outlying features in the embedding space:
$$
\operatorname {d e n s i t y} (\mathcal {X}, \mathcal {Y}) := \frac {1}{M} \sum_ {j} ^ {M} \sum_ {i} ^ {N} 1 _ {Y _ {j} \in f (X _ {i})}, \operatorname {c o v e r a g e} (\mathcal {X}, \mathcal {Y}) := \frac {1}{N} \sum_ {i} ^ {N} 1 _ {\exists j. s. t. Y _ {j} \in f (X _ {i})},
$$
$$
w h e r e f (X _ {i}) = \mathcal {B} (X _ {i}, N N D _ {k} (X _ {i}))
$$
In the equation,  $NND_{k}(X_{i})$  means the  $k$ th-nearest neighborhood distance of  $X_{i}$  and  $\mathcal{B}(X_i,NND_k(X_i))$  denotes the hyper-sphere centered at  $X_{i}$  with radius  $NND_{k}(X_{i})$ . However, D&C is only a partial solution because it still gives an inaccurate evaluation when a real outlying feature exists. Unlike coverage, density is not upper-bounded which makes it unclear exactly which ideal score a generative model should achieve.
Geometric Evaluation of Data Representations (GCA). Poklukar et al. [27] proposes a metric called GCM that assesses the fidelity and diversity of fake images. GCM uses the geometric and topological properties of connected components formed by vertex  $\mathcal{V}$  (feature) and edge  $\mathcal{E}$  (connection). Briefly, when the pairwise distance between vertices is less than a certain threshold  $\epsilon$ , a connected component is formed by connecting two vertices with an edge. This set of connected components can be thought of as a graph  $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ , and each connected component separated from each other in  $\mathcal{G}$  is defined as a subgraph  $\mathcal{G}_i$  (i.e.,  $\mathcal{G} = \cup_i \mathcal{G}_i$ ). GCA uses the consistency  $c(\mathcal{G}_i)$  and the quality  $q(\mathcal{G}_i)$  to select the important subgraph  $\mathcal{G}_i$  that is formed on both real vertices and fake vertices ( $\mathcal{V} = \mathcal{X} \cup \mathcal{Y}$ ).  $c(\mathcal{G}_i)$  evaluates the ratio of the number of real vertices and fake vertices and  $q(\mathcal{G}_i)$  computes the ratio of the number of edges connecting real vertices and fake vertices to the total number of edges in  $\mathcal{G}_i$ . Given the consistency threshold  $\eta_c$  and quality threshold  $\eta_q$ , the set of important subgraphs is defined as  $S(\eta_c, \eta_q) = \cup_{q(\mathcal{G}_i) > \eta_q, c(\mathcal{G}_i) > \eta_c} \mathcal{G}_i$ . Based on this, GCA precision and recall are defined as follows:
$$
\text {G C A p r e c i s i o n} = \frac {| \mathcal {S} ^ {\mathcal {Y}} | _ {\mathcal {V}}}{| \mathcal {G} ^ {\mathcal {Y}} | _ {\mathcal {V}}}, \text {G C A r e c a l l} = \frac {| \mathcal {S} ^ {\mathcal {X}} | _ {\mathcal {V}}}{| \mathcal {G} ^ {\mathcal {X}} | _ {\mathcal {V}}},
$$
In above,  $S^{\mathcal{V}}$  and  $\mathcal{G}^{\mathcal{V}}$  represent a subset  $(\mathcal{V} = \mathcal{V},\mathcal{E})$  of each graph  $S$  and  $\mathcal{G}$ , respectively. One of the drawbacks is that the new hyper-parameters  $\epsilon$ ,  $\eta_c$ , and  $\eta_q$  need to be set arbitrarily according to the image dataset in order to evaluate fake images well. In addition, even if we use the hyper-parameters that seem most appropriate, it is difficult to verify that the actual evaluation results are correct because they do not approximate the underlying feature distribution.
Manifold Topology Divergence (MTD). Barannikov et al. [28] proposes a new metric that uses the life-length of the  $k$ -dimensional homology of connected components between real features and fake features formed through Vietoris-Rips filtration [41]. MTD is simply defined as the sum of the life-length of homologies and is characterized by an evaluation trend consistent with the FID. Specifically, MTD constructs the graph  $\mathcal{G}$  by connecting edges between features with a distance smaller than the threshold  $\epsilon$ . Then the birth and death of the  $k$ -dimensional homologies in  $\mathcal{G}$  are recorded by adjusting the threshold  $\epsilon$  from 0 to  $\infty$ . The life-length  $l_{i}(h)$  is obtained through  $death_{i} - birth_{i}$  of any  $k$ -dimensional homology  $h$ , and let life-length set of all homologies as  $L(h)$ . MTD repeats the process of obtaining  $L(h)$  on randomly sampled subsets  $\mathcal{X}' \in \mathcal{X}$  and  $\mathcal{Y}' \in \mathcal{Y}$  at each iteration, and defined as the following:
$$
M T D (\mathcal {X}, \mathcal {Y}) = \frac {1}{n} \sum_ {i} ^ {n} L _ {i} (h), \text {w h e r e} L _ {i} (h) \text {i s t h e l i f e - l e n g t h s e t d e f i n e d a t i t h i t e r a t i o n}.
$$
However, MTD has a limitation of only considering a fixed dimensional homology for evaluation. In addition, MTD lacks interpretability as it evaluates the model with a single assessment score.
G Philosophy of our metric & Practical Scenarios
G.1 Philosophy of our metric
All evaluation metrics have different resolutions and properties. The philosophy of our metric is to propose a reliable evaluation metric based on statistically and topologically certain things. In a real situation, the data often contain outliers or noise, and these data points come from a variety of sources (e.g. human error, feature embedding network). These errors may play as outliers, which leads to an overestimation of the data distribution, which in turn leads to a false impression of improvement when developing generative models. As discussed in Section 2, P&R and its variants have different ways to estimate the supports, which overlook the possible presence and effect of noise. Naeem et al. [14] revealed that previous support estimation approaches may inaccurately estimate the true support under noisy conditions, and partially solved this problem by proposing to only use the real support. However, this change goes beyond the natural definition of precision and recall, and results in losing some beneficial properties like boundedness. Moreover, this is still a temporary solution since real data can also contain outliers. We propose a solution to the existing problem by minimizing the effects of noise, using topologically significant data points and retaining the natural definition of precision and recall.
G.2 Practical Scenarios
From this perspective, we present two examples of realistic situations where outliers exist in the data and filtering out them can have a significant impact on proper model analysis and evaluation. With real data, there are many cases where outliers are introduced into the data due to human error [17, 18]. Taking the simplest MNIST as an example, suppose our task of interest is to generate 4. Since image number 7 is included in data set number 4 due to incorrect labeling (see Figure 1 of [17]), the support of the real data in the feature space can be overestimated by such outliers, leading to an unfair evaluation of generative models (as in Section 5.1.2 and 5.2.2); That is, the sample generated with weird noise may be in the overestimated support, and existing metrics without taking into account the reliability of the support could not penalize this, giving a good score to a poorly performing generator.
A similar but different example is when noise or distortions in the captured data (unfortunately) behave adversely on the feature embedding network used by the current evaluation metrics (as in Section 5.1.2 and 5.2.2); e.g., visually it is the number 7, but it is mismapped near the feature space where there are usually 4 and becomes an outlier. Then the same problem as above may occur. Note that in these simple cases, where the definition of outliers is obvious with enough data, one could easily examine the data and exclude outliers a priori to train a generative model. In the case of more complicated problems such as the medical field [18], however, it is often not clear how outliers are to be defined. Moreover, because data are often scarce, even outliers are very useful and valuable in practice for training models and extracting features, making it difficult to filter outliers in advance and decide not to use them.
On the other hand, we also provide an example where it is very important to filter out outliers in the generator sample and then evaluate them. To evaluate the generator, samples are generated by sampling from the preset latent space (typically Gaussian). Even after training is complete and the generators' outputs are generally fine, there's a latent area where generators aren't fully trained. Note that latent space sampling may contain samples from regions that the generator does not cover well during training ("unfortunate outlier"). When unfortunate outliers are included, the existing evaluation metrics may underestimate or overestimate the generator's performance than its general performance. (To get around this, it is necessary to try this evaluation several times to statistically stabilize it, but this requires a lot of computation and becomes impractical, especially when the latent space dimension is high.)
Especially considering the evaluation scenario in the middle of training, the above situation is likely to occur due to frequent evaluation, which can interrupt training or lead to wrong conclusions. On the other hand, we can expect that our metric will be more robust against the above problem since it pays more attention to the core (samples that form topologically meaningful structures) generation performance of the model.
G.3 Details of the noise framework in the experiments
We have assessed the robustness of TopP&R against two types of non-IID noise perturbation through toy data experiments (in Section 5.1.3) and real data experiments (in Section 5.2.2). The scatter noise we employed consists of randomly extracted noise from a uniform distribution. This noise, even when extensively added to our data, does not form a data structure with any significant signal (topological feature). In other words, from the perspective of TopP&R, the formation of topologically significant data structures implies that data samples should possess meaningful probability values in the feature space. However, noise following a uniform distribution across all feature dimensions holds very small probability values, and thus, it does not constitute a topologically significant data structure. Consequently, such noise is filtered out by the bootstrap bands  $c_{\mathcal{X}}$  or  $c_{\mathcal{Y}}$  that we approximate (see Section 2).