SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The alternate noise we utilized in our experiments, swap noise, possesses distinct characteristics from scatter noise. Initially, given the real and fake data that constitute important data structures, we introduce swap noise by randomly selecting real and fake samples and exchanging their positions. In this process, the swapped fake samples follow the distribution of real data, and conversely, the real samples adhere to the distribution of fake data. This implies the addition of noise that follows the actual data distribution, thereby generating significant data structures. When the number of samples undergoing such positional swaps remains small, meaningful probability values cannot be established within the distribution. Statistically, an extremely small number of samples cannot form a probability distribution itself. As a result, our metric operates robustly in the presence of noise. However, as the count of these samples increases and statistically significant data structures emerge, TopP&R estimates the precise support of such noise as part of their distribution. By examining Figure 4 and 5, as well as Table A12, it becomes evident that conventional metrics struggle with accurate support estimation due to vulnerability to noise. This limitation results in an inability to appropriately evaluate aspects involving minority sets or data forming long distributions.
Since the KDE filtration requires extensive computations in high dimensions, a random projection that preserves high-dimensional distance and topological properties is inevitable. Based on the topological structure of the features present in the embedded low dimensional space, we have shown that TopP&R theoretically and experimentally has several good properties such as robustness to the noise from various sources and sensitivity to small changes in distribution. However, matching the distortion from the random projection to the noise level allowed by the confidence band is practically infeasible: The bootstrap confidence band is of order  $\left(nh_n^d\right)^{-\frac{1}{2}}$ , and hence for the distortion from the random projection to match this confidence band, the embedded dimension  $d$  should be of order  $\Omega \left(nh_n^d\log n\right)$  from Remark (B.2). However, computing the KDE filtration in dimension  $\Omega \left(nh_n^d\log n\right)$  is practically infeasible. Hence in practice, the topological distortion from the random projection is not guaranteed to be filtered out by the confidence band, and there is a possibility of obtaining a less accurate evaluation score compared to calculations in the original dimensions.
Exploring the avenue of localizing uncertainty at individual data points separately presents an intriguing direction of research. Such an approach could potentially be more sample-efficient and disregard less data points. However, considering our emphasis on preserving topological signals, achieving localized uncertainty estimation in this manner is challenging within the current state of the art. For instance, localizing uncertainty in the kernel density estimate  $\hat{p}$  is feasible, as functional variability is inherently local. To control uncertainty at a specific point  $x$ , we primarily need to analyze the function value  $\hat{p}(x)$  at that point. In contrast, localizing uncertainty for homological features situated at point  $x$  requires the analysis of all points connected by the homological feature. Consequently, even if our intention is to confine uncertainty to a local point, it demands estimating the uncertainty at the global structural level. This makes localizing the uncertainty for topological features difficult given the tools in topology and statistics we currently have.
H.1 Implementation details of embedding
We summarize the detailed information of our embedding networks implemented for the experiments. In Figure 2, 3, 4, A10, A4, and 5, P&R and D&C are computed from the features of ImageNet pretrained VGG16 (fc2 layer), and TopP&R is computed from features placed in  $\mathbb{R}^{32}$  with additional random linear projection. In the experiment in Figure A10, the SwAV embedder is additionally considered. We implement ImageNet pre-trained InceptionV3 (fc layer), VGG16 (fc2 layer), and SwAV as embedding networks with random linear projection to 32 dimensional feature space to compare the ranking of GANs in Table 1.
H.2 Implementation details of confidence band estimator
Algorithm 1 Confidence Band Estimator
KDE: kernel density estimator # grid search for the confidence band   
# h: kernel bandwidth parameter for  $q\in [\min (\hat{\theta}),\max (\hat{\theta})]$  do   
# k: number of repeats count = 0   
#  $\hat{\theta}$  : set of difference for element  $\in \hat{\theta}$  do   
Given  $\mathcal{X} = \{X_1,X_2,\dots ,X_n\}$  count = count + 1   
 $\hat{p}_h = KDE(\mathcal{X})$  end if   
for iteration  $= 1,2,\ldots ,k$  do end for   
# Define  $\mathcal{X}^*$  with bootstrap sampling # define the band threshold   
 $\mathcal{X}^{*} =$  random sample n times with repeat from X if count/k ≈ α then   
#  $\hat{p}_h^*$  replaces population density  $q_{\alpha} = q$ $\hat{p}_h^* = KDE(\mathcal{X}^*)$  end if   
# compute  $\hat{\theta}$  with bootstrap samples end for   
Append  $\hat{\theta}$  with  $\sqrt{n} ||\hat{p}_h - \hat{p}_h^* ||_\infty$  # define estimated confidence band   
end for  $c_{\alpha} = q_{\alpha} / \sqrt{n}$
H.3 Choice of confidence level
For the confidence level  $\alpha$ , we would like to point out that  $\alpha$  is not the usual hyperparameter to be tuned: It has a statistical interpretation of the probability or the level of confidence to allow error, noise, etc. The most popular choices are  $\alpha = 0.1, 0.05, 0.01$ , leading to  $90\%$ ,  $95\%$ ,  $99\%$  confidence. We used  $\alpha = 0.1$  throughout our experiments.
H.4 Estimation of Bandwidth parameter
As we discussed in Section 2, since TopP&R estimates the manifold through KDE with kernel bandwidth parameter  $h$ , we need to approximate it. The estimation techniques for  $h$  are as follows: (a) a method of selecting  $h$  that maximizes the survival time  $(S(h))$  or the number of significant homological features  $(N(h))$  based on information obtained about persistent homology using the filtration method, (b) a method using the median of the k-nearest neighboring distances between features obtained by the balloon estimator (for more details, please refer to [42], [43], and [44]). Note that, the bandwidths  $h$  for all the experiments in this paper are estimated via Balloon Bandwidth Estimator.
For (a), following the notation in Section A, let the  $i$ th homological feature of persistent diagram be  $(b_i, d_i)$ , then we define its life length as  $l_i(h) = d_i - b_i$  at kernel bandwidth  $h$ . With confidence band  $c_\alpha(h)$ , we select  $h$  that maximizes one of the following two quantities:
$$
N (h) = \# \{i: l _ {i} (h) > c _ {\alpha} (h) \}, S (h) = \sum_ {i} [ l _ {i} (h) - c _ {\alpha} (h) ] _ {+}.
$$
Note that, we denote the confidence band  $c_{\alpha}$  as  $c_{\alpha}(h)$  considering the kernel bandwidth parameter  $h$  of KDE in Algorithm 1.
For (b), the balloon bandwidth estimator is defined as below:
Algorithm 2 Balloon Bandwidth Estimator
#h: kernel bandwidth  $d(X_{idx},\mathcal{X}) = ||X_{idx} - \mathcal{X}||_2^2,$  # KND: kth nearest distance i.e.,  $\{d(X_{idx},X_1),d(X_{idx},X_2),\ldots \}$  # idx: index # Calculate kth nearest neighbor distance # sort: sort in ascending order  $KND_{idx} = \mathrm{sort}(d(X_{idx},\mathcal{X}))[\mathrm{k}]$  end for Given  $\mathcal{X} = \{X_1,X_2,\dots ,X_n\}$  # Define the estimated bandwidth  $\hat{h}$  # Compute L2 distance between  $X_{idx}$  and  $\mathcal{X}$ $\hat{h} =$  median(KND)
H.5 Computational complexity
TopP&R is computed using the confidence band  $c_{\alpha}$  and KDE bandwidth  $h$ . The computational cost of the confidence band calculation (Algorithm 1) and the balloon bandwidth calculation (Algorithm 2) are  $O(k*n^2*d)$  and  $O(n^2*d)$ , respectively, where  $n$  represents the data size,  $d$  denotes the data dimension, and  $k$  stands for the number of repeats ( $k = 10$  in our implementation). The resulting computational cost of TopP&R is  $O(k*n^2*d)$ . In our experiments based on real data, calculating TopP&R once using real and fake features, each consisting of 10k samples in a 4096-dimensional space, takes approximately 3-4 minutes. This computation speed is notably comparable to that of P&R and D&C, utilizing CPU-based computations. Note that, P&R and D&C that we primarily compare in our experiments are algorithms based on  $k$ -nearest neighborhood for estimating data distribution, and due to the computation of pairwise distances, the computational cost of these algorithm is approximately  $O(n^2*d)$ .
Table A1: Wall clock, CPU time and Time complexity for evaluation metrics for 10k real and 10k fake features in 4096 dimension. For TopP&R,  $r$  is random projection dimension  $\left( {r \ll  d}\right)$  . The results are measured by the time module in python.
Time	PR	DC	TopPR (Ours)
Wall Clock	1min 43s	1min 40s	1min 58s
CPU time	2min 44s	2min 34s	2min 21s
Time complexity	O(n2*d)	O(n2*d)	O(k*n2*r)
H.6 Mean hamming distance
Hamming distance (HD) [45] counts the number of items with different ranks between A and B, then measures how much proportion differs in the overall order, i.e. for  $A_{i} \in A$  and  $B_{i} \in B$ ,  $HD(A,B) \coloneqq \sum_{i=1}^{n} 1\{k : A_{i} \neq B_{i}\}$  where  $k$  is the number of differently ordered elements. The mean HD is calculated as follows to measure the average distances of three ordered lists: Given three ordered lists  $A, B$ , and  $C$ ,  $\bar{H}D = (HD(A,B) + HD(A,C) + HD(B,C)) / 3$ .
H.7 Explicit values of bandwidth parameter
Since our metric adaptively reacts to the given samples of  $P$  and  $Q$ , we have two  $h$ 's per experiment. For example, in the translation experiment (Figure 2), there are 13 steps in total, and each time we estimate  $h$  for  $P$  and  $Q$ , resulting in a total of  $26h$ 's. To show them all at a glance, we have listed all values in one place. We will also provide the code that can reproduce the results in our experiments upon acceptance.
Table A2: Bandwidth parameters  $h$  of distribution shift in Section 5.1.1.
μ	← shift	shift →
real h	8.79	8.60	8.58	8.67	8.63	8.71	8.66	8.73	8.54	8.63	8.54	8.75	8.78
fake h	8.46	8.86	8.60	8.49	8.44	8.80	8.55	8.63	8.74	8.61	8.58	8.83	8.60
Table A3: Bandwidth parameters  $h$  of sequential mode drop in Section 5.1.2.
Steps	0	1	2	3	4	5	6
real h	10.5	10.6	10.5	10.7	10.5	10.8	10.6
fake h	11.0	10.2	10.0	9.82	9.57	9.29	8.78
Table A4: Bandwidth parameters  $h$  of simultaneous mode drop in Section 5.1.2.
Steps	0	1	2	3	4	5	6	7	8	9	10
real h	10.5	10.6	10.6	10.8	10.3	10.6	10.7	10.6	10.4	10.4	10.6
fake h	10.6	10.5	10.7	10.3	10.4	10.0	9.88	9.31	9.19	9.07	8.79
Table A5: Bandwidth parameters  $h$  of non-IID noise perturbation in Section 5.1.3.
Steps	0	1	2	3	4	5	6	7	8	9
Scatter	real h	8.76	8.71	8.78	8.59	8.85	8.71	8.61	8.97	9.00	8.95
fake h	8.61	8.76	8.63	8.93	8.46	8.72	8.67	8.68	9.07	8.88
Swap	real h	8.65	8.44	8.76	8.62	8.55	8.75	8.96	8.59	8.53	8.74
fake h	8.52	8.55	8.68	8.97	8.87	8.94	8.84	9.05	8.94	8.90
Table A6: Bandwidth parameters  $h$  of FFHQ truncation trick in Section I.11.
Ψ	Ψ ↓	Ψ ↑
real h	6.26	6.41	6.31	6.23	6.06	6.24	6.20	6.61	6.39	6.28	6.26
fake h	1.04	1.67	2.36	2.90	3.41	3.93	4.31	4.90	5.51	5.60	6.46
Table A7: Bandwidth parameters  $h$  of CIFAR10 sequential mode drop in Section 5.2.1.
Steps	0	1	2	3	4	5	6	7	8	9
real h	7.09	6.95	6.96	6.92	7.02	6.81	6.86	6.88	6.78	6.67
fake h	6.87	6.75	6.88	6.64	6.40	6.64	6.41	6.44	6.62	6.04
Table A8: Bandwidth parameters  $h$  of CIFAR10 simultaneous mode drop in Section 5.2.1.
Steps	0	1	2	3	4	5	6	7	8	9	10
real h	6.78	6.95	7.05	6.62	6.85	6.51	6.94	6.91	6.80	7.00	6.78
fake h	6.75	6.69	7.14	6.68	6.75	6.84	6.61	6.68	6.50	6.32	6.17
Table A9: Bandwidth parameters  $h$  of FFHQ non-IID noise perturbation in Section 5.2.2.
Steps	0	1	2	3	4	5	6	7	8	9
Scatter	real h	7.24	7.03	7.01	7.45	7.38	7.66	7.46	7.36	7.79	7.66
fake h	5.41	5.77	5.67	5.49	5.71	5.65	5.82	6.08	5.81	6.02
Swap	real h	7.25	7.20	6.99	6.67	7.07	6.96	7.03	6.94	6.83	6.86
fake h	6.87	6.63	6.60	6.91	6.74	6.77	6.92	6.83	6.83	6.82
I.1 Survivability of minority sets in the long-tailed distribution
An important point to check in our proposed metric is the possibility that a small part of the total data (i.e., minority sets), but containing important information, can be ignored by the confidence band. We emphasize that since our metric takes topological features into account, even minority sets are not filtered conditioned that they have topologically significant structures. We assume that signals or data
Table A10: Bandwidth parameters  $h$  of FFHQ noise addition in Section 5.2.3.
Noise intensity	0	1	2	3
Gaussian noise	real h	6.52	6.27	6.41	6.49
fake h	6.32	6.15	4.06	2.64
Gaussian blur	real h	6.08	6.26	6.26	6.47
fake h	6.12	5.13	4.60	3.40
Black rectangle	real h	6.06	6.28	6.49	6.44
fake h	6.36	6.66	6.31	5.93
Table A11: Bandwidth parameters  $h$  of GAN ranking in Section 5.2.4.
StyleGAN2	ReACGAN	BigGAN	PDGAN	ACGAN	WGAN-GP
InceptionV3	real h	2.605	2.629	2.581	2.587	2.615	2.606
fake h	2.226	2.494	2.226	2.639	2.573	2.063
VGG16	real h	7.732	8.021	8.175	7.792	7.845	8.036
fake h	7.880	6.839	7.178	6.174	6.839	3.615
SwAV	real h	0.774	0.765	0.785	0.780	0.780	0.822
fake h	0.740	0.667	0.6746	0.619	0.627	0.502
that are minority sets have topological structures, but outliers exist far apart and lack a topological structure in general.
To test this, we experimented with CIFAR10 [46], which has 5,000 samples per class. We simulate a dataset with the majority set of six classes (2,000 samples per class, 12,000 total) and the minority set of four classes (500 samples per class, 2,000 total), and an ideal generator that exactly mimics the full data distribution. As shown in the Table A12, the samples in the minority set remained after the filtering process, meaning that the samples were sufficient to form a significant structure. Both D&C and TopP&R successfully evaluate the distribution for the ideal generator. To check whether our metric reacts to the change in the distribution even with this harsh setting, we also carried out the mode decay experiment. We dropped the samples of the minority set from 500 to 100 per class, which can be interpreted as an  $11.3\%$  decrease in diversity relative to the full distribution (Given (a) ratio of the number of samples between majority and minority sets  $= 12,000:2,000 = 6:1$  and (b)  $80\%$  decrease in samples per minority class, the true decay in the diversity is calculated as  $\frac{1}{(1 + 6)} \times 0.8 = 11.3\%$  with respect to the entire samples). Here, recall and coverage react somewhat less sensitively with their reduced diversities as 3 p.p. and 2 p.p., respectively, while TopR reacted most similarly (9 p.p.) to the ideal value. In summary, TopP&R shows much more sensitivity to the changes in data distribution like mode decay. Thus, once the minority set has survived the filtering process, our metric is likely to be much more responsive than existing methods.
Table A12: Proportion of surviving minority samples in the long-tailed distribution after the noise exclusion with confidence band  $c_{\chi^{\prime}}$ . The p.p. indicates the percentage points.
Before mode drop	After mode drop
Proportion of survival (before / after filtering)	100% → 59%	100% → 57%
TopP&R (Fidelity / Diversity)	0.99 / 0.96	0.99 / 0.87 (9 p.p ↓)
P&R (Fidelity / Diversity)	0.73 / 0.73	0.74 / 0.70 (3 p.p ↓)
D&C (Fidelity / Diversity)	0.99 / 0.96	1.02 / 0.94 (2 p.p ↓)
I.2 Experiment on Non-  $\mathbf{II}\mathbf{D}$  perturbation with outlier removal methods
We compared the performance of metrics on a denoised dataset using Local Outlier Factor (LOF) [47] and Isolation Forest (IF) [48]. The experimental setup is identical to that of Figure 4 (see Section G.3 for details of our noise framework). In this experiment, we applied the outlier removal method before calculating P&R and D&C. From the result, P&R and D&C still do not provide a stable evaluation, while TopP&R shows the most consistent evaluation for the two types of noise without changing its trend.
(a) Non-IID scatter noise
(b) Non-IID swap noise
Figure A2: Behaviors of evaluation metrics on Non-IID perturbations. The dotted line in the graph shows the performance of metrics after the removal of outliers using the IF. We use "Clean" as a prefix to denote the evaluation after IF.
(a) Non-IID scatter noise
Figure A3: Behaviors of evaluation metrics on Non-IID perturbations. The dotted line in the graph shows the performance of metrics after the removal of outliers using the LOF. We use "Clean" as a prefix to denote the evaluation after LOF.
Since TopP&R can discriminate topologically significant signals by estimating the confidence band (using KDE filtration function), there is no need to arbitrarily set cutoff thresholds for each dataset. On the other hand, in order to distinguish inliers from specific datasets using LOF and IF methods, a threshold specific to the dataset must be arbitrarily set each time, so there is a clear constraint that different results are expected each time for each parameter setting (which is set by the user). Through this, it is confirmed that our evaluation metric guarantees the most consistent scoring based on the topologically significant signals. Since the removal of outliers in the TopP&R is part of the support estimation process, it is not appropriate to replace our unified process with the LOF or IF. In detail, TopP&R removes outliers through a clear threshold called confidence band which is defined by KDE, and this process simultaneously defines KDE's super-level set as the estimated support. If this process is separated, the topological properties and interpretations of estimated support also disappear. Therefore, it is not practical to remove outliers by other methods when calculating TopP&R.
I.3 Sequential and simultaneous mode dropping with real dataset
Visualization on Baby ImageNet dataset BabyImageNet
Concentration on the first mode a) Sequential mode dropping
Figure A4: Comparison of evaluation metrics under sequential and simultaneous mode dropping scenario with Baby ImageNet[15].
Concentration on the first mode
(b) Simultaneous mode dropping
I.4 Sensitiveness to noise intensity
The purpose of this experiment (Figure A5) is to closely observe the noise sensitivity of metrics in Figure 6. As the noise intensity is incrementally increased until all the metrics converge to 0, the most ideal outcome for the metrics is to exhibit a linear trend to capture these changes most effectively. From the results, it is able to observe that both TopP&R and P&R exhibit the most linear trend in their evaluation outcomes, while D&C demonstrates that least capability to reflect differences based on noise intensity.
Figure A5: Verification of whether TopP&R can make an accurate quantitative assessment of noise image features. Gaussian noise, gaussian blur, salt and pepper, and black rectangle noise are added to the FFHQ images and emeded with T4096. The noise intensity is linearly increased until all metrics converge to zero.
I.5 Evaluating state-of-the-art generative models on the ImageNet
We conducted our evaluation using the ImageNet dataset, which covers a wide range of classes. We employed commonly accepted metrics in the community, including FID and KID, to rank the considered generative models effectively. The purpose of this experiment is not only to establish the applicability of our ranking metric for generative models, akin to existing single-score metrics, but also to emphasize its interpretability. In Table A13, we calculated the HD between P&R variants and the more reliable KID metric. This comparison aimed to assess the alignment of our metric with established single-score metrics. The results consistently showed that TopP&R's evaluation results are the most consistent with single-score metrics across various embedding networks. Conversely, P&R exhibited an inconsistent evaluation trend compared to KID, which struggled to effectively differentiate between ReACGAN and BigGAN. Note that, Kang et al. [15] previously demonstrated that BigGAN performs worse than ReACGAN. However, surprisingly, P&R consistently rated BigGAN as better than ReACGAN across all embedding networks. These findings imply that the P&R's unreliable support estimation under noisy conditions might hinder its ability to accurately distinguish between generative models.
Table A13: Ranking results based on FID, KID, MTD, TopP&R, P&R and D&C for ImageNet (128 × 128) trained generative models. SwAV, VGG16, and InceptionV3 represent the embedders, and the numbers in parentheses indicate the ranks assigned by each metric to the evaluated models. The HD is the hamming distance between KID and metrics.
Model	ADM	StyleGAN-XL	ReACGAN	BigGAN	HD
InceptionV3	FID (↓)	9.8715 (1)	10.943 (2)	24.409 (3)	33.935 (4)	0
KID (↓)	0.0010 (1)	0.0025 (2)	0.0062 (3)	0.0205 (4)	-
TopP&R (↑)	0.9168 (1)	0.8919 (2)	0.8886 (3)	0.7757 (4)	0
D&C (↑)	1.0333 (2)	1.0587 (1)	0.8468 (3)	0.5400 (4)	2
P&R (↑)	0.6988 (1)	0.6713 (2)	0.4462 (4)	0.5079 (3)	2
MTD (↓)	1.7308 (2)	1.5518 (1)	2.2122 (3)	3.0891 (4)	2
VGG16	TopP&R (↑)	0.9778 (1)	0.8896 (2)	0.8376 (3)	0.5599 (4)	0
D&C (↑)	1.0022 (3)	0.9159 (4)	1.1322 (1)	1.1020 (2)	4
P&R (↑)	0.7724 (2)	0.7785 (1)	0.4092 (4)	0.5237 (3)	4
MTD (↓)	15.748 (3)	22.313 (4)	10.268 (1)	13.992 (2)	4
SwAV	TopP&R (↑)	0.8851 (2)	0.9102 (1)	0.6457 (3)	0.4698 (4)	2
D&C (↑)	1.0717 (1)	0.8898 (4)	1.0654 (2)	1.0578 (3)	3
P&R (↑)	0.6096 (1)	0.5596 (2)	0.1335 (4)	0.1544 (3)	2
MTD (↓)	0.4165 (3)	0.7789 (4)	0.3601 (2)	0.3200 (1)	4
Table A14: Ranking results based on TopP&R, P&R, and D&C for ImageNet (128 × 128) trained generative models. SwAV, VGG16, and InceptionV3 represent the embeddings, and the numbers in parentheses indicate the ranks assigned by each metric to the evaluated models. Note that, the random projection is applied to TopP&R, P&R, and D&C.
Model	ADM	StyleGAN-XL	ReACGAN	BigGAN
VGG16 InceptionV3	TopP&R (↑)	0.9168 (1)	0.8919 (2)	0.8886 (3)	0.7757 (4)
D&C w/ rand proj (↑)	1.0221 (2)	1.0249 (1)	0.9218 (3)	0.7099 (4)
P&R w/ rand proj (↑)	0.7371 (1)	0.7346 (2)	0.6117 (4)	0.6305 (3)
TopP&R (↑)	0.9778 (1)	0.8896 (2)	0.8376 (3)	0.5599 (4)
SwAV	D&C w/ rand proj (↑)	1.0086 (3)	0.9187 (4)	1.0895 (1)	1.0803 (2)
P&R w/ rand proj (↑)	0.7995 (2)	0.8056 (1)	0.5917 (4)	0.6304 (3)
TopP&R (↑)	0.8851 (2)	0.9102 (1)	0.6457 (3)	0.4698 (4)
D&C w/ rand proj (↑)	1.0526 (3)	0.9479 (4)	1.1037 (2)	1.1127 (1)
P&R w/ rand proj (↑)	0.6873 (1)	0.6818 (2)	0.4179 (3)	0.4104 (4)
I.6 Verification of random projection effect in generative model ranking
We also applied the same random projection used by TopP&R to P&R and D&C in Section I.6. From the results of the Table A14, it is evident that the application of random projection to P&R leads to different rankings than its original evaluation tendencies. Similarly, the rankings of D&C with random projection still exhibit significant variations based on the embedding. Thus, we have demonstrated that random projection does not provide consistent evaluation tendencies across different embeddings, while also highlighting that this characteristic is unique to TopP&R. To quantify these differences among metrics, we computed the mean Hamming Distance (MHD) between the results in the embedding spaces of each metric. The calculated MHD scores were 1.33 for TopP&R, 2.67 for P&R, and 3.33 for D&C, respectively.
I.7 Verifying the effect of random projection to the noisy data
(a) All metrics with random projection to 32-dimensional space
Figure A6: Behaviors of evaluation metrics for outliers on real and fake distribution. For both real and fake data, the outliers are fixed at  $3 \in \mathbb{R}^d$  where  $d$  is a dimension, and the parameter  $\mu$  is shifted from -1 to 1. (a) depicts the comparison of all metrics using a random projection from 64 dimensions to 32 dimensions. (b) illustrates the difference between metrics without using random projection, where real and fake data are considered in 32 dimensions.
(b) All metrics without random projection