SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We experiment to ascertain whether utilizing a random projection address the drawbacks of existing metrics that are susceptible to noise, and we also conduct tests to validate whether TopP&R possesses noise robustness without using random projection. Through Figure A6 (a), it is evident that P&R and D&C still retain vulnerability to noisy features, and it is apparent that random projection itself does not diminish the impact of noise. Furthermore, Figure A6 (b) reveals that TopP&R , even without utilizing a random projection, exhibits robustness to noise through approximating the data support based on statistically and topologically significant features.
Building upon the insight from the toy data experiment in Figure A6 that random projection does not confer noise robustness to P&R and D&C, we aim to further demonstrate this fact through a real data experiment. In the experiment depicted in Figure A7, we follow the same setup as the previous experiment in Figure 5, applying the same random projection to P&R and D&C only. The results of
Diversity
Figure A7: Comparison of evaluation metrics on Non-IID perturbations using FFHQ dataset. We replaced certain ratio of  $\mathcal{X}$  and  $\mathcal{Y}$  (a) with outliers and (b) by switching some of real and fake features. P&R and D&C were computed using the same 32-dimensional random projection.
this experiment affirms that random projection does not provide additional robustness against noise for P&R and D&C, while TopP&R continues to exhibit the most robust performance.
I.8 Robustness of TopP&R with respect to random projection dimension
(a) Random projection to 64-dimensional space
Figure A8: Comparison of evaluation trends of TopP&R with respect to random projection dimension in both sequential and simultaneous mode drop scenarios. Baby ImageNet dataset is used for the comparison, and TopP&R is computed using random projections to (a) 64 dimensions and (b) 128 dimensions.
(b) Random projection to 128-dimensional space
To investigate whether the dimension of random projection influences the evaluation trend of TopP&R, we conduct mode dropping experiment using the Baby ImageNet dataset, similar to the experiment in Section I.3. Results in (a) and (b) of Figure A8 respectively compare the performance of TopP&R using random projection in dimensions of 64 and 128 with other metrics. From the experimental results, we observe that TopP&R is the most sensitive to the distributional changes, akin to the tendencies observed in the previous toy mode dropping experiment. Furthermore, upon examining the differences across random projection dimensions, TopP&R shows consistent evaluation trends regardless of dimensions.
I.9 Trucation trick
$\psi$  is a parameter for the truncation trick and is first introduced in [4] and [1]. We followed the approach in [4] and [1]. GANs generate images using the noise input  $z$ , which follows the standard normal distribution  $\mathcal{N}(0, I)$  or uniform distribution  $\mathcal{U}(-1, 1)$ . Suppose GAN inadvertently samples noise outside of distribution, then it is less likely to sample the image from the high density area of the image distribution  $p(z)$  defined in the latent space of GAN, which leads to generate an image with artifacts. The truncation trick takes this into account and uses the following truncated distribution. Let  $f$  be the mapping from the input to the latent space. Let  $w = f(z)$ , and  $\bar{w} = \mathbb{E}[f(z)]$ , where  $z$  is either from  $\mathcal{N}(0, I)$  or  $\mathcal{U}(-1, 1)$ . Then we use  $w' = \bar{w} + \psi(w - \bar{w})$  as a truncated latent vector. If the value of  $\psi$  increases, then the degree of truncation decreases which makes images have greater diversity but possibly lower fidelity.
I.10 Toy experiment of trade-off between fidelity and diversity
We have designed a new toy experiment to mimic the truncation trick. With data distributions of 10k samples,  $\mathcal{X} \sim \mathcal{N}(\mu = 0, I) \in \mathbb{R}^{32}$  and  $\mathcal{Y} \sim \mathcal{N}(\mu = 0.6, \sigma^2) \in \mathbb{R}^{32}$ , we measure the fidelity and diversity while incrementally increasing  $\sigma$  from 0.7 to 1.3. For smaller  $\sigma$  values, the evaluation metric should exhibit higher fidelity and lower diversity, while increasing  $\sigma$  should demonstrate decreasing
Figure A9: Trade-off between fidelity and diversity in a toy experiment. In this experiment, the  $\mu$  of  $\mathcal{X}$  was set to 0, and the  $\mu$  of  $\mathcal{Y}$  was set to 0.6. By incrementally increasing  $\sigma$  from 0.7 to 1.3, the evaluation trend based on changes in the  $\mathcal{Y}$  distribution is measured.
fidelity and higher diversity. From the results in Figure A9, both TopP&R and P&R exhibit a trade-off between fidelity and diversity.
I.11 Resolving fidelity and diversity
Figure A10: Behaviors of metrics with truncation trick. The horizontal axis corresponds to the value of  $\psi$  denoting the increased diversity. The images are generated via StyleGAN2 with FFHQ dataset [1].
To test whether TopP&R responds appropriately to the change in the underlying distributions in real scenarios, we test the metric on the generated images of StyleGAN2 [2] using the truncation trick [1]. StyleGAN2, trained on FFHQ as shown by Han et al. [49], tends to generate samples mainly belonging to the majority of the real data distribution, struggling to produce rare samples effectively. In other words, while StyleGAN2 excels in generating high-fidelity images, it lacks diversity. Therefore, the evaluation trend in this experiment should reflect a steady high fidelity score while gradually increasing diversity. Han et al. [49] demonstrates that the generated image quality of StyleGAN2 that lies within the support region exhibits sufficiently realistic visual quality. Therefore, particularly in the case of TopP&R, which evaluates fidelity excluding noise, the fidelity value should remain close to 1. As shown in Figure A10, every time the distribution is transformed by  $\psi$ , TopP&R responds well and shows consistent behavior across different embeddings with bounded scores in  $[0,1]$ , which are important virtues as an evaluation metric. On the other hand, Density gives unbounded scores (fidelity  $>1$ ) and shows inconsistent trend depending on the embedder. Because Density is not capped in value, it is difficult to interpret the score and know exactly which value denotes the best performance (e.g., in our case, the best performance is when TopP&R = 1).