SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure A2: Statistics of different samples' activations. There are clear margins between different statistics (e.g., Max and Min) of each sample, suggesting that the activation distribution is very peaky.
C User Preference Study
Here we present more details about our human perceptual judgment. Recall that two strategies are designed for different investigations, namely benchmarking the synthesis quality of one specific generative model and comparing two paired generative models. Fig. A5 shows the user interface for benchmarking the synthesis quality of one specific generative model (i.e., BigGAN on ImageNet here). To be more specific, considerable randomly generated images are shown to the user, and the user is required to determine the fidelity of synthesized images. We then obtain the final scores by averaging the judgments of the participants (i.e., 100 individuals).
Fig. A6 and Fig. A7 show the human evaluation results on FFHQ and ImageNet dataset, respectively. The percentages denote how many samples of the selected images are considered photo-realistic. Together with the quantitative results in our main paper, we could tell that the proposed metric shows a better correlation with human visual comparison.
Recall that in our main paper, we find that our evaluation system gives the opposite ranking to the existing metric (i.e., FID) in some circumstances. For instance, the synthesis quality of ICGAN is determined basically the same as that of the class-conditional ICGAN (C-ICGAN) under our evaluation, whereas the FID votes C-ICGAN for the much better one. We thus conduct the other user
Figure A3: The class distribution of randomly generated images (left) and histogram matched images (right), predicted by the fully-supervised Inception-V3 [56].
Figure A4: Fréchet Distance (FD) and Centered Kernel Alignment (CKA) scores evaluated under various data regimes on ImageNet dataset. FID scores are scaled for better visualization.  $\downarrow$  denotes the results fluctuate downward. The percentages represent the magnitude of the numerical variation.
study to compare two paired generative models. Concretely, we prepare groups of paired images of different generative models and ask 100 individuals to assess which model could produce high-quality images. The same groups are repeated several times by changing the order of images, ensuring the human evaluation is reliable and consistent.
Fig. A8 provides the interface of comparing two paired generative models, users are asked to choose which set of images looks more plausible. Additionally, Fig. A9 shows the pipeline of analyzing the paired comparison results. Specifically, the same groups of images are repeated for 4 times in random order and users are shown 16 images from two models to determine the more photorealistic one. In this way, the results of choosing both Projected-GAN and StyleGAN2 two times are identified as indistinguishable for enduring the consistency. Namely, the users choose different rankings between the two sets when the order of images is changed, which does not meet the consistency. Consequently, the final scores for paired comparison are obtained by quantifying the percentage of the human preferences that correlate the consistency.
Figure A5: User interface for benchmarking the synthesis quality.
Figure A6: Human judgment results of various generative models on FfHQ.  $2K$  images randomly generated by different models are selected for comparison.
D More Quantitative and Qualitative Results
In this section, we provide more quantitative and qualitative results to further demonstrate the efficacy of our newly developed measurement system.
Comparing with more metrics. In order to further investigate the efficacy of our proposed metric, we we further involve Kernel Inception Distance (KID) [4], precision, and recall [49] into our comparison. Note that the original KID employs Inception-V3 as the feature extractor, and there is a large "perceptual null space" in Inception-V3. Therefore, we first investigate whether KID scores can be altered by attacking the feature extractor with the histogram matching mechanism. The experimental details are consistent with computing Fréchet Distance  $(\mathrm{FD}_{\downarrow})$  in Tab.2 of the main paper. Tab. A4 presents the quantitative results. Still, some extractors, such as Inception, Swin-Transformer, and ResMLP, are susceptible to the histogram matching attack. For instance, the KID score of Swin-Transformer is improved by  $5.31\%$  when the chosen set is used. These observations agree with our findings in our main paper, suggesting that certain extractors can be hacked when KID is employed as the distributional distance. Then, we investigate the sample efficiency of KID, Precision, and Recall to probe the impacts of the amount of generated samples. Fig. A10 presents the curves of KID, Precision, and Recall scores computed under different data regimes. Similarly, we could observe that the KID scores can be improved by synthesizing more images. Interestingly, the recall scores decrease as the generated sample size increases whereas the precision is stable. This is caused by the definition of recall: recall measures the proportion of the real distribution that is covered by
Figure A7: Human judgment results of various generative models on ImageNet.  $2K$  images randomly generated by different models are selected for comparison.
Table A4: Quantitative comparison results of Kernel Inception Distance  $(\mathbf{KID}_{\downarrow}, \times e^{-3})$  on FFHQ dataset. "Random, Chosen  $I$  " respectively represent the synthesized distribution of randomly generated and matching the class prediction of Inception-V3. Moreover,  $R$  and  $V$  respectively denote the architecture of ResNet and ViT.  $(\downarrow)$  indicates the results are hacked by the histogram matching mechanism. Notably, the values across different rows are not comparable and we report the stds of three testings to better illustrate the numerical fluctuation of various extractors towards the histogram attack.
Model	Inception	ConvNeXt	SWAV	MoCo-R	RepVGG	CLIP-R
Random	1.88±0.02	34.81±0.11	9.61±0.06	5.31±0.06	33.88±0.29	2.85±0.05
ChosenI	1.71±0.02↓	34.82±0.10	9.61±0.06	5.31±0.05	33.89±0.27	2.85±0.05
Model	Swin	ViT	DeiT	CLIP-V	MoCo-V	ResMLP
Random	21.64±0.10	16.74±0.10	18.01±0.19	38.06±0.20	15.41±0.09	4.86±0.02
ChosenI	20.49±0.09↓	16.74±0.12	19.39±0.22	38.09±0.19	15.40±0.07	4.70±0.02↓
the synthesized distribution. In practical computation, the denominator increases as the synthesized samples increases, while the numerator (i.e., images from the real distribution) remain unchanged. In this way, the recall scores decrease as the generated sample size increases and vice versa. By contrast, CKA scores are stable under different data regimes, (please see Fig. 2 in the main paper). Moreover, CKA can provide reliable synthesis evaluation that agrees with human visual judgment. Accordingly, CKA is a proper choice for building a consistent and reliable measurement system.
More hackability results of Fréchet Distance on ImageNet. In addition to evaluating the robustness of these extractors on the FFHQ dataset, we further perform the same experiment on the ImageNet dataset. Tab. A5 presents the quantitative results. We can tell from these results that the chosen feature extractors are robust to the attack, further demonstrating their reliability.
More results of MLP-based extractors. We further incorporate two MLP-based models as the feature extractor for synthesis evaluation, namely namely gMLP [37] and MLP-mixer [59]. Following
Figure A8: User interface for comparing the synthesis quality of two paired generative models. People are asked to determine which set of images look more photorealistic.
Figure A9: The pipeline of analyzing the paired comparison results.
the experimental settings in our main paper, we identify the reliability and robustness of these MLP-based models via 1) visualizing the highlighted regions that contribute most significantly to the measurement results, and 2) attacking the feature extractor with histogram matching attack. Tab. A6 presents the qualitative (left) and quantitative (right) results. On one hand, the heatmap visualization results indicate that both gMLP and mixer-MLP capture limited semantics. Considering that more visual semantics should be considered for a more comprehensive evaluation, gMLP and MLP-mixer might not be adequate for synthesis comparison. On the other hand, the quantitative results demonstrate that their FD scores could be altered by the histogram matching attack, without actually improving the synthesis quality. That is, gMLP and MLP-mixer are susceptible to the histogram attack. Together with the finding that the FD scores of ResMLP could be manipulated without any improvement to the generative models in Tab. 2 of our main paper, we do not integrate MLP-based feature extractors into our measurement system.
Figure A10: Kernel Inception Distance (KID), Precision, and Recall scores evaluated under various data regimes on FFHQ dataset. The scores are scaled for better visualization.  $\downarrow$  denotes the results fluctuate downward. The percentages represent the magnitude of the numerical variation.
Table A5: Quantitative comparison results of Fréchet Distance  $(\mathbf{FD}_{\downarrow})$  on ImageNet dataset. "Random, Chosen  $I$  " respectively represent the synthesized distribution of randomly generated and matching the class prediction of Inception-V3. Moreover,  $R$  and  $V$  respectively denote the architecture of ResNet and ViT.  $(\downarrow)$  indicates the results are hacked by the histogram matching mechanism. Notably, the values across different rows are not comparable and we report the stds of three testings to better illustrate the numerical fluctuation of various extractors towards the histogram attack.
Model	Inception	ConvNeXt	SWAV	MoCo-R	RepVGG	CLIP-R
Random	34.29±0.09	78.02±0.16	0.13±0.003	0.32±0.002	54.98±0.22	27.64±0.15
ChosenI	33.05±0.08↓	78.10±0.14	0.13±0.002	0.32±0.002	54.30±0.24	27.66±0.17
Model	Swin	ViT	DeiT	CLIP-V	MoCo-V	ResMLP
Random	323.12±0.88	50.97±0.20	621.98±1.02	5.46±0.09	50.01±0.21	145.32±1.02
ChosenI	301.91±0.92↓	50.96±0.18	597.32±1.11↓	5.46±0.07	50.00±0.19	133.06±1.09↓
Table A6: Heatmaps from MLP-based extractors, namely Mixer-MLP [59] and gMLP [37]. (left) and Quantitative comparison results of MLP-based extractors' Fréchet Distance  $(\mathbf{FD}_{\downarrow})$  on the ImageNet dataset (right). ViT [16] serves as the feature extractor for hierarchical evaluation here.
Extractor	Random	ChosenI
gMLP	2.93±0.004	2.89±0.004↓
mixer-MLP	5.51±0.01	5.35±0.01↓
Similarities between various representation spaces. Recall that we filtered out extractors that define similar representation spaces to avoid redundancy in the main paper. The correlation between
Figure A11: Representation similarity of various extractors. Darker Yellow denotes higher similarity.
representations of high dimension in different feature extractors is calculated following [32]. In particular, a considerable number of images (i.e.,  $10K$  images from ImageNet) are fed into these extractors for computing their correspondence. Fig. A11 shows the similarity of their representations. Obviously, the representations of CLIP-ResNet and MoCo-ResNet have higher similarity with other extractors. Considering these two extractors are both CNN-based and they capture similar semantics with other CNN-based extractors, we remove the CLIP-ResNet and MoCo-ResNet to avoid redundancy. Accordingly, we obtain a set of feature extractors that 1) capture rich semantics in a complementary way, 2) are robust toward the histogram matching attack, and 3) define meaningful and distinctive representation spaces for synthesis comparison. The following table presents these feature extractors. These extractors, including both CNN-based and ViT-based architectures,
have demonstrated strong performance in pre-defined and downstream tasks, facilitating more comprehensive and reliable evaluation. Notably, the inclusion of self-supervised extractors SWAV, CLIP-V, and MoCo-V aligns with previous findings [41, 35, 3]. This selection of feature extractors provides a diverse and complementary set of representations, enabling a more comprehensive and reliable evaluation of synthesis quality in generative models.
More results of hierarchical levels from various extractors. Tab. A7, Tab. A8, Tab. A9, Tab. A10, and Tab. A11 respectively present the heatmaps and quantitative results of various semantic levels. We could tell that despite the Fréchet Distance (FD) scores consistently reflect synthesis quality, their numerical values fluctuate dramatically. On the contrary, CKA provides normalized distances w.r.t the numerical scale across various levels. Also, the heatmaps from various semantic levels reveal that hierarchical features encode different semantics. Such observation provides interesting insights that feature hierarchy should be also considered for synthesis comparison. Notably, benefiting from the bounded quantitative results, CKA demonstrates great potentials for comparison across hierarchical layers.
Table A7: Heatmaps from various semantic levels on FFHQ dataset (left) and quantitative results of Fréchet Distance (FD ↓) and Centered Kernel Alignment (CKA ↑) on ImageNet dataset (right). ConvNext [40] serves as the feature extractor for hierarchical evaluation here.
Model	BigGAN	BigGAN-deep	StyleGAN-XL
Layer	FD↓	CKA↑	FD↓	CKA↑	FD↓	CKA↑
Layer1	2.64	96.08	2.56	96.35	0.58	98.24
Layer2	40.20	-	32.32	-	11.84	-
Layer3	687.40	58.76	364.95	60.25	264.87	62.53
Layer4	140.04	68.86	102.26	69.27	19.22	70.52
Overall	N/A	74.57	N/A	75.29	N/A	77.10
Table A8: Heatmaps from various semantic levels on FFHQ dataset (left) and quantitative results of Fréchet Distance (FD ↓) and Centered Kernel Alignment (CKA ↑) on ImageNet dataset (right). RepVGG [15] serves as the feature extractor for hierarchical evaluation here.
Model	BigGAN	BigGAN-deep	StyleGAN-XL
Layer	FD↓	CKA↑	FD↓	CKA↑	FD↓	CKA↑
Layer1	0.35	96.92	0.32	97.51	0.04	98.79
Layer2	0.35	96.19	0.33	96.32	0.03	98.32
Layer3	0.23	90.05	0.18	91.15	0.04	93.51
Layer4	67.53	74.40	58.85	76.68	15.93	80.28
Overall	N/A	89.39	N/A	90.42	N/A	92.73
Table A9: Heatmaps from various semantic levels on FFHQ dataset (left) and quantitative results of Fréchet Distance (FD ↓) and Centered Kernel Alignment (CKA ↑) on ImageNet dataset (right). SWAV [6] serves as the feature extractor for hierarchical evaluation here.
Model	BigGAN	BigGAN-deep	StyleGAN-XL
Layer	FD↓	CKA↑	FD↓	CKA↑	FD↓	CKA↑
Layer1	0.67	99.90	0.46	99.91	0.07	99.99
Layer2	0.87	97.89	0.60	98.87	0.31	99.51
Layer3	16.15	95.60	12.02	96.21	1.90	98.15
Layer4	11.18	86.10	8.69	87.71	1.85	92.54
Overall	N/A	94.87	N/A	95.68	N/A	97.55
Table A10: Heatmaps from various semantic levels on FFHQ dataset (left) and quantitative results of Fréchet Distance (FD ↓) and Centered Kernel Alignment (CKA ↑) on ImageNet dataset (right). ViT [16] serves as the feature extractor for hierarchical evaluation here.
Model	BigGAN	BigGAN-deep	StyleGAN-XL
Layer	FD↓	CKA↑	FD↓	CKA↑	FD↓	CKA↑
Layer1	0.20	99.62	0.19	99.67	0.01	99.97
Layer2	1.31	97.75	1.19	97.92	0.18	99.76
Layer3	6.93	97.53	6.06	97.63	1.22	99.67
Layer4	29.95	96.49	23.98	97.20	8.51	98.72
Overall	N/A	97.85	N/A	98.11	N/A	99.53
Table A11: Heatmaps from various semantic levels on FFHQ dataset (left) and quantitative results of Fréchet Distance (FD ↓) and Centered Kernel Alignment (CKA ↑) on ImageNet dataset (right). MoCo-ViT [9] serves as the feature extractor for hierarchical evaluation here.
Shallow	Deep
Model	BigGAN	BigGAN-deep	StyleGAN-XL
Layer	FD↓	CKA↑	FD↓	CKA↑	FD↓	CKA↑
Layer1	0.10	98.62	0.05	99.04	0.04	99.97
Layer2	1.01	97.15	0.68	97.30	0.43	99.64
Layer3	9.18	96.07	9.01	96.77	4.30	99.11
Layer4	3.35	97.25	3.22	97.82	1.85	99.00
Overall	N/A	97.27	N/A	97.73	N/A	99.43
E Visualized quantitative results
To make our results easier to parse, we visualize the correlation between different metrics and the human evaluation results. Specifically, we plot the correlation of the averaged ranks of various models given by human judgment, CKA, and FID. Fig. A12 and Fig. A13 respectively present the visualization results of the ImageNet, FFHQ, and LSUN-Church datasets. Obviously, the averaged ranks given by CKA are more consistent with that of the human evaluation, demonstrating the accuracy of CKA. Moreover, we plot the comparison between the stds and the improvements obtained by the histogram attack for better illustration. Fig. A14 presents the results. Similarly, we could observe that the improvement is actually caused by the histogram attack rather than the variance of attempts.
Figure A12: The correlation of the averaged ranks of various models on ImageNet given by human judgment, CKA, and FID.
Figure A13: The correlation of the averaged ranks of various models on FFHQ and LSUN-Church given by human judgment, CKA, and FID.
Figure A14: The quantitative comparison between the stds and the improvements obtained by the histogram attack.