SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CKA demonstrates great potential for quantitative comparison and combination across hierarchical layers. Here we investigate the distributional distances across various layers of the neural network, as these layers typically extract multi-level features that span from high-level semantics to low-level details. Accordingly, considering hierarchical features can provide a more comprehensive measurement. The left part of Tab. 4 presents the qualitative results of different layers' heatmaps. We can observe that different layers indeed extract different semantics, highlighting the importance of considering hierarchical features in evaluation. Additionally, we provide the quantitative results of FD and CKA in the right part of Tab. 4. Still, the FD scores of various layers fluctuate dramatically, e.g., the FD results of 0.60 in  $\text{Layer}_1$  while 104.10 in  $\text{Layer}_4$ . Differently, the CKA results from hierarchical layers are comparable and the overall score could be derived by averaging multi-level scores. Importantly, the overall score still reflects synthesis quality consistently and reliably.
CKA shows satisfactory sample-efficiency and stability under different number of samples. Typically, the synthesis quality are measured between real and synthesized distributions, where the
Table 3: Quantitative comparison results of Fréchet Distance  $(\mathbf{FD}_{\downarrow})$  on ImageNet dataset.  $^\dagger$  scores are quoted from the original paper and other results are tested three times. Notably, the values across different columns are not comparable.
Model	FID†	ConvNeXt	RepVGG	SWAV	ViT	MoCo-ViT	CLIP-ViT	Overall	User study
BigGAN [5]	8.70	140.04	67.53	1.12	29.95	238.78	3.35	N/A	53%
BigGAN-deep [5]	6.02	102.26	58.85	0.87	23.98	85.83	3.22	N/A	55%
StyleGAN-XL [51]	1.81	19.22	15.93	0.18	8.51	29.38	1.85	N/A	67%
Table 4: Heatmaps from various semantic levels on FFHQ dataset (left) and Fréchet Distance  $(\mathbf{FD}_{\downarrow})$  and Centered Kernel Alignment  $(\mathbf{CKA}_{\uparrow})$  scores on ImageNet dataset (right). CLIP-ViT serves as the feature extractor here, more results can be found in Supplementary Material.
Model	BigGAN	BigGAN-deep	StyleGAN-XL
Layer	FD↓	CKA↑	FD↓	CKA↑	FD↓	CKA↑
Layer1	0.60	99.06	0.54	98.95	0.05	99.84
Layer2	7.45	86.89	5.58	90.09	0.77	91.06
Layer3	30.24	82.80	23.55	83.63	6.11	85.75
Layer4	104.10	80.13	81.02	81.05	35.77	83.55
Overall	N/A	87.22	N/A	88.43	N/A	90.05
whole training data is used as the real distribution and  $50K$  generated images as the synthesized, regardless of how many samples contained in the training data. However, when evaluating on the large-scale datasets (e.g., 1.28 million images for ImageNet),  $50K$  images may be insufficient to represent the entire distribution. Therefore, we study the impacts of the amount of generated samples. Concretely, we prepare several synthesized sets with different numbers of samples and calculate their distances to the real distribution (more details are presented in Supplementary Material).
Fig. 2 demonstrates the curves of FD and CKA scores evaluated under different data regimes on the FFHQ dataset. Obviously, the FD scores can be drastically improved by synthesizing more data regardless of different extractors until sufficient samples  $(\sim 100K)$  are used, whereas the CKA scores are stable under different data regimes. Moreover, CKA could measure the distributional distances precisely with only  $5K$  synthesized samples, suggesting significant sample efficiency. Such observations demonstrate CKA's impressive adaptability toward the amount of synthesized data.
Developing a reliable and comprehensive measurement system for synthesis evaluation. Overall, a set of feature extractors that 1) are robust to the histogram matching attack, 2) capture sufficient semantics, and 3) define distinctive representation spaces could serve as reliable extractors for synthesis comparison. Together with a bounded distance (i.e., CKA) that is comparable across various representation spaces and hierarchical layers, as well as enjoys satisfactory sample efficiency. These two essential components constitute a reliable system to deliver the distributional discrepancy.
Figure 2: Fréchet Distance (FD) and Centered Kernel Alignment (CKA) scores evaluated under various data regimes on FFHQ dataset. FID scores are scaled for better visualization.  $\downarrow$  denotes the results fluctuate downward. The percentages represent the magnitude of the numerical variation. The curve of KID [4], Precision and Recall [49] can be found in Supplementary Material.
Table 5: Quantitative comparison results of Centered Kernel Alignment  $(\mathbf{CKA}_{\uparrow})$  on ImageNet dataset.  $^\dagger$  scores are quoted from the original paper and others are tested three times. To make our results more trivial to parse, we visualize the correlation between different metrics the human evaluation results in the Supplementary Material.
Model	FID†	ConvNeXt	RepVGG	SWAV	ViT	MoCo-ViT	CLIP-ViT	Overall	User study
ICGAN [7]	15.60	62.65	72.25	86.10	97.04	77.99	92.16	81.37	32%
ADM [14]	10.94	63.12	72.90	87.71	97.39	78.69	92.92	82.12	45%
BigGAN [5]	8.70	63.89	74.62	87.94	97.60	79.60	93.25	82.82	53%
C-ICGAN [7]	7.50	62.53	72.20	86.12	97.05	78.01	92.08	81.33	31%
BigGAN-deep [5]	6.95	64.97	76.45	88.31	97.77	80.27	94.11	83.65	55%
Guided-ADM [14]	4.59	65.99	78.99	89.44	98.13	80.46	94.96	84.66	57%
BigRoc [18]	3.69	67.86	79.48	89.93	98.23	82.25	96.07	85.64	65%
GigaGAN [27]	3.45	68.01	79.93	90.15	98.34	82.40	96.52	85.89	65%
DG-Diffusion [31]	3.18	68.22	80.06	90.56	98.46	82.51	96.88	86.12	66%
StyleGAN-XL [51]	2.30	68.75	80.28	91.54	98.52	82.64	97.41	86.52	67%
DiT [45]	2.27	68.94	80.65	91.03	99.05	82.90	97.08	86.61	67%
MDT [19]	1.79	69.64	81.68	91.78	99.43	83.43	98.19	87.36	69%
Table 6: Quantitative comparison results of Centered Kernel Alignment (CKA $\uparrow$ ) on FHHQ dataset. † scores are quoted from the original paper and others are tested three times.
Model	FID†	ConvNeXt	RepVGG	SWAV	ViT	MoCo-ViT	CLIP-ViT	Overall	User study
StyleGAN2 [29]	3.66	92.64	62.10	99.55	99.32	98.59	97.46	91.61	45%
Projected-GAN [50]	3.39	92.34	61.62	99.37	98.99	98.81	97.31	91.41	39%
InsGen [66]	3.31	94.17	65.72	99.60	99.36	98.90	97.75	92.58	58%
EqGAN [62]	2.89	94.54	64.36	99.69	99.49	99.10	98.61	92.63	62%
StyleGAN-XL [51]	2.19	93.78	67.59	99.68	99.49	99.25	97.33	92.85	66%
4 Benchmark Existing Generative Models
Based on the findings regarding the feature extractors and distributional distances, we construct a new synthesis valuation system. Concretely, our system leverages a set of models with both CNN and ViT architectures as feature extractors, namely, CNN-based ConvNeXt [40], RepVGG [15], SWAV [6] and ViT-based ViT [16], MoCo-ViT [9], CLIP-ViT [46], with which more comprehensive evaluation could be accomplished. Further, Centered Kernel Alignment (CKA) serves as the similarity indicator.
Accordingly, in this part we re-evaluate and compare the progress of existing generative models with our measurement system. Concretely, the latest generative models are re-evaluated with our system in Sec. 4.1, followed by our discussion about the diffusion models and GANs in Sec. 4.2. Notably, user studies are conducted for investigating the correlation between our system and human judgment.
4.1 Comparison on Existing Generative models
In order to investigate the actual improvement of existing generative models, we collect multiple publicly available generators trained on several popular benchmarks (i.e., FFHQ, LSUN Church, and ImageNet) for comparison. Benefiting from the impressive sample efficiency of CKA, we generate  $50K$  images as the synthesized distribution and use the whole datasets as the real distribution. Results from various selected extractors and their averaged scores are reported for a thorough comparison.
Our system can measure the synthesis quality in a consistent and reliable way. Tab. 5, Tab. 7, and Tab. 6 respectively demonstrate the quantitative results of different generative models on the ImageNet, LSUN Church, and FFHQ datasets. In most cases, CKA scores from various extractors and the overall scores provide a consistent ranking with FID scores, as well as agree with human perceptual judgment. These results suggest that our new metric could precisely measure the synthesis quality. However, for the Projected-GAN [50] and StyleGAN2 [29] (resp., Aided-GAN [33]) evaluated on FFHQ (resp., LSUN Church) dataset in Tab. 6 (resp., Tab. 7), our evaluation system gives the opposite ranking to the FID. Namely, the quantitative results of StyleGAN2 (resp., Aided-GAN) are determined better than that of Projected-GAN under our evaluation, whereas the FID scores vote Projected-GAN for the better one. Additionally, the performances of ICGAN [7] and class-conditional ICGAN on ImageNet in Tab. 5 are identified basically the same by our metric, while FID scores indicate that the class-conditional one significantly surpasses the unconditional one.
Table 7: Quantitative comparison results of Centered Kernel Alignment (CKA $_{\uparrow}$ ) on LSUN Church dataset. † scores are quoted from the original paper and others are tested three times.
Model	FID†	ConvNeXt	RepVGG	SWAV	ViT	MoCo-ViT	CLIP-ViT	Overall	User study
StyleGAN2 [29]	3.86	96.99	67.43	98.70	97.68	89.18	76.25	87.71	69%
Diffusion-GAN [64]	3.17	97.15	69.28	99.22	97.70	90.25	79.06	88.78	71%
EqGAN [66]	3.02	97.34	71.12	99.36	98.09	90.64	80.26	89.47	73%
Aided-GAN [33]	1.72	97.91	75.63	99.42	99.62	92.56	81.69	91.14	77%
Projected-GAN [50]	1.59	96.89	72.91	97.98	98.09	91.43	78.63	89.34	72%
Table 8: Fine-grained investigation of human judgment. Percentages indicate the ratio of generated images that are considered to be more plausible.
Dataset	FFHQ	LSUN Church	ImageNet
Model	Projected-GAN	StyleGAN2	Projected-GAN	Aided-GAN	ICGAN	Conditional-ICGAN
User Preference	45%	55%	43%	57%	50%	50%
In order to compare the performance of these models in a fine-grained way, we further perform paired-wise human evaluation. Specifically, groups of paired images synthesized by different models (e.g., StyleGAN2 and Projected-GAN on FFHQ dataset) are randomly picked for visual comparison. Then, presented with two sets of images produced by different models, users are asked to determine which set of images is more plausible.
Our measurement system provides the right rankings and better correlations with human visual judgment. Tab. 8 presents the quantitative results of human visual comparison. Observably, the synthesis quality of StyleGAN (resp. Aided-GAN) is more preferred by human visual judgment. That is, our measurement system produces the same rankings as the human perceptual evaluation, demonstrating the reliability of our metric. Moreover, our metric's indication that there's no significant gap between ICGAN and class-conditional ICGAN is also verified by the human evaluation. Considering the perceptual null space of Inception-V3, one possible reason for the FID performance gains of Projected-GAN might be the usage of pre-trained models, which is also identified by [35]. By contrast, our measurement produces the right rankings and agrees well with human evaluation, reflecting the actual synthesis quality in a comprehensive and reliable way.
4.2 Comparison between GANs and Diffusion Models
Diffusion models [23, 54, 55, 14, 45, 48, 19, 70, 42, 24] have demonstrated significant advancements in visual synthesis and became the new trend of generative models recently. Benefiting from the reliability of our new measurement system, here we perform a comprehensive comparison between GANs and diffusion models. Specifically, we report the FID and the overall CKA scores, as well as human judgment for quantitative comparison. Additionally, the model parameters and the synthesis speed (tested on an A100 GPU) are also included.
Tab. 9 presents the quantitative results. Obviously, diffusion model (i.e., DiT) obtains comparable results with GAN (i.e., StyleGAN-XL), yet with much more parameters (i.e.,  $675M$  vs.  $166.3M$ ). Moreover, diffusion models usually require extra inference time to obtain realistic images. Such comparisons reveal that GANs achieve better trade-offs between efficiency and synthesis quality, and designing computation-efficient diffusion models is essential for the community.
4.3 Comparing the performance of image-to-image translation
In order to testify the compatibility of our metric, here we employ our measurement system to evaluate the performance of image-to-image translation. We collect publicly available image-to-image translation models that are officially released to translate images from one domain to another domain for evaluation. Specifically, three translation benchmarks are involved here, namely Horse-to-Zebra [57, 71, 43], Cat-to-Dog [68, 43], and Dog-to-Cat [68, 25]. For each benchmark, we translate the tested images to the target domain following the original experimental settings. Then we compute the distributional discrepancies between the translated images and the real target images. Tab. 10 presents the quantitative results of the evaluated three image-to-image translation benchmarks. It can
Table 9: Quantitative comparison results of diffusion models and GANs on ImageNet dataset.  $\dagger$  scores are quoted from the original paper and  $\ddagger$  results are overall scores measured by our system.
Model	FID†	CKA‡	User	#Params	Sec/Kimg (s)
BigGAN	8.70	82.82	53%	158.3 M	33.6
BigGAN-deep	6.95	83.65	55%	85 M	27.6
StyleGAN-XL	2.30	86.52	67%	166.3 M	64.8
ADM	10.94	82.12	45%	500 M	17274
Guided-ADM	4.59	84.66	57%	554 M	17671
DiT	2.27	86.61	67%	675 M	3736.8
be seen from these results that CKA provides consistent ranks with FID among various extractors, and the averaged score can reflect the performance of different image translation models. For instance, the performance of CUT [43] on Horse-to-Zebra is identified better than that of CycleGAN [71] by both FID and our proposed metric. And the qualitative results in the original paper of CUT [43] also suggest that the performance of CUT surpasses CycleGAN. That is, our measurement system can provide a reliable evaluation under such settings. This indicates that our measurement system can also be used for evaluating the performance of image translation tasks.
Table 10: Quantitative comparison results of Centered Kernel Alignment  $(\mathbf{CKA}_{\uparrow})$  on Image-to-Image translation tasks.
Horse-to-Zebra dataset
Model	FID	ConvNeXt	RepVGG	SWAV	ViT	MoCo-ViT	CLIP-ViT	Overall
CycleGAN [71]	83.32	73.55	88.67	85.82	83.96	74.72	73.74	80.08
AttentionGAN [57]	76.05	75.59	91.73	86.37	85.16	76.65	75.49	81.83
CUT [43]	51.29	78.48	93.22	88.83	87.84	78.75	77.36	84.08
Cat-to-Dog
Model	FID	ConvNeXt	RepVGG	SWAV	ViT	MoCo-ViT	CLIP-ViT	Overall
CUT [43]	74.95	84.93	78.75	88.83	84.31	93.56	70.91	83.55
GP-UNIT [68]	60.96	90.45	87.79	94.05	90.12	95.91	75.32	88.94
Cat-to-Dog
Model	FID	ConvNeXt	RepVGG	SWAV	ViT	MoCo-ViT	CLIP-ViT	Overall
GP-UNIT [68]	31.66	79.58	78.18	96.79	86.93	93.92	77.42	85.47
MUNIT [25]	18.88	84.87	84.11	98.51	88.11	95.95	86.10	89.61
This work revisits the evaluation of generative models from the perspectives of the feature extractor and the distributional distance. Through extensive investigation regarding the potential contribution of various feature extractors and distributional distances, we identify the impacts of several potential factors that contribute to the final similarity index. With these findings, we construct a new measurement system that provides a more comprehensive and holistic comparison for synthesis evaluation. Importantly, our system could present more consistent measurements with human judgment, enabling more reliable evaluation.
Discussion. Despite a comprehensive investigation, our study could still be extended in several aspects. For instance, the impacts of different low-level image processing techniques (e.g., resizing) could be identified since they also play an important role in synthesis evaluation [44]. Besides, comparing datasets with various resolutions could be further studied. Nonetheless, our study could be considered an empirical revisiting towards the paradigm of evaluating generative models. We hope this work could inspire more fascinating works of synthesis evaluation and provide potential insight to develop more comprehensive evaluation protocols. We will also conduct more investigation on the unexplored factors and compare more generative models with our system.
[1] M. Alfarra, J. C. Pérez, A. Frühstück, P. H. Torr, P. Wonka, and B. Ghanem. On the robustness of quality measures for gans. Eur. Conf. Comput. Vis., pages 18-33, 2022.
This Supplementary Material is organized as follows: appendix B provides the implementation details of our experiments, appendix C demonstrates how human visual judgment is performed and presents the interface of our user study. appendix D presents additional quantitative and qualitative results, including 1) comparison results with more metrics (KID, Precision and Recall), 2) more results of MLP-based extractors, 3) the similarities between various representation spaces, as well as 4) more hierarchical results from different semantic levels of various extractors. Finally, in appendix E, we provide 2D plots of the correlation between different metrics with human visual judgment to make our results easier to parse.
FFHQ [58] contains unique 70,000 human-face images with large variations in terms of age, ethnicity, and facial expressions. We employ the resolution of  $256 \times 256 \times 3$  for our experiments.
ImageNet [13] includes 1,280,000 images with 1,000 classes of different objects such as goldfish, bow tie, etc. All experiments on ImageNet are performed with the resolution of  $256\times 256\times 3$  unless otherwise specified.
LSUN Church [69] consists of 126, 227 images of the church, varies in the background, perspectives, etc. We employ the resolution of  $256 \times 256 \times 3$  for our experiments.
B.2 Experimental Settings and Hyperparameters
Kernel selection. We consistently employ the RBF kernel
$$
K (\mathbf {x _ {i}}, \mathbf {x _ {j}}) = \exp (- \frac {\| \mathbf {x _ {i}} - \mathbf {x _ {j}} \| ^ {2}}{2 \sigma^ {2}})
$$
for calculating the CKA. The bandwidth  $\sigma$  is set as a fraction of the median distance between examples. In practice, three commonly used kernels could be employed for calculation, namely linear, polynomial, and RBF kernels. In order to investigate their difference, three publicly available models with clear performance margins are collected for evaluation. Concretely, we gather models of InsGen [66] trained on FFHQ with different data regimes (i.e.,  $2K$ ,  $10K$ ,  $140K$ ), the ranking of their synthesis quality is clear and reasonable.
Tab. A1 demonstrates the quantitative results of CKA with different kernels. Obviously, these kernels give similar results and rankings. However, the RBF kernel contributes to the distinguishability of quantitative results, making the results more comparable. Consequently, the RBF kernel is employed in our experiments.
Table A1: CKA results with different kernels. The publicly available models are gathered for comparison. † results are quoted from the original paper.
Kernel	InsGen-2k	InsGen-10k	InsGen-140k
FID†	11.92	4.90	3.31
Linear	99.83	99.93	99.98
Poly	99.58	99.87	99.92
RBF	95.72	98.65	99.10
ViT features calculation. The feature maps of ViT-based extractors are three-dimensional tensors (N, W, C), where W contains the global token and local features. The global token captures the same semantic information as the local features. Thus the global taken features are used for computation in implementation. Tab. A2 shows the comparison results of using local features and the global token. Consistently, they give similar results and rankings, so we use the global token for calculation in our experiments.
Feature normalization. In practice, the activations of features play an essential role in computing the similarity index. Namely, the quantitative results would be dominated by a few activations with large
Table A2: CKA results with different features for calculation.
Metrics	InsGen-2k	InsGen-10k	InsGen-140k
Local Features	96.62	97.42	97.38
Global Token	97.46	97.88	97.93
Table A3: CKA scores with different normalization techniques.
Metrics	InsGen-2k	InsGen-10k	InsGen-140k
CKANo	97.46	97.88	97.93
CKAL1	96.62	98.91	99.33
CKAL2	96.62	98.91	99.32
CKASoftmax	95.72	98.65	99.10
peaks, neglecting other correlation patterns [63]. To investigate the activations of our self-supervised extractors, we visualize the activations of different samples and their statistics.
Fig. A1 and Fig. A2 respectively illustrate the activation of different samples and their statistics. Obviously, there are several peaks in the activations. And these peaks may dominate the similarity index as they are substantially larger than other activations. To mitigate the peaks and create a more uniform distribution, we employ the softmax transformation [63] to normalize the features. Such operation smooths the activations while maintaining the original distributional information of features. Thus the similarity index remains consistent to deliver the distribution discrepancy. Besides the softmax transformation, we also compare the behavior of different normalization techniques (i.e.,  $L_{1}$  and  $L_{2}$  normalization).
Tab. A3 demonstrate the quantitative results with different normalization techniques. They consistently provide similar results and rankings, and the softmax transformation ameliorates the peaks more significantly, providing more comparable results. Consequently, we adopt Softmax normalization in our experiments.
Histogram matching. In order to investigate the robustness of the measurement system, we employ the histogram matching [35] to attack the system. To be specific, a subset with a considerable number  $(e.g., 50K)$  of images is chosen as the referenced distribution, and the corresponding class distribution is predicted by a given classifier (i.e., Inception-V3 [56]). With the guidance of the classifier, the generator is encouraged to produce a synthesis distribution that matches the predicted class distribution of real images. Recall that the generator used to produce these synthesized distributions stays unchanged, thus a robust measurement system should give consistent similarities between the randomly generated and the matched distribution.
Fig. A3 provides the class distribution of real and synthesized FFHQ images predicted by Inception-V3. Obviously, the class distribution of the matched distribution is well-aligned with the predicted real distribution.
Sample-efficiency. In order to investigate the impacts of the number of synthesized samples, we compute the distributional distances between the real distribution with synthesized distributions with various numbers of generated images. Concretely, FFHQ (with  $70K$  images) and ImageNet (with 1.28 million images) are investigated for universal conclusions. For both datasets, we synthesis  $500K$  images as candidate, and randomly choose  $5K$ ,  $10K$ ,  $50K$ ,  $100K$ ,  $250K$ , and  $500K$  images as the synthesized distribution for computing the metrics. The entire training data is utilized as the real distribution, and the publicly accessible models on  $\mathrm{FFHQ}^3$  and  $\mathrm{ImageNet}^4$  are employed.
The curve of FD and CKA under various data regimes on ImageNet dataset is shown in Fig. A4. Consistent with the aforementioned results in the main paper, CKA could measure the distributional distances precisely with only  $5K$  samples, whereas FID fails to deliver the actual measurement until sufficient samples are used. That is, CKA could give reliable results even when limited data is given, suggesting impressive sample efficiency. Equipped with the bounded quantitative results and consistency under different data regimes, as well as the robustness to the histogram matching attack, CKA outperforms FID as a reliable distance for delivering the distributional discrepancy.
<sup>3</sup>https://github.com/NVlabs/stylegan3
<sup>4</sup>https://github.com/autonomousvision/stylegan-xl
Figure A1: Visualization of different samples' activations. The large peaks may dominate the similarity index as their numerical values substantially surpass smaller values.