SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mengping Yang $^{1,\dagger}$  Ceyuan Yang $^{1,\dagger}$  Yichi Zhang $^{1}$  Qingyan Bai $^{3}$  Yujun Shen $^{2}$  Bo Dai $^{1}$
$^{1}$ Shanghai AI Laboratory  $^{2}$ Ant Group  $^{3}$ Tsinghua University
A good metric, which promises a reliable comparison between solutions, is essential for any well-defined task. Unlike most vision tasks that have per-sample ground-truth, image synthesis tasks target generating unseen data and hence are usually evaluated through a distributional distance between one set of real samples and another set of generated samples. This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set. Extensive experiments conducted on multiple datasets and settings reveal several important findings. Firstly, a group of models that include both CNN-based and ViT-based architectures serve as reliable and robust feature extractors for measurement evaluation. Secondly, Centered Kernel Alignment (CKA) provides a better comparison across various extractors and hierarchical layers in one model. Finally, CKA is more sample-efficient and enjoys better agreement with human judgment in characterizing the similarity between two internal data correlations. These findings contribute to the development of a new measurement system, which enables a consistent and reliable re-evaluation of current state-of-the-art generative models.
Through reproducing realistic data distribution, generative models [67, 2, 19, 38, 28, 52] have enabled thrilling opportunities to go beyond existing observations via content recreation. Their technical breakthroughs in recent years also directly lead to the blooming of metaverse, AI Generated Content (AIGC), and various other downstream applications. However, accurately measuring the progress and performance of generative models poses significant challenges, as it requires a consistent and comprehensive evaluation of the divergence between real and synthesized data distributions. Among existing evaluation metrics [41, 4, 34, 44], Fréchet Inception Distance (FID) [22] is the most popular evaluation paradigm for synthesis comparison. Despite its popularity, recent studies [35, 44, 36, 1, 26] have identified several flaws in the FID metric that may miscalcute the actual improvements of generative models. Consequently, there is a pressing need for a more systematic and thorough investigation in order to provide a more accurate assessment of synthesis performance.
Therefore, this paper presents an empirical study that rigorously revisits the consistency and comprehensiveness of evaluation paradigms for generative models. Commonly used paradigms including FID typically contain two key components: a feature extractor  $\phi(\cdot)$  and a distributional distance  $d(\cdot)$ . Through  $\phi(\cdot)$ , i.e., Inception-V3 [56], a number of real samples  $(x \in X)$  and synthesized
$\dagger$  denotes equal contribution.
$^{1}$ code is available at https://github.com/kobeshegu/Synthesis-Measurement-CKA.
ones  $(y\in \mathrm{Y})$  are projected into a pre-defined representation space to approximate the respective data distributions.  $d(\cdot)$ , i.e., Fréchet Distance [17] is then calculated in the space to deliver the similarity index, indicating the synthesis quality. Following the philosophy, our study revolves around the representation space defined by the feature extractor  $\phi (\cdot)$ , and the distributional distance  $d(\cdot)$ .
The most commonly used feature extractor, i.e., Inception-V3, has been found to encode limited semantics and possesses a large perceptual null representation space [35], making it hardly reflect the actual improvement of synthesis. Accordingly, we gather multiple models that vary in supervision signals, architectures, and representation similarities to investigate the impact of the feature extractor  $\phi(\cdot)$ , which are respectively motivated by 1) representation spaces defined by the extractor  $\phi(\cdot)$  usually encode different levels of semantics, understanding which extractor or set of extractors can capture rich semantics is crucial yet less explored; 2) and it remains uncertain how the correlations between various representation spaces affect the evaluation results. In addition to studying the feature extractor  $\phi(\cdot)$ , we further delve into the consistency across spaces, the choice of distances, and the number of evaluated samples for the distributional distance  $d(\cdot)$ . It is imperative that the distributional distance consistently provides a reliable similarity index, even when measured in various representation spaces. Similarly, selecting an appropriate number of samples to represent the synthesis distribution is a critical consideration. Regarding the choice of  $d(\cdot)$ , besides Fréchet Distance, we incorporate Centered Kernel Alignment (CKA) [11, 10] to explore alternatives for a more accurate evaluation.
In order to qualitatively and quantitatively compare different choices of the aforementioned aspects of  $\phi(\cdot)$  and  $d(\cdot)$ , we re-implement the visualization pipeline and the histogram matching technique as in [35]. These techniques allow us to respectively highlight the most relevant semantics of an image w.r.t the similarity indexes and to attack the measurement system through a selected subset. Moreover, we conduct an extensive user study involving 100 participants to investigate the correlation between the synthesis measurement and human judgment. Through these analysis tools, we make several significant findings: 1) One specific extractor (e.g., Inception-V3) tends to capture limited semantics and provide unreliable measurement results. 2) Various extractors naturally focus on different aspects of semantics thus demonstrating the potential generalization across different domains, motivating us to incorporate multiple extractors to deliver a comprehensive and reliable measurement. 3) With respect to features obtained from multiple representation spaces defined by different extractors, CKA proves to be effective in measuring the discrepancy and produces bounded values that facilitate comparison across different spaces. 4) In conjunction with the extensive user study, CKA consistently agrees with human judgment whereas FID failed in some circumstances, further underscoring the advantages of using CKA as the evaluation metric.
After revisiting various factors, we leverage the newly developed measurement system to re-evaluate extensive generative models under various settings. Current state-of-the-art generative models on several domains are first benchmarked through our system. Moreover, the performances and intrinsic properties of diffusion models and GANs are comprehensively compared with the new measurement system. Furthermore, the measurement system is employed to evaluate the performance of image-to-image translation models. It turns out that our system not only delivers a similar assessment with FID and human evaluation in most cases, but also demonstrates a more reliable and consistent correlation with human judgment than FID, demonstrating the robustness and superiority of the proposed metric.
This section briefly introduces the feature extractor  $\phi(\cdot)$ , distributional distance  $d(\cdot)$ , evaluated datasets and generative models, as well as auxiliary analysis approaches used in our study.
2.1 Representation Spaces
To investigate the effect of feature extractors  $\phi(\cdot)$ , we gather multiple models that are pre-trained on different objectives in fully-supervised/self-supervised manners, and with various architectures (e.g., ViT, CNN, and MLP).
Supervision. Models that are trained in fully-supervised and self-supervised manners are collected due to their potential for generalization. In particular, we include the backbone networks which are
well-trained on supervised ImageNet classification tasks [13, 21]. Further, we gather the weights derived from single-modal/multi-modal self-supervised learning approaches [46, 8, 9, 6].
Architecture. In addition to considering models trained with different supervisions, we also include models with various architectures. Concretely, models with CNNs [56, 53, 21, 6, 8], ViTs [9, 39, 46], as well as MLPs [61] architectures are gathered together for our investigation.
Properties	Feature Extractors
Supervision Architecture	Fully-supervised [56, 21], Self-supervised [46, 8, 9, 6] CNNs [21, 46, 8], ViTs [39, 46, 9], MLPs [61, 59, 37]
2.2 Distributional Distances
Frechet Inception Distance (FID) computes the Fréchet Distance [17] between two estimated Gaussian distributions, i.e.,  $\mathcal{N}(\mu_s,\Sigma_s)$  and  $\mathcal{N}(\mu_r,\Sigma_r)$ , which represent the feature distributions of synthesized and real images extracted by the pre-trained Inception-V3. Formally, Fréchet Distance (FD) is calculated by
$$
\mathrm {F D} (\mathrm {X}, \mathrm {Y}) = \| \mu_ {s} - \mu_ {r} \| ^ {2} + \operatorname {T r} \left(\Sigma_ {s} + \Sigma_ {r} - 2 \left(\Sigma_ {s} \Sigma_ {r}\right) ^ {\frac {1}{2}}\right), \tag {1}
$$
where  $\mathrm{X}$  and  $\mathrm{Y}$  represent the real distribution and synthesized distribution, respectively.  $\mu$  and  $\Sigma$  correspond to the mean and variance of Gaussian distribution, and  $\operatorname{Tr}(\cdot)$  is the trace operation.
Centered Kernel Alignment (CKA) as a widely used similarity index for quantifying neural network representations [11, 32, 12], could also serve as a metric of similarity between two given data distributions. CKA has been identified to have several advantages: 1) CKA is invariant to orthogonal transformation and isotropic scaling, making it stable under various image transformations; 2) CKA can capture the non-linear correspondence between representations benefit from its kernel mapping; and 3) CKA can determine the correspondence across different features and with different widths, whereas previous metrics fail [32].
Formally, CKA is normalized from Hilbert-Schmidt Independence Criterion (HSIC) [20] to ensure invariant to isotropic scaling and is calculated by
$$
\operatorname {C K A} (\mathrm {X}, \mathrm {Y}) = \frac {\operatorname {H S I C} (x , y)}{\sqrt {\operatorname {H S I C} (x , x) \operatorname {H S I C} (y , y)}}. \tag {2}
$$
Here, HSIC determines whether two distributions are independent. Formally, let  $K_{ij} = k(\mathrm{x}_i,\mathrm{x}_j)$  and  $L_{ij} = l(\mathrm{y}_i,\mathrm{y}_j)$ , where  $k$  and  $l$  are two kernels. HSIC is defined as
$$
\operatorname {H S I C} (K, L) = \frac {1}{(n - 1) ^ {2}} \operatorname {T r} (K H L H), \tag {3}
$$
where  $H$  denotes the centering matrix (i.e.,  $H_{n} = I_{n} - \frac{1}{n}\mathbf{11}^{T}$ ). For kernel selections of  $k$  and  $l$ , we find that different kernels (RBF, polynomial, and linear) give similar results and rankings, and the RBF kernel contributes to the distinguishability of quantitative results. Therefore, RBF kernel is used for all experiments, and the bandwidth is set as a fraction of the median distance between examples [32]. These metrics are compared in a consistent setting for fair comparison, more implementation details and comparisons on CKA are given in Supplementary Material.
2.3 Benchmarks and Analysis Approaches
Benchmarks. In order to analyze various factors of measurement, we also collect multiple generators to produce synthesized images. To be more specific, we employ state-of-the-art generative models trained on various datasets for comparison in Tab. 1. We download corresponding publicly available models for comparison. Unless otherwise specified, all of these models are compared in a consistent setting.
Visualization tool. To qualitatively compare where these feature extractors "focus on", we employ the visualization technique of [35] to localize the regions that contribute the most to the similarity index. The highlighted regions reveal the most relevant parts of an image regarding the measurement results. Accordingly, larger highlighted regions indicate that more visual semantics are involved
Table 1: Generative models used in our study. Publicly available models are gathered for evaluation.
Method	Year	Training Datasets
StyleGAN2 [29]	2020	FFHQ, LSUN Church
BigGAN, BigGAN-deep [5]	2019	ImageNet
ADM, ADM-G [14]	2021	ImageNet
Projected-GAN [50]	2021	FFHQ, LSUN Church
InsGen [66]	2021	FFHQ
StyleGAN-XL [66]	2022	FFHQ, ImageNet
Aided-GAN [33]	2022	LSUN Church
EqGAN [62]	2022	FFHQ, LSUN Church
Diffusion-GAN [64]	2022	LSUN Church
DiT [45], BigRoC [18]	2022	ImageNet
GigaGAN [27], DG-Diffusion [31], MDT [19]	2023	ImageNet
in the evaluation. Note that generating a realistic image requires all parts, even each pixel, to be well-synthesized. Thus, a metric that focuses on more visual regions appears to be more dependable.
Histogram matching attack. We employ the histogram matching [35] to attack the system to investigate the robustness of the measurement results. Concretely, a subset is selected from a superset by matching the class distribution of the synthesized set with that of the real set. As pointed out by [35], the synthesis performance could be substantially improved using the chosen subset. We thus prepare two distinct sets of synthesized images. One reference set ("Random") is produced by generating images randomly, and the other set ("Chosen  $I$ ) is carefully curated by matching the class distribution histogram of the supervised Inception-V3 [56]. The matched histograms are available in Supplementary Material. Since the generator and real data remain unaltered, the evaluation should keep consistent, and any performance gains directly indicate the unreliability of a given extractor.
Human judgment. In order to examine the correlation between the evaluation system and human perceptual judgment, we conduct extensive user studies employing two strategies. Firstly, to benchmark the synthesis quality of various generative models, we prepare a substantial number of randomly generated images (i.e.,  $5K$ ), and ask 100 individuals to assess the photorealism of these images. The final scores are averaged across all 100 participants. Secondly, to qualitatively compare two paired generative models with similar quantitative performances (e.g., Projected-GAN [50] and Aided-GAN [33] on LSUN Church dataset in Sec. 4.1), we prepare groups of paired images generated by different models and ask 100 individuals to assess the perceptual quality of paired images. More details of our user studies can be found in Supplementary Material. In this way, we could obtain reliable and consistent human judgments, facilitating better investigation with the evaluation system.
3 Analysis on Representation Spaces and Distributional Distances
In this section, we investigate the potential impacts of the representation space and distributional distances with respect to the final similarity index. Concretely, Sec. 3.1 presents the study of extractors that define the representation spaces, followed by the analysis of distributional distances in Sec. 3.2.
3.1 Representation Spaces
Prior works [35, 36, 41, 26] have demonstrated that the most commonly used feature extractor i.e., Inception-V3 [56], could hardly reflect the exact improvement of synthesis due to its limited consideration of semantics. We thus conduct a comprehensive study by incorporating various models that differ in supervision signals and architectures, to identify which or which set of extractors serve as reliable feature extractors for synthesis comparison.
Distinct feature extractors with different architectures yield varying semantic areas of focus. Fig. 1 shows the highlighted regions that contribute most significantly to the measurement results. Obviously, CNN-based extractors consistently emphasize concentrated regions with or without manual labels for pre-training, including limited semantics. Specifically, CNN-based extractors remain to highlight objects (e.g., microphone, hat, and sunglasses), rather than the main focus of the evaluation domains (i.e., Human Faces here). In contrast, ViT-based extractors capture larger regions that encompass more synthesis details and semantics. This observation aligns with the finding that ViTs possess a global and expansive receptive field compared to the local receptive field of
Figure 1: Heatmaps from extractors with various architectures. CNN-based extractors (i.e., Inception [56], ConvNeXt [40], SWAV [6], MoCo-R [9], CLIP-R [46], and RepVGG [15]) focus on objects whereas ViT-based (i.e., Swin-Transformer [39], ViT [16], DeiT [60], CLIP-V [46], and MoCo-V [9]) and MLP-based (i.e., ResMLP [61]) ones pour attention on wider areas.
Table 2: Quantitative comparison results of Fréchet Distance  $(\mathbf{FD}_{\downarrow})$  on FFHQ dataset. "Random, Chosen  $I$  " respectively represent the synthesized distribution of randomly generated and matching the class prediction of Inception-V3. Moreover, "-R" and "-V" respectively denote the architecture of ResNet and ViT.  $(\downarrow)$  indicates the results are hacked by the histogram matching mechanism. Notably, the values across different rows are not comparable and we report the stds of three testings to better illustrate the numerical fluctuation of various extractors towards the histogram attack. The results of Fréchet Distance (FD) on ImageNet can be found in Supplementary Material.
Model	Inception	ConvNeXt	SWAV	MoCo-R	RepVGG	CLIP-R
Random	2.81±0.01	78.03±0.10	0.13±0.002	0.24±0.003	129.61±0.41	10.34±0.06
ChosenI	2.65±0.01↓	78.19±0.11	0.13±0.002	0.24±0.003	129.67±0.39	10.36±0.08
Model	Swin	ViT	DeiT	CLIP-V	MoCo-V	ResMLP
Random	142.87±0.12	15.11±0.09	437.80±0.14	1.06±0.01	7.32±0.03	99.11±0.06
ChosenI	140.01±0.12↓	15.11±0.10	430.81±0.16↓	1.06±0.01	7.40±0.03	95.36±0.06↓
CNN-based extractors [47, 65, 16]. Furthermore, the ViT-based and CNN-based extractors appear to complement each other, with the former capturing broader regions and the latter focusing on specific objects with high density.
Multiple extractors incorporate more visual semantics in a complementary manner. When reproducing the whole data distribution, generative models are required to synthesize not only the main objects but also the background, texture, and intricate details. Similarly, the extractors should strive to capture more regions of given images to approximate the data distribution, enabling better visual perception. However, the above observation regarding the heatmaps of different extractors reveals that each individual extractor could only capture partial semantics of the entire image for measurement, inadequately reflecting the overall synthesis performance. Consequently, various extractors with different architectures should be considered since they could involve more semantics, enhancing the reliability of the evaluation.
Extractors that are vulnerable to the histogram matching attack are not reliable for evaluation. Prior study [35] has highlighted that extractors focusing on limited semantics may be susceptible to the histogram matching attack, undermining the trustworthiness of the evaluation. Motivated by this, we investigate the robustness of the above extractors toward the attack as they attach different importance to the visual concepts. Concretely, we obtain the publicly available generator of StyleGAN $^2$  [30] and calculate the Fréchet Distance (FD) results on the FFHQ dataset. We compare two evaluated sets: one is randomly generated using the model, while the other is chosen by matching the predicted class distribution of Inception-V3 (the matched histograms are provided in Supplementary Material).
Tab. 2 shows the quantitative results. Comparing the performances of the random set and chosen set, we could tell that certain extractors, regardless of their architectures (e.g., CNN-based Inception-V3 [56], ViT-based Swin-Transformer [39], and MLP-based ResMLP [61]), are susceptible to the histogram matching attack. For instance, the FD score of Inception shows an improvement of  $5.7\%$  when the chosen set is used, and ResMLP exhibits a  $3.8\%$  improvement (More quantitative
$^{2}$ https://github.com/NVlabs/stylegan3
and qualitative results of MLP-based extractors gMLP [37] and MLP-mixer [59] are shown in Supplementary Material). Namely, the FD score could be improved without making any changes to the generators, aligning with the observations of [35]. We thus filter out the extractors that are vulnerable to the attack since their rankings can be manipulated without actual improvement to the generative models.
Extractors that define similar representation spaces are redundant. So far, we have demonstrated the importance of considering multiple extractors for a more comprehensive evaluation and filtered out the extractors that are susceptible to the histogram attack. However, it is also crucial to avoid redundancy among the remaining extractors, as they may define similar representation spaces. To address this, we examine the correlation between representation spaces across different feature extractors, following the approach outlined in [32]. Specifically, a significant number of images (10K images from ImageNet) are fed into these extractors to compute their correspondence. After calculating the similarity matrix, we can further filter out extractors that define homogeneous representation spaces (the similarity analysis is presented in Supplementary Material). The remaining extractors are presented in the table below. These extractors 1) capture rich semantics in a
CNN-based	ConvNeXt [40], SWAV [6], RepVGG [15]
ViT-based	CLIP-ViT [46], MoCo-ViT [9], ViT [16]
complementary way, 2) are robust toward the histogram matching attack, and 3) define meaningful and distinctive representation spaces. Besides, both CNN-based and ViT-based extractors are considered and all of them have demonstrated strong performance for the pre-defined and downstream tasks, facilitating more comprehensive and reliable evaluation. Notably, the selection of self-supervised extractors SWAV, CLIP-ViT, and MoCo-ViT agrees with previous studies [41, 35, 3].
The selected extractors can serve as reliable tools for synthesis evaluation. In order to investigate the reliability of the selected extractors, we employ them to test the synthesis quality of representative generative models on the ImageNet dataset. Tab. 3 presents the quantitative FD scores of various extractors. Although their results differ in numerical scales, they consistently indicate that StyleGAN-XL outperforms both BigGAN and BigGAN-deep by a significant margin, and BigGAN-deep performs slightly better than BigGAN. Such observation also agrees with the human judgment in Tab. 5. Overall, the consistent trends observed across different extractors and the alignment with human judgment confirm that the selected extractors are reliable for comparing the synthesis quality.
3.2 Distributional Distances
After the study of feature extractors, we shift our focus to another essential component of measurement, i.e., the distributional distance  $d(\cdot)$ . Besides Fréchet Distance (FD), we also incorporate Centered Kernel Alignment (CKA) for our investigation.
CKA provides normalized distances w.r.t numerical scales in variable representation spaces. Tab. 5 demonstrates the quantitative results of Centered Kernel Alignment (CKA). Unlike the Fréchet Distance (FD) scores that exhibit significant fluctuations across various extractors, the CKA scores demonstrate remarkable stability when evaluated in different representation spaces. For instance, the FD score of BigGAN on MoCo-ViT is 238.78 whereas 3.35 on CLIP-ViT, making it challenging to combine the results of different extractors. By contrast, the stability of CKA scores allows the ability to combine results from multiple extractors (i.e., average) for better comparison.