SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Top: Fréchet distance, precision, and human error rate for each generative model as measured by different encoders (columns) on different datasets (rows). Marker styles denote different generative techniques. Panels with a shaded background do not have strong  $(|r| \geq 0.5)$  and significant  $(p \leq 0.05)$  correlations between FD and human error rate. Bottom: Pearson correlation of metrics over the three high-resolution datasets.
to the Inception model in [72] (in Appendix B.4.1 we also show poor alignment with a smaller CLIP-B/32 model investigated in a number of toy examples in [7]). CLIP VIT-L/14, DINOv2, and MAE display far greater alignment with the human experiment baseline, as does DreamSim, except on ImageNet – which we believe to be a particularly important dataset, as it is the most complex one being considered here, and it is thus used to train the most realistic models. Note that large precision values (which aims to quantify fidelity) do not consistently correspond to high human error rate, even for encoders whose FD strongly correlates with human evaluation: precision is thus likely measuring more than just fidelity, and FD should be preferred over it. We include analogous results for recall in Appendix D.1, showing that it does not only capture diversity.
We find that diffusion models are often driving the discrepancy in alignment with human evaluation for the Inception network: FID prefers GANs over diffusion while FD determined by self-supervised models trained on very large and diverse datasets does not. This discrepancy does not only occur on non-ImageNet benchmarks, as previous works have shown for GANs, but is also true on ImageNet.
5 Alternative explanations: diversity, rarity, and memorization
We have found that FID does not correlate with human error rate, and have shown that replacing the Inception-V3 network with an SSL encoder such as DINOv2-ViT-L/14 both recovers the correlation of FD with human error rate, and results in a metric which qualitatively focuses on the more relevant parts of images. While these are very promising characteristics for an evaluation metric, alternative explanations need to be ruled out before such a metric can be confidently adopted as a community standard. In this section we first verify that the lack of correlation between FID and human error rate is not due to models with large human error rates lacking diversity, nor to humans wrongly classifying rare real images as fake. This justifies our use of human error rate as "ground truth" for fidelity, and confirms that a lack of alignment with human error rate is a flaw of the encoder/metric pair. Then, we investigate whether the best performing generative models are memorizing their training data.
Figure 5: Left: Per-class Vendi scores of ImageNet models, in decreasing order of  $\mathrm{FD_{DINOv2}}$  score. Right:  $\mathrm{FD_{DINOv2}}$  on ImageNet, coloured by average per-class Vendi score (white corresponds to the train dataset).
5.1 Diversity and rarity
Diversity FD-based metrics combine both fidelity and diversity into a single score, whereas human evaluation focuses only on the former. We must thus independently measure model diversity in order to confirm whether the lack of strong correlation between FID and human evaluation is due to the FID score being flawed as an evaluation metric which focuses on fidelity, or if the discrepancy can simply be explained by high fidelity models having worse diversity. To decide between these two alternatives we explore the extent to which  $\mathrm{FD}_{\mathrm{DINOv2}}$  and FID align with diversity measures.
Our diversity analysis focuses on the Vendi score [33]. We justify this choice in Appendix D.4 and verify that the Vendi score meaningfully quantifies diversity locally, but the same is not true globally. For example, Figure 6 displays samples from a DiT model on ImageNet with and without strong classifier-free guidance (cfg=4 and 1.5, respectively; we refer to the former model as DiT-guided), where it is evident that DiT-guided exhibits much lower per-class semantic diversity. Yet, Appendix D.4 shows that the overall Vendi score is higher for DiT
Figure 6: DiT-guided and DiT samples, labelled with per-class Vendi scores using DINOv2.
guided for almost all choices of encoders, while per-class Vendi scores are consistently lower for DiT-guided across encoders. These results justify the use of the per-class Vendi score as a sensible diversity metric; the overall Vendi score is mostly measuring inter-class diversity – which is not particularly meaningful for class-conditional models such as the ones commonly used on ImageNet – whereas the per-class scores focus on intra-class diversity consistently across encoders, and thus provide a more meaningful quantification of semantic diversity.
Equipped with the per-class Vendi scores, we evaluate the diversity of ImageNet models using the DINOv2 encoder in Figure 5 (left), where we can see that differences in diversity do not explain discrepancies between FID and human evaluations. For example, GigaGAN has diversity scores which are much farther away from those of the training data than most diffusion models, yet achieves a better FID (Figure 2). We see this as strong evidence of a limitation of the use of the Inception network to measure fidelity with FID. We perform the same analysis using the FD metric with the DINOv2 encoder in Figure 5 (right): this evaluation metric is not only much more correlated to human evaluators, but diversity also better explains the few discrepancies between the two, e.g. the lack of diversity in DiT-guided results in a worse  $\mathrm{FD}_{\mathrm{DINOv2}}$  score than DiT despite having a better human error rate. Nonetheless, we highlight that the  $\mathrm{FD}_{\mathrm{DINOv2}}$  score emphasizes fidelity more than it does diversity (e.g. the DiT-guided model still obtains a very strong score).
Rarity To ensure that participants are not confusing "unrealism" with "un likeliness" and assessing rare images as fake – which would result in more diverse generative models ranking worse on human error rate – we investigated whether the human error rate on each real image (individual images were evaluated by an average of 13 humans) was correlated with the image's "rarity score" [42]. Experiments are detailed in Appendix D.4, which show that human evaluators are not confusing
"unrealism" with "unlikeliness", and thus human error rate is a sensible ground truth for image fidelity. Combined with the diversity analysis above, this rules out diversity as an alternative explanation for the lack of alignment between human assessment and FID, and proves that alignment of FD metrics with human error rate is a desirable property for the studied generative models.
5.2 Memorization
Recent works have shown that diffusion models are particularly prone to memorization issues, in which models memorize individual images from their training data and emit them at generation time. Memorized samples may be near-pixel-wise identical, or semantically equivalent to their source object while differing in terms of pixel-wise identity, the latter termed reconstructive memory [101]. Both predominantly occur either when the training set is small [13] or when there are a number of duplicate training samples [101]. While it has been shown that diffusion and GAN models can memorize CIFAR10 samples [101] – a dataset that contains many duplicates [6] – to our knowledge it is currently unknown whether this occurs on larger, more diverse, and higher resolution datasets such as ImageNet, FFHQ, or LSUN-Bedroom, and whether this affects any of the metrics we report. We set out to investigate this here.
Our experiments (refer to Appendix D.6) suggest that the larger datasets (e.g. ImageNet) are not memorized by even the largest models we considered. This indicates that the models that are measured as superior in DINoV2 space are not capitalizing on memorization of the training data.
Collecting memorized samples We perform a direct check for pixel-wise memorization for each of the 100k images from each of our 41 generative models using the calibrated  $l_{2}$  distance proposed in [13]. We find strong evidence on CIFAR10 that most generative models exhibit exact memory and showcase a set of memorized samples in Figure 7. We also report the memorization ratio of all models – the proportion of generated samples that match source samples in the training set – to illustrate degrees of exact memorization across different models. We found no conclusive evidence of pixel-wise memorization by any model on ImageNet, FFHQ, or LSUN-Bedroom, but found evidence of models exhibiting reconstructive memory on ImageNet and LSUN-Bedroom [101]. We find that less than  $0.5\%$  of DiT-XL-2 images exhibit close reconstructive memory. Although reconstructive memory on complex datasets is not necessarily a major concern, we recommend monitoring the memorization ratio, especially as models become more powerful and pixel-wise memorization becomes more likely. We include examples in Figure 7 (left) and additional visualizations and details in Appendix D.6. We note that our study was not explicitly designed to detect a more “copy-paste” approach wherein models memorize aspects of different training images and combine them when sampling, and thus we cannot rule out all forms of memorization. We leave such investigations for future work.
**Evaluating memorization metrics** We evaluate the main memorization metrics in the literature in light of the memorized CIFAR10 samples we uncovered: the percentage of authentic samples (AuthPct) [1], the  $C_T$  score [70], and the percentage of overfit Gaussians in FLS (FLS-POG) [53]. We first measure each metric's sensitivity to memorization in a controlled experiment, where we sample from the approximate posterior at different depths of a VDVAE's [21] hierarchical structure to generate a collection of synthetic datasets that serve as increasingly less faithful reconstructions of the training set. Appendix D.6.1 contains the full description and results, and establishes that AuthPct, the  $C_T$  score, and FLS-POG are sensitive to memorization in an ideal scenario where all samples become increasingly memorized. In Figure 7 (right) we investigate whether these automated metrics can differentiate models in practice, based on their measured CIFAR10 memorization ratios. Here we used the DINOV2 representation space, while results for other encoders are shown in Appendix D.6.2. For most encoders, the  $C_T$  score trends in the correct direction, whereas AuthPct and FLS-POG trend inconsistently and fail to differentiate between models with different numbers of memorized samples.
By swapping the training set with a test set that the model cannot possibly memorize, we check whether memorization metrics are sensitive to confounding properties other than memorization, such as fidelity or mode collapse. This experiment is not possible for FLS-POG as it does not use the training set, and for the  $C_T$  score we split the test set into two as it is already used. The results for  $C_T$  and AuthPct on test data are depicted with low opacity in Figure 7 (right). Surprisingly, in both cases the results against the test set closely follow those against the training set, meaning that the  $C_T$  score and AuthPct are dominated by some property other than memorization. Based on our analysis in Appendix D.6.3, we postulate that these metrics focus more on mode shrinkage and image fidelity
Figure 7: Left: Generated samples on top with matched training samples on the bottom showcasing exact memorization on CIFAR10 and reconstructive memorization on ImageNet and LSUN-Bedroom. Right: CIFAR10 models plotted by memorization ratio vs. metric. Memorization metrics against the test set instead of the training set are shown with low opacity.
(see Figure 1), respectively. From this analysis, we note that modifying the  $C_T$  score by swapping the roles of the training and generated datasets makes it insensitive to mode collapse. We include this modification of the  $C_T$  score in Figure 7 (right).
In conclusion, we find that none of AuthPct, the  $C_T$  score, or FLS-POG is a reliable metric for memorization. FLS-POG correlates poorly with our estimates of the percentage of memorized samples, while the  $C_T$  score and AuthPct detect mode shrinking and image fidelity more than memorization. The reason behind these deficiencies is left to future work. Concurrent work [59] shows that in high dimensions moving the support of a distribution can drastically change precision and recall (measured using  $k$ -nearest neighbors), and the observed phenomenon here might have a similar cause. Our recommended modification to the  $C_T$  score improves on the  $C_T$  score and AuthPct, but still does not correlate well with the memorization ratio. Instead of using these metrics, we recommend researchers directly search for and collect memorized images using the calibrated  $l_2$ -distance as described above, even though it is labour-intensive and requires tuning.
We carried out the largest and most comprehensive assessment of generative model evaluation metrics to date. We found that currently prevalent metrics such as FID are not strongly predictive of human error rate, and that diffusion models achieve a higher human error rate than their GAN counterparts, yet often are ranked worse according to FID. Our multiple investigations show that this discrepancy is not caused by model diversity, nor by humans assessing rare but real images as fake. Together, these findings imply that the differences between FID and human assessment are unfairly punitive towards diffusion models (in terms of assessing fidelity). We also showed that these deficiencies can be mostly addressed by replacing the Inception encoder by DINOv2 ViT-L/14. We include a table with the  $\mathrm{FD_{DINOv2}}$  score and many other metrics for all the models we considered in Appendix E, which we hope will be useful as an updated leaderboard of model performance. Finally, FD-based metrics are not designed to detect memorization, and we show that except on CIFAR10, the best performing models in terms of  $\mathrm{FD_{DINOv2}}$  are not memorizing their training data. In doing so, we also found that while the calibrated  $l_2$  distance proposed in [13] is a reliable way to identify memorized samples, other metrics to detect memorization are not ideal. We thus advocate for work proposing new generative models to report the ratio of memorized samples using the calibrated  $l_2$  metric alongside metrics computed with DINOv2-ViT-L/14. Because it requires tuning, finding more automated metrics that reliably detect memorization will be a productive avenue for future research.
Broader impact While we do not develop or release any generative models in this work, our work can be used to improve the quality of generative models in the future. Such generative models, despite having a number of beneficial uses, have the potential to be used for negative applications such as the generation of deepfakes. We note that all generative models, encoders, and datasets explored in this work are already publicly available assets.
Limitations We take human perceptions of image realism as a ground truth for measuring image fidelity, noting the expansive efforts of the community to generate images that appear photo-realistic to humans, and will be used by humans. But, for certain generative applications, such as the use of generative models as augmentations to improve performance of classifiers [2], humans are not always the end users. In this case the end user is the downstream classifier, and models that achieve the best  $\mathrm{FD_{DINOv2}}$  score may not directly translate to optimal improvements in downstream classification. Additionally, while the image datasets (and hence generative models) in our study were chosen to span a wide range of natural images, they were all 3-channel RGB format of human-identifiable objects. It is an open question whether our proposed methods of evaluating generative models translate to e.g. medical images or astronomical images, which are also RGB, or more broadly to scientific data of N-dimensions.
Compute Experiments were conducted without a requirement for significant computational resources. Generating datasets was performed either on NVIDIA Titan V GPUs with 12GB of RAM or a small local cluster, and required a total of  $\sim 1100$  hours of GPU time, a significant fraction of which was spent generating images from the DiT and LDM models. The analysis was conducted on NVIDIA Titan V GPUs with 12GB of RAM, and computing all metrics for each encoder and generative model required  $\sim 48$  hours on a single GPU.
Assets and code As stated in the main text, to facilitate further development of generative models and their evaluation we have prepared a public release of all generated image datasets, human evaluation data, and a modular library to compute 17 common metrics for 9 different encoders at https://github.com/layer6ai-labs/dgm-eval. We provide detailed descriptions and links to the public generative models, metrics, and encoders used in Appendices A and B.
A Generated datasets
For each generative model across the four datasets studied in this work we generated 100,000 images, unless otherwise stated. For class-conditional models we generated an equal number of samples from each class - 10,000 per class for CIFAR10 and 100 per class for ImageNet. We chose the checkpoints and set parameters (if applicable) to the ones that achieved the lowest FID as stated in the paper, or on the project's github if not otherwise stated. For GANs we set truncation=1.0 throughout, and for class-conditional diffusion models we set the guidance parameter to the reported values. To validate that our set of images for each model was generated correctly we compare the FID calculated with our code (using 50k samples) to the reported value in the paper or in the code release. Any discrepancies were investigated by contacting the authors or raising issues on the project github.
When downsampling images from their native sizes to the ones used in this work we note that significant differences in FID can result from different choices of interpolation methods. As such, we report the interpolation techniques we used to downsample ImageNet and FFHQ to  $256 \times 256$  pixels, and note that these are required for reproducing our results when using the original images (the resized datasets are provided along with our code). Differences in our FIDs and the reported values in various papers could partially be attributed to this effect - for example using Box filtering instead of Lanczos increases the FID of StyleGAN2-ada on FFHQ by 1.0.
Below we list  $(i)$  the generative models used for each dataset,  $(ii)$  links to the project pages and checkpoints used,  $(iii)$  the reported FID in comparison to the value determined from our generated images, and  $(iv)$  any special notes.
A.1 CIFAR10
- StudioGAN models [55] https://github.com/POSTECH-CVLab/PyTorch-StudioGAN, checkpoints are in folder https://huggingface.co/
Mingguksky/PyTorch-StudioGAN/tree/main/studiogan_official_ckpt/CIFAR10_tailored/. We used current weights from each folder.
- ACGAN [79] CIFAR10-ACGAN-Mod-train-2022_03_06_02_24_19. Reported FID=33.39,ours=35.47.
- BigGAN-Deep [12] CIFAR10-BigGAN-Deep-train-2022_02_02_21_56_10 Reported FID N/A, ours=3.91.
- LOGAN [111] CIFAR10-LOGAN-train-2022_03_12_04_15_31. Reported FID=20.65,ours=17.87.
- ReACGAN [54] CIFAR10-ReACGAN-train-2022_01_24_23_46_59. Reported FID=3.87, ours=4.40.
-MHGAN [104] CIFAR10-MHGAN-train-2022_02_14_18_23_18. Reported FID=3.95,ours=4.22.
-WGAN-GP [41] CIFAR10-WGAN-GP-train-2022_01_25_16_34_00. Reported FID=53.98,ours=26.25.
- LSGM-ODE [107] https://github.com/NVlabs/LSGM. We used the FID checkpoint, sampling with ODE framework. Reported FID=2.10, ours=2.12.
- iDDPM-DDIM [78] https://github.com/openai/improved-diffusion. Reported FID=2.94, ours=3.27.
- PFGM++ [112] https://github.com/newbeeer/pfgmpp. We used the following checkpoint https://drive.google.com/drive/folders/1IADJcuoUb2wc-Dzg42-F8RjgKVSZE-Jd?usp=share_link. Reported FID 1.74, ours=1.79.
- RESFLOW [18] https://github.com/rtqichen/residual-flows. Reported FID=46.37,ours=48.29.
- NVAE [106]. Reported FID N/A, ours=32.53.
- StyleGAN2-ada [57] https://github.com/NVlabs/stylegan2-adad-pytorch. We used the following checkpoint https://nvlabs-fi-cdn.nvidia.com/ stylegan2-adad-pytorch/pretrained/cifar10.pkl. Reported FID=3.49,ours=2.55.
- StyleGAN-XL [98] https://github.com/autonomousvision/stylegan-x1. We used the following checkpoint https://s3.eu-central-1.amazon.com/avg-projects/stylegan_xl/models/cifar10.pkl. Reported FID=1.85,ours=1.87.
A.2 ImageNet
- Four models used sets of 50k publicly available images provided at https://github.com/openai/guided-diffusion/tree/main/evaluations [25].
- ADM [25]. Reported FID=10.94, ours=11.84.
- ADMG [25]. Reported FID=4.59, ours=5.58.
- ADMG-ADMU [25]. Reported FID=3.94, ours=4.75.
- BigGAN [12]. Reported FID=6.95, ours=7.94.
- DiT-XL-2 [84] https://github.com/facebookresearch/DiT. Reported FID=2.27, ours=2.80.
- DiT-XL-2-guided [84]. This model was equivalent to the above, but used a stronger classifier free guidance term  $\text{cfg} = 4.0$  instead of  $\text{cfg} = 1.5$ . As described in Section 5.1, this model was included to study the effects of a model that produces very realistic images, but sampled with a lower intra-class diversity. Reported FID=N/A, ours=17.24.
- GigaGAN [56] with 100k images provided privately by authors. Reported FID=3.45, ours=4.16.
- LDM [93] https://github.com/CompVis/latent-diffusion. Reported FID=3.60, ours=4.29.
- Mask-GIT [17] https://github.com/google-research/maskgit. We used the following checkpoint https://storage.googleapis.com/maskgit-public/checkpoints/maskgit_imagenet256_checkpoint. Reported FID=6.06, ours=5.63.
- RQ-Transformer [65] https://github.com/kakaobrain/rq-vae-transformer. We used the 1.4B model from the following checkpoint https://arena.kakaocdn.net/ brainrepo/models/RQVAE/6714b47bb9382076923590eff08b1ee5/imagenet_1. 4B_rqvae_50e.tar.gz. Reported FID=8.71,ours=9.71.
- StyleGAN-XL [98] https://github.com/autonomousvision/stylegan-xl. We used the following checkpoint https://s3.eu-central-1.amazon.com/avg-projects/stylegan_xl/models/imagenet256.pkl. Reported FID=2.26, ours=2.91.
A.2.1 A note on the curation of ImageNet
The treatment of the raw ImageNet dataset [23] is often not explicitly described in the literature - particularly for building generative models on ImageNet - and we have found some inconsistencies on how FID is reported. Therefore we describe our exact approach to calculating FID on ImageNet, and provide our parsing scripts with the goal of standardizing the process across papers. The full dataset was obtained from https://www.kaggle.com/competitions/imagenet-object-localization-challenge/data, as the data is no longer available from the ImageNet website (https://www(image-net.org/index.php).
Raw ImageNet is a dataset of 1,000 classes, each with roughly 1,300 images. The images themselves are generally non-square, with image height and width both always at least 256. To get from the raw rectangular ImageNet images to the  $256 \times 256$  resolution of ImageNet256 we perform the following two operations:
1. Center crop along the long edge. This results in a square image of side length  $N = \min\{H, W\}$ , given individual image height  $H$  and width  $W$ .
2. Downsample from  $N \times N$  to  $256 \times 256$  using bicubic interpolation, as suggested by [82]. We also experimented with Lanczos interpolation with only minimal change to the results.