SlowGuess/NeurIPS_2023_OCR huggingface dataset:
George Stein* Jesse C. Cresswell† Rasa Hosseinzadeh† Yi Sui†
Brendan Leigh Ross† Valentin Villecroze‡ Zhaoyan Liu‡
Anthony L. Caterini $^{§}$  J. Eric T. Taylor $^{§}$  Gabriel Loaiza-Ganem $^{§}$
{george, jesse, rasa, amy, brendan, valentin.v,
zhaoyan, anthony, eric, gabriel}@layer6.ai
Layer 6 AI
We systematically study a wide variety of generative models spanning semantically-diverse image datasets to understand and improve the feature extractors and metrics used to evaluate them. Using best practices in psychophysics, we measure human perception of image realism for generated samples by conducting the largest experiment evaluating generative models to date, and find that no existing metric strongly correlates with human evaluations. Comparing to 17 modern metrics for evaluating the overall performance, fidelity, diversity, rarity, and memorization of generative models, we find that the state-of-the-art perceptual realism of diffusion models as judged by humans is not reflected in commonly reported metrics such as FID. This discrepancy is not explained by diversity in generated samples, though one cause is over-reliance on Inception-V3. We address these flaws through a study of alternative self-supervised feature extractors, find that the semantic information encoded by individual networks strongly depends on their training procedure, and show that DINOv2-ViT-L/14 allows for much richer evaluation of generative models. Next, we investigate data memorization, and find that generative models do memorize training examples on simple, smaller datasets like CIFAR10, but not necessarily on more complex datasets like ImageNet. However, our experiments show that current metrics do not properly detect memorization: none in the literature is able to separate memorization from other phenomena such as underfitting or mode shrinkage. To facilitate further development of generative models and their evaluation we release all generated image datasets, human evaluation data, and a modular library to compute 17 common metrics for 9 different encoders at https://github.com/layer6ai-labs/dgm-eval.
The capability of modern generative models to synthesize fake images that are seemingly indistinguishable from real samples has resulted in much public interest [90, 89, 94]. While the evaluation of such models has a longstanding history [10, 11], the unprecedented fidelity of modern synthetic images (e.g. [25, 93]) raises the question of whether the current tools in use by researchers are sufficient to measure the extent to which these models have truly learned the ground truth distribution, and whether striving to achieve state-of-the-art performance on current metric leaderboards provides optimal targets to drive further algorithmic progress.
Evaluating a single generated image is straightforward, since humans can act as the "ground truth" for determining realism. Evaluating the quality of a model as a whole is much more difficult. Beyond
*†‡§ Authors within each of the four groups contributed equally, see Appendix F for a list of contributions.
Figure 1: An illustration of learned distributions and samples (orange, crosses) having different properties with respect to the true distribution and training set (blue, squares). Italicized text indicates metrics that purport to detect these properties.
quantifying to what extent images resemble those from the training set (fidelity), we must determine how well the generated samples span the full training distribution (diversity), and whether they are truly novel or are simply reproductions of training samples (memorization), as illustrated in Figure 1. An ideal generative model will synthesize high fidelity and diverse samples without memorizing the training set (the latter becoming a prominent concern for models trained on unlicensed data [13, 101]). Researchers are well-practiced in ranking generative models by metrics such as the Fréchet Inception distance (FID) [48], Inception score (IS) [96], and many others [8, 76, 53, 91] which group fidelity and diversity into a single value without a clear tradeoff. Other popular diagnostic metrics separate sample quality from diversity such as precision/recall [95, 64] and density/coverage [75]. However, relating such metrics to human evaluation of image quality is not straightforward [120].
These metrics generally follow a two-step design: extract a lower-dimensional representation of each image, then calculate a notion of distance between true and generated samples in this space. The goal of the representation extractor, or encoder, is to embed images into a representation space that has a generalized perceptual relevance across the span of natural images. The implicit assumption in the de-facto use of the (pool3, 2048 dimensional) Inception-V3 network [103] trained for ImageNet1k [23] classification is that it provides such a space. Yet major concerns have been raised: it has been shown to be agnostic to features unrelated to the 1k classes of ImageNet [63], ImageNet classifiers in general are biased towards texture over shape [36, 47], and many other criticisms [120, 82, 72, 7].
Obvious choices for more universally applicable representation spaces are modern self-supervised learning (SSL) models [4] trained on large and diverse datasets, as they have proven to extract representations that excel at a number of generalized downstream tasks [19, 14, 20, 40, 15, 87, 44, 80]. While initial studies of a few self-supervised encoders reported that representations from these networks can produce more adequate rankings for generative adversarial networks (GANs) [38] on non-ImageNet domains [72, 7], it is an open question as to which SSL methods and families provide the best perceptual representation space for evaluating natural images more generally. For example, SSL methods based on contrastive learning strategies utilizing strong augmentations tend to learn features that are more invariant to those augmentations [30]. Thus, while the criteria for choosing an SSL encoder for classification tasks is straightforward – choose one that achieves strong linear classification accuracy – it is not clear that such a model will extract a general representation for generative evaluation rather than one that over-relies on object-based semantic information.
Understanding the interdependence of evaluation metrics, representation extractors, and their relation to human evaluation of generated images requires a large scale study of each component across a diverse set of datasets. Here we select 41 state-of-the-art generative models spanning diffusion models [100], GANs, variational autoencoders (VAEs) [60], normalizing flows [92, 26], transformer-based models [9], and consistency models [102], and generate 4.1M images to provide such a study:
Human evaluation We designed and funded extensive human subject experiments to establish a robust baseline for generated image fidelity, and find that  $(i)$  no current metric strongly correlates with human evaluators, and that  $(ii)$  diffusion models significantly outperform GANs and all other generative techniques at producing images that are indistinguishable from training data.
Self-supervised representations and evaluation metrics We show that (iii) the Fréchet distance, kernel distance (KD) [8], precision, and density calculated with the Inception-V3 network do not correlate well with human evaluation. We then investigate the semantic information distilled from self-supervised methods spanning a wide variety of families, showing that (iv) the perceptual qualities
of their representation spaces can strongly depend on training procedure and architecture, that  $(v)$  supervised networks do not provide a perceptual space that generalizes well for image evaluation, and that  $(vi)$  replacing Inception-V3 with DINOv2 ViT-L/14 [80] solves the discrepancy with human evaluators while the previously proposed SwAV and CLIP-B/32 replacements [72, 7] are sub-optimal.
Diversity, rarity, and memorization By leveraging the recently proposed Vendi [33] and rarity [42] scores, we show that  $(vii)$  the discrepancy between human evaluators and FID is not due to models trading off fidelity for diversity, nor to human evaluators assessing rare images as fake. We see these results as evidence that human error rate is a sensible "ground truth" to align FD metrics with. Finally, we answer the question: are the best performing models according to our DINOv2-based metrics memorizing their training data? In doing so, we  $(viii)$  find clear evidence of memorized samples across models, particularly on CIFAR10, and show that current memorization metrics and tests fail to capture this [70, 1].
Summary Our multifaceted investigation of generative evaluation shows that diffusion models are unfairly punished by the Inception network: they synthesize more realistic images as judged by humans and their diversity more closely resembles the training data, yet are consistently ranked worse than GANs on metrics computed with Inception-V3. While FID is already known to have shortcomings, we advocate for a complete replacement of Inception-V3 in all evaluation metrics of images, and show that DINov2-ViT-L/14 allows for much richer evaluation of generative models.
2 Datasets, metrics, and encoders
Generated datasets We investigate a wide range of generative models trained on a diverse set of image datasets (CIFAR10 [61], ImageNet1k [23], FFHQ [58], LSUN-Bedroom [115]) using a variety of generative techniques (diffusion, GAN, VAE, normalizing flow, Transformer-based, consistency). For each dataset we include current state-of-the-art models as ranked by FID, as well as models spanning different generative procedures. We include 13, 11, 9, and 8 models for the respective datasets listed above, for a total of 41 generated datasets. To decouple the effects of model architecture/training procedure from the training data used, we focus only on generative models that did not include any data external to the respective dataset during training. Models for ImageNet1k, FFHQ, and LSUN-bedroom were trained at a resolution of  $256 \times 256$ . We chose to generate 100k images from each model, with an equal number of images per-class for class conditional models, and used the checkpoints and hyperparameters that achieved the lowest FID for each model; Appendix A details the full generation procedure. In total we assess each generative model across 17 metrics. We group the metrics by category here and provide full definitions in Appendix B.
Metrics for ranking generative models We include the well-known FID [48], which computes the Fréchet distance (FD) between sets of 50k real and generated samples in the representation space of the Inception-V3 network. We study FD in several alternative representation spaces, and refer to these by the model used to extract the representation from each image, e.g.  $\mathrm{FD}_{\mathrm{CLIP}}$ . We include alternatives to the FID such as the spatial FID (sFID) [76],  $\mathrm{FID}_{\infty}$  [22], IS [96], kernel Inception distance (KID) [8], and the feature likelihood score (FLS) [53].
Metrics for diagnosing fidelity, diversity, rarity, or memorization As proxies for sample fidelity we consider precision [95, 64] and density [75], while our human evaluation baseline, human error rate, provides a direct measurement of fidelity. To quantify sample diversity we consider recall [95, 64] and coverage [75], and for sample rarity we consider the rarity score [42]. To study inter- vs. intra-class diversity for class-conditional image generation we utilize the Vendi score [33]. While recall and coverage can also be determined per class, the small number of generated samples available for each class (e.g. 100 for each ImageNet class) results in difficulty constructing robust nearest-neighbour-based estimates. We also investigate a form of overfitting that we term **memorization**, in which models memorize individual images from their training data and emit them at generation time. We perform a direct check for pixel-wise memorization for each generative model for each of our datasets and report this value as the memorization ratio in Section 5.2. We include automated metrics which claim to isolate memorization from the effects of overfitting or mode collapse: the percentage of authentic samples [1],  $C_T$  score [70], and the percentage of overfit Gaussians from FLS [53].
Representation spaces for generative evaluation With the aim of finding a more general perceptual representation space across the span of natural images, we employ a number of alternative encoders beyond the standard Inception-V3 [103] trained for supervised classification on ImageNet. First, as a
more modern supervised benchmark we use a ConvNeXt-large architecture trained on ImageNet22k [67], a larger dataset with more classes. We also include a number of self-supervised feature extractors as alternatives to supervised learners. While such networks have proven to be useful for concurrent vision tasks including classification, object-detection, and segmentation [14, 119, 5], it remains an open question as to how the objective and augmentations used for training affect the representation space for generative evaluation. Thus we include seven self-supervised methods from a variety of families [4] - contrastive (SimCLRv2 [19]), self-distillation (DINOv2 [80]), canonical correlation analysis (SwAV [14]), masked image modelling (MAE [44] and data2vec [3]), and language-image (CLIP [87], using the OpenCLIP implementation [51] trained on DataComp-1B [35]). We also consider DreamSim [34], an ensemble of three self-supervised models (DINO [15], CLIP, and OpenCLIP) that have each been fine-tuned to better align with human perception of image similarity using a dataset of human similarity judgments over image pairs. We design experiments to qualitatively and quantitatively understand their respective feature spaces.
For CNN-based models we use the ResNet50 [45] architecture while for vision transformers (ViT) [28] we use ViT-L/14 as we found it provides a good tradeoff between representation quality and computation cost. For both we select weights that were trained with the dataset and hyperparameters that achieved the highest linear classification on ImageNet. Full details are in Appendix B.4.
3 Human evaluation of generated data
The goal of our human subject experiments is to establish a large-scale, scientifically grounded baseline for image generation fidelity. Our experiments specifically target image realism, not the diversity of a model's learned distribution, nor whether samples are memorized, as realism is unequivocally a property for which humans can provide "ground truth". We find that the vast majority of models tested per dataset can be separated in terms of their realism with statistical significance, thus providing a clear ranking which can be compared to calculated metrics like FID. To our knowledge, this is the largest human subject experiment on the evaluation of generative models performed to date, with over 1000 paid participants, and 207k individual responses collected.
Experimental design We evaluated human perception of generated image realism for each of the 41 models across 4 datasets described in Section 2. Our experimental design was informed by experts and best practices in psychophysics, the study of the human perceptual system. It follows the design of experiments for the  $\mathrm{HYPE}_{\infty}$  metric [120], with several modifications to increase the quality of collected data and expand the scope of models and datasets tested. Each trial is a two alternative forced choice task where a participant is shown either a generated image from one model, or an image from the training dataset, and must choose if it is real or fake. Models were evaluated based on human error rate [120], the fraction of images which were incorrectly classified. This is a simple metric with the intuition that models with better fidelity produce images that are more difficult to distinguish from real images. We take human perceptions of image realism as ground truth, noting the expansive efforts of the community to generate images that appear photo-realistic to humans, and will be used by humans. Full design details are provided in Appendix C.
Concurrent work has also targeted human perceptual benchmarks for image synthesis on a smaller subset of GAN and diffusion models [114] with 10 times fewer participants. While we are excited to see human benchmarks gaining popularity, we note a number of concerns with their methods and thus downstream conclusions. Specifically, observers had no training period nor knowledge of the training set, and were tasked to judge whether images were "photo-realistic". We believe this task contains much more ambiguity than our two alternative forced choice assessment, and introduces various response biases into participants' judgments. In their second user study, observers were asked to evaluate the relative realism of sets of images. This task is difficult to evaluate because participants can use a single image in the set to guide their decision for the entire set.
Results and analysis The main results of our experiments are shown in Figure 2. We plot the mean human error rate along with standard error for each model, and sort models on the  $x$ -axis by their FID (lower is better). If FID correlated well with human perception of fidelity, each plot would have a monotonically increasing trend, but this does not appear to hold. By inspection, the models with highest fidelity are almost always diffusion models, although GAN models often have lower FID. We find similar results for alternatives to the FID score in Appendix D.1.
Figure 2: Human error rate on models ranked by FID. Data is displayed as the mean across participants with error bars showing the unbiased standard error.
To formally assess the relative performance of different model types, we separate participants' mean error rate into four one-way analyses of variance (ANOVA) – one per dataset – with model type (e.g. GAN, diffusion) as a between-subjects variable. We also performed planned comparisons between model types to probe omnibus effects. Analysis revealed effects of model type in all four ANOVAs, indicating significant differences between model types' effect on human error rate (all  $F$ 's  $> 12.09$ , all  $p$ 's  $< 0.001$  [77]). Post hoc comparisons of mean error rate between model types using Bonferroni correction are shown in Table 1, where  $>$  indicates a significant difference,  $=$  indicates no significant difference, and model type is listed in order of descending mean error rate. Diffusion models were a clear standout for all datasets except FFHQ, where they performed on par with other model types (noting that only GANs appeared in more than one experiment on FFHQ). Coupling the results in Table 1 with the FID rankings in Figure 2, we conclude that current diffusion models produce the most realistic images according to human perception, but are downranked by FID.
4 Improved representation spaces for generative evaluation
4.1 Qualitative examination of perceptual spaces
To qualitatively visualize what parts of an image the Fréchet distance "perceives", we follow the gradient based visualization technique of [63], which focused on FID. Here we adapt and apply it to each of our CNN and ViT encoders. For CNN encoders our method is identical, while for ViTs we use the Grad-CAM variation introduced by [37]. Experimental details can be found in
Table 1: Model types ranked by human error rate.
Dataset	Error rate ranking
CIFAR10	Diff. > GAN > VAE > Flow
ImageNet	Diff. > Transf. = GAN
LSUN	Diff. > Transf. = GAN = Consistency
FFHQ	GAN = Diff. = Transf. > VAE
Appendix D.2.1. Figure 3 shows two visualized samples from each high-resolution dataset. We find qualitative differences between CNN and ViT architectures - ViTs have a more global receptive field [88] - but also find starkly different characteristics between supervised and self-supervised models. In agreement with [63], regions deemed important by Inception are far from optimal for datasets outside of the ImageNet domain. For FFHQ the important features according to Inception are typically not part of the person's face, while for LSUN-Bedroom the focus is on a single object in the scene; features simply correspond with the Inception model's top-1 class prediction. We find that Inception does not perceive a holistic view of images even on its ImageNet training set, and see similar characteristics for ConvNeXt, indicating that such glaring issues are not mitigated by supervised training on more modern architectures, larger datasets, nor a larger numbers of classes.
Meanwhile SwAV and SimCLR often ignore important features, with SwAV exhibiting the strongest overlap with the classification networks. CLIP puts a large focus on the few main objects of an image – typically the main facial features (lips, eyes, etc.) or objects (beds) – which we presume is an outcome of the language-image pretext arising from image captions that do not describe texture or finer details of the image. DINOv2 usually focuses on the image structure as a whole while still identifying objects of importance. We argue that this is closer to the behaviour we would hope to
Figure 3: Heatmaps visualizing what the Fréchet distance "perceives" for each encoder. The sign of the heatmap is given by the activations of the saliency layer that is visualized and does not reflect the sign of the gradient w.r.t. the FD - both bright yellow and deep blue can thus show an encoder's focus. Additional examples are shown in Appendix D.2.1.
have from an encoder meant to evaluate images, as it emphasizes the important objects while still being able to pick up on elements elsewhere. MAE and data2vec, both trained using masked image modelling, have a widespread focus on textures and shapes, with an often smaller importance for the semantic information related to the main object. We exclude DreamSim from this analysis as it is not straightforward to use Grad-CAM on its output, which is the concatenation of the output of multiple encoders. Appendix D.2.2 includes quantitative analyses, finding that masked models put more weight towards low-level image features rather than clustering classes by object semantics, while others (CLIP in particular) distill a more object-focused representation space – both in alignment with the qualitative analysis shown here.
In summary, we conclude that the representation spaces of self-supervised methods are more appropriate for generative evaluation than supervised approaches, and that self-distillation (DINOv2) provides the best balance between focusing on important objects and holistic image structure.
4.2 The (mis)alignment of evaluation metrics and human assessment
In conjunction with our human evaluation baseline, we evaluated the 17 ranking and diagnostic metrics outlined in Section 2 for each encoder and generated dataset. Figure 4 (top) shows the relation of human error rate with FD and precision. We investigate diversity further in the following section. We include the correlation of 7 common metrics and human error rate (bottom, best viewed while zoomed in). Note that across encoders, FD,  $\mathrm{FD}_{\infty}$ , and KD are very highly correlated, resulting in essentially the same model rankings. This suggests that all these metrics provide sensible ways of quantifying distances between probability distributions, provided a good encoder is chosen. Overall, we find that despite some sample complexity issues, the FD metric – when paired with an appropriate encoder – provides a strong way of evaluating generative models, see Appendix D.3 for details.
We find no strong correlation between human evaluation and any common metrics computed in the Inception representation space outside of the simplistic CIFAR10 dataset, showing that Inception fails to encode perceptually relevant features for the larger, more diverse, and complex datasets that are the testbeds driving advancement of generative model development. In agreement with the previous section we find relatively poor performance of the SwAV model, which was proposed as an alternative