Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models
Authors: George Stein
Jesse C. Cresswell
Rasa Hosseinzadeh
Yi Sui
Brendan Leigh Ross
Valentin Villecroze
Zhaoyan Liu
Anthony L. Caterini
Eric Taylor
Gabriel Loaiza-Ganem
Venue: NeurIPS 2023 poster (2023)
Keywords: generative models
generative model evaluation
self-supervised learning
representation learning
metrics
TL;DR: The perceptual fidelity of diffusion models is not reflected in commonly reported metrics of generative model evaluation; we diagnose and fix the issue, providing a new leaderboard for image-based generative models.

Abstract: We systematically study a wide variety of generative models spanning semantically-diverse image datasets to understand and improve the feature extractors and metrics used to evaluate them.
Using best practices in psychophysics, we measure human perception of image realism for generated samples by conducting the largest experiment evaluating generative models to date, and find that no existing metric strongly correlates with human evaluations.
Comparing to 17 modern metrics for evaluating the overall performance, fidelity, diversity, rarity, and memorization of generative models, we find that the state-of-the-art perceptual realism of diffusion models as judged by humans is not reflected in commonly reported metrics such as FID. This discrepancy is not explained by diversity in generated samples, though one cause is over-reliance on Inception-V3.
We address these flaws through a study of alternative self-supervised feature extractors, find that the semantic information encoded by individual networks strongly depends on their training procedure, and show that DINOv2-ViT-L/14 allows for much richer evaluation of generative models. Next, we investigate data memorization, and find that generative models do memorize training examples on simple, smaller datasets like CIFAR10, but not necessarily on more complex datasets like ImageNet. However, our experiments show that current metrics do not properly detect memorization: none in the literature is able to separate memorization from other phenomena such as underfitting or mode shrinkage. To facilitate further development of generative models and their evaluation we release all generated image datasets, human evaluation data, and a modular library to compute 17 common metrics for 9 different encoders at https://github.com/layer6ai-labs/dgm-eval.

Exposing flaws of generative model evaluation metries and their unfair treatment of diffusion models
George Stein' Jesse C. Cresswell' Rasa Hosseinzadehl
Yi Suit
Brendan Leigh Ross
Valentin Villecroze Zhaoyan Liu
Anthony L. Caterini J. Eric T. Taylor Gabriel Loaiza-Ganem" {george, jesse, rasa, amy, brendan, valentin.v, zhaoyan, anthony, eric, gabriel)@layer6.. ai Layer Al
Abstract
We systematically study wide variety of generative models spanning semantically- diverse image datasets to understand and improve the feature extractors and metrics used to evaluate them. Using best practices in psychophysics, we measure human perception of image realism for generated samples by conducting the largest experiment evaluating generative models date, and find that no existing metric strongly correlates with human evaluations. Comparing to 17 modern metrics for evaluating the overall performance, fidelity, diversity, rarity, and memorization of generative models, we find that the state-of-the-ar perceptual realism of diffusion models as judged by humans not reflected in commonly reported metrics such as FID. This discrepancy is not explained by diversity in generated samples, though one cause over-reliance on Inception-V3. We address these flaws through a study of alternative self-supervised feature extractors, find that the semantic information encoded by individual networks strongly depends on their training procedure, and show that DINOv2-ViT-L/LE allows for much richer evaluation generative models. Next, we investigate data memorization, and find that generative models do memorize training examples simple, smaller datasets like CIFARIO, but not necessarily on more complex datasets like ImageNet. However, experiments show that current metrics do not properly detect memorization: none in the literature able to separate memorization from other phenomena such underfitting mode shrinkage. To facilitate further development of generative models and their evaluation we release all generated image datasets, human evaluation data, and modular library compute 17 common metrics for 9 different encoders at https::///// /et:::/gtttttt com com/layer6ai-1abe waw.rraailllbbb/daa dabaade-eaaddddee
Introduction
The capability of modern generative models to synthesize fake images that are seemingly indistin- guishable from real samples has resulted in much public interest 900 89 94] While the evaluation of such models has a longstanding history [00 [Л] the unprecedented fidelity of modern synthetic images (e.g. [25] 93) raises the question of whether the current tools in use by researchers are sufficient to measure the extent to which these models have truly learned the ground truth distribution, and whether striving achieve faate-of-the-ree performance current metric leaderboards provides optimal targets to drive further algorithmic progress.
Evaluating a single generated image is straightforward, since humans can act as the "ground truth" for determining realism. Evaluating the quality of a model as a whole is much more difficult. Beyond
#33 Authors within each of the four groups contributed equally, see Appendix for list contributions.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

High Fidelity Precision, Density
High Diversity
Memorization Autheet, FLS-POG
Mode Collapse
Mode Shrinkage
Recall, Coverage, Vendi
#
An illustration of learned distributions and samples (orange, crosses) having different properties with respect to the true distribution and training set (blue, squares). Italicized text indicates metrics that purport to detect these properties.
Figure 1:
quantifying what extent images resemble those from the training set (fideiity, we must determine how well the generated samples span the full training distribution (diversity), and whether they are simply reproductions of training samples (memorization), illustrated in Figure] An ideal generative model will synthesize high fidelity and diverse samples without memorizing the training set (the latter becoming a prominent concern for models trained on unlicensed data [1 101). Researchers are well-practiced in ranking generative models by metrics such as the Fréchet Inception distance (FID) [48), Inception score (IS) [96), and many others g
truly novel
76 53 91] which group fidelity
and diversity into single value without a clear tradeoff. Other popular diagnostic metries separate sample quality from diversity such as precision/recall [95] 64] and density/coverage [75]. However, relating such metrics to human evaluation of image quality is not straightforward [120].
These metrics generally follow two-step design: extract lower-dimensional representation of each image, then calculate a notion of distance between true and generated samples in this space. The goal of the representation extractor, or encoder, is to embed images into representation space that has generalized perceptual relevance across the span of natural images. The implicit assumption in the de-facto use of the (pool3, 2048 dimensional) Inception-V33 network [103] trained for ImageNet [23] classification is that it provides such a space. Yet major concerns have been raised: has been shown be agnostic to features unrelated the 1k classes of ImageNet [63], ImageNet classifiers in
general are biased towards texture over shape [3 47]. and many other criticisms [120 82 72, (
Obvious choices for more universally applicable representation spaces modern self-supervised learning (SSL) models 4 trained on large and diverse datasets, as they have proven to extract representations that excel at number of generalized downstream tasks [19] 14 20 15
80]
While initial studies of a few self-supervised encoders reported that representations from these networks can produce more adequate rankings for generative adversarial networks (GANs) 38 on non-ImageNet domains 772 7] it is an open question as to which SSL methods and families provide the best perceptual representation space for evaluating natural images more generally. For example, SSL methods based on contrastive learning strategies utilizing strong augmentations tend to learn features that are more invariant those augmentations [30]. Thus, while the criteria for choosing an SSL encoder for classification tasks is straightforward - choose one that achieves strong linear classification accuracy
not clear that such model will extract a general representation for generative evaluation rather than one that over-relies on object- based semantic information.
Understanding the interdependence of evaluation metrics, representation extractors, and their relation to human evaluation of generated images requires a large scale study of each component across a diverse set of datasets. Here we select 41 state-of-the-art generative models spanning diffusion models 100), GANs, variational autoencoders (VAFs) 60) normalizing flows [92 26 transformer-baser
models g and consistency models [02 and generate 4. IM images to provide such a study:
Human evaluation We designed and funded extensive human subject experiments to establish a robust baseline for generated image fidelity, and find that (i) no current metric strongly correlates with human evaluators, and that (ii) diffusion models significantly outperform GANs and all other generative techniques producing images that indistinguishable from training data.
Self-supervised representations and evaluation metrics We show that (iii) the Fréchet distance, kernel distance (KD) [8), precision, and density calculated with the Inception-V3 V3 network do not correlate well with human evaluation. We then investigate the semantic information distilled from self-supervised methods spanning wide variety of families, showing that (iv) the perceptual qualities

---

of their representation spaces can strongly depend on training procedure and architecture, that (v) supervised networks do not provide a perceptual space that generalizes well for image evaluation. and that (vi) replacing Inception-V V3 with DINOv2 ViT-L/14 [)] solves the discrepancy with human evaluators while the previously proposed SwAV and CLIP-B/32 replacements 72 7
are sub-optimal.
Diversity, rarity, and memorization By leveraging the recently proposed Vendi 33 and rarity [42] scores, we show that (vii) the discrepancy between human evaluators and FID not due to models trading off fidelity for diversity, nor to human evaluators assessing rare images as fake. We see these results as evidence that human error rate is sensible "ground truth" to align FD metrics with. Finally, we answer the question: are the best performing models according to our DINOv2-based metrics memorizing their training data? In doing so, we (užl)) find clear evidence of memorized samples across models, particularly on CIFARIO, and show that current memorization metrics and tests fail to capture this [70
Summary Our multifaceted investigation of generative evaluation shows that diffusion models are unfairly punished by the Inception network: they synthesize more realistic images as judged by humans and their diversity more closely resembles the training data, yet are consistently ranked worse than GANs on metrics computed with Inception-V3. While FID is already known to have shortcomings, we advocate for a complete replacement of Inception-V3s in all evaluation metrics of images, and show that --VT-L44 allows for much richer evaluation of generative models.
2 Datasets, metrics, and encoders
Generated datasets We investigate a wide range of generative models trained on diverse set of image datasets (CIFAR1O [l], ImageNetlk [23], FFHQ 58). LeUN-Bedroom (155) using a variety of generative techniques (diffusion, GAN, VAE, normalizing flow, Transformer- based, consistency). For each dataset we include current state-of-the teef-thh-rrttt art models as ranked by FID, as well as models spanning different generative procedures. We include 13, 11, 9, and 8 models for the respective datasets listed above, for a total of 41 generated datasets. To decouple the effects of model architecture/triiingg procedure from the training data used, we focus only on generative models that did not include any data external to the respective dataset during training. Models for ImageNet Ik, FFHQ, and SUUN-bedroom were trained at resolution of 256 256. We chose to generate 100k images from each model, with an equal number of images per-class for class conditional models, and used the checkpoints and hyperparameters that achieved the lowest FID for each model; Appendix A details the full generation procedure. In total we assess each generative model across 17 metrics. We group the metrics by category here and provide full definitions in Appendix B
Metries for ranking generative models We include the well-known FID [48] which computes the Fréchet distance (FD) between sets of 50k real and generated samples in the representation space of the Inception- V3 network. We study FD in several alternative representation spaces, and refer to these by the model used to extract the representation from each image, e.g. FDCLIP. We include
alternatives to the FID such as the spatial FID (sFID) (76). FID [22 IS 196, kernel Inception distance (KID) g and the feature likelihood score (FLS) [53]
Metrics for diagnosing fidelity, diversity, rarity, or memorization As proxies for sample fidelity we consider precision [95 64) and density [75), while our human evaluation baseline, human error rate, provides direct measurement of fidelity. To quantify sample diversity we consider recall
[95] (4] and coverage (75) and for sample rarity we consider the rarity score [42). To study inter- vs.
intra-class diversity for class-conditional image generation we utilize the Vendi score [3]. While recall and coverage can also be determined per class, the small number of generated samples available for each class (c.g. 100 for each ImageNet class) results in difficulty constructing robust nearest- neighbour-based estimates. We also investigate a form of overfitting that we term memorization, in which models memorize individual images from their training data and emit them at generation time. We perform direct check for pixel-wise memorization for each generative model for each of our datasets and report this value as the memorization ratio in Section 5.2 We include automated metrics which claim to isolate memorization from the effects of overfitting mode collapse: the percentage of authentic samples [. Cr score [70]. and the percentage of overfit Gaussians from FLS (3).
Representation spaces for generative evaluation With the aim of finding more general perceptual representation space across the span of natural images, we employ a number of alternative encoders beyond the standard Inception-Ve V3 [003 trained for supervised classification ImageNet. First, as
3

---

more modern supervised benchmark we use a ConyNeXt-large architecture trained on ImageNet222 677. a larger dataset with more classes. We also include a number of self self-super feature extractors as alternatives to supervised learners. While such networks have proven to be useful for concurrent vision tasks including classification, object-detection, and segmentation [14 119,
remains an open question as to how the objective and augmentations used for training affect the representation space for generative evaluation. Thus we include seven self-supervised methods from variety of families [4] contrastive (SimCLRv2 19).. self-distillation (DINOv2 [80), canonical correlation analysis (SwAV [4), masked image modelling (MAE 44 and 2ata2vec 3.) and
Enguage-image (CLIP 187. using the OpenCLIP implementation [51 trained on DataComp- IB (35.. We also consider DreamSim [3], an ensemble of three self-supervised models (DINO [[5], CLIP, and OpenCLIP) that have each been fine- menettned to better align with human perception of image similarity using a dataset of human similarity judgments over image pairs. We design experiments to qualitatively and quantitatively understand their respective feature spaces.
architecture while for vision transformers (ViT) 28 we use ViT-L/14 as we found it provides a good tradeoff' between representation quality and computation cost. For both we select weights that were trained with the dataset and hyperparameters that achieved the highest linear classification ImageNet. Full details are in Appendix B.4
For -ased models we use the ResNet50 45
3 Human evaluation of generated data
The goal of our human subject experiments is to establish a large-scale, scientifically grounded baseline for image generation fidelity. Our experiments specifically target image realism, not the diversity of a model's ss learned distribution, nor whether samples are memorized, as realism is unequivocally a property for which humans can provide "ground truth". We find that the vast majority of models tested per dataset can be separated in terms of their realism with statistical significance, thus providing clear ranking which can be compared to calculated metries like FID. To our knowledge, this is the largest human subject experiment on the evaluation of generative models performed to date, with over 1000 paid participants, and 207k individual responses collected.
Experimental design We evaluated human perception of generated image realism for each of the 41 models across 4 datasets described in Section 2 Our experimental design was informed by experts and best practices in psychophysics, the study of the human perceptual system. It follows the design of experiments for the HYPE, metric [120] with several modifications to increase the quality of collected data and expand the scope of models and datasets tested. Each trial is a two alternative forced choice task where a participant is shown either a generated image from one model, or an image from the training dataset, and must choose if it is real or fake. Models were evaluated based on human error rate [20], the fraction of images which were incorrectly classified. This a simple metric with the intuition that models with better fidelity produce images that are more difficult to distinguish from real images. We take human perceptions of image realism as ground truth, noting the expansive efforts of the community to generate images that appear photo-realistic to humans, and will be used by humans. Full design details are provided in Appendix C
Concurrent work has also targeted human perceptual benchmarks for image synthesis on smaller subset of GAN and diffusion models [14 with 10 times fewer participants. While we are excited to see human benchmarks gaining popularity, we note number of concerns with their methods and thus downstream conclusions. Specifically, observers had no training period nor knowledge of the training set, and were tasked to judge whether images were "photo-realistic.. We believe this task contains much more ambiguity than our two alternative forced choice assessment, and introduces various response biases into participants' judgments. In their second user study, observers were asked to evaluate the relative realism of sets of images. This task is difficult to evaluate because participants can use single image in the set to guide their decision for the entire set.
Results and analysis The main results of our experiments are shown Figure 2 human error rate along with standard error for each model, and sort models on the x-axis by their FID (lower better). FID correlated well with human perception of fidelity, each plot would have a monotonically increasing trend, but this does not appear to hold. By inspection, the models with highest fidelity are almost always diffusion models, although GAN models often have lower FID, We find similar results for alternatives to the FID score in Appendix D.I
We plot the mean

---

Consistency
Diffusion
Flow
GAN
+
Transformer
VAE
CIFARIO
ImageNet
LSUN Bedroom
FFHQ
0.4
Human Error Rate
0.3
0.2
0.1
Lower FID
Lower FID
Lower FID
Lower FID
RESE RESELOW
ACC/
WCAND
R MHOC
IDDMA SOY SRONFOD LSGN styde
DiT-XI----o0 2-guided
RO- RQ-Thind Trand conaffrme
ADM
BIGAN
Mask:
ADMG-ADME
ADMO
LD GigGGN
ByLGANN-
DITXI-
Consistency
DDPM
iDDPH
Unlessh Salash-Traan
Trans
StyLGAN
Projezrd-GAs
ADM- ADM-drepout
Diff- Diff-RujGAN
Efficient-ve Mihiient-vVVbb
Unleash- Trans
MyyeGANN--ok
LDM
Projected-GAN -GAN
Inr
StyeSwin
StyleNAr
WG
Stgle
Figure 2: Human error rate models ranked by FID. Data is displayed the mean across participants with error bars showing the unbiased standard error.
To formally assess the relative performance of different model types, we separate participants' mean error rate into four one-way analyses of variance (ANOVA) - one per dataset - with model type (e.g. GAN, diffusion) as a between-subjects variable. We also performed planned comparisons between model types to probe omnibus effects. Analysis revealed effects of model type all four ANOVAS, indicating significant differences between model types' effect on human error rate (all F's 12.09, all p's < 0.001 (77). Post hoc comparisons of mean error rate between model types using Bonferroni correction are shown in Table I where indicates significant difference, indicates no significant difference, and model type is listed in order of descending mean error rate. Diffusion models were a clear standout for all datasets except FFHQ, where they performed par with other model types (noting that only GANs appeared in more than one experiment on FFHQ). Coupling the results in Table I with the FID rankings in Figure 2 conclude that current diffusion models produce the most realistic images according to human perception, but are downranked by FID.
Improved representation spaces for generative evaluation
4.1 Qualitative examination of perceptual spaces
To qualitatively visualize what parts of an image the Fréchet distance "perceives", we follow the gradient based visualization technique of (63), which focused on FID.
Table 1: Model types ranked by human error rate.
Dataset
Error rate ranking
Here we adapt and apply it to each of our CNN and ViT encoders. For CNN encoders our method is identical, while for ViTs we use the Grad-CAM variation introduced by 37.. Experimental details can be found in Appendix 1 Figure shows two visualized samples from each high-resolution dataset. We find qualitative differences between CNN and ViT architectures ViTs have a more global receptive field 88 but also find starkly different characteristics between supervised and self-supervised models. In agreement with 63 regions deemed important by Inception are far from optimal for datasets outside of the ImageNet domain. For FFHQ the important features according to Inception are typically not part of the person's face, while for LSUN-Bedroom the focus is on single object in the scene; features simply correspond with the Inception model's 's top-1 class prediction. We find that Inception does not perceive holistic view of images even its ImageNet training set, and see similar characteristics for CotvNeXt, indicating that such glaring issues are not mitigated by supervised training more modern architectures, larger datasets, nor larger numbers of classes.
CIFARIO Diff. > GAN > VAE
> Flow
ImageNet Diff. Transf. GAN
LSUN Diff. >
Transf. = GAN Consistency
FFHQ GAN Diff.
Transf VAE
Meanwhile SwAV and SimCLR often ignore important features, with SwAV exhibiting the strongest overlap with the classification networks. CLIP puts a large focus on the few main objects of an image - typically the main facial features (lips, eyes, etc.) or objects (beds) - which we presume an outcome of the language-image pretext arising from image captions that do not describe texture or finer details of the image. DINOv2 usually focuses on the image structure as a whole while still identifying objects of importance. We argue that this is closer to the behaviour would hope to
5

---

Image
Insepriion
ConvNeXi
Sw.V
SimCi Rv?
CT.IP
DINOV2
MAR
darativee
high
StyleGAN-XI
Samegyed
DIF-XI-2
low
Image
Inception
Cun-NeXi
SwAV
StmCLRv2
CLIPP
DINOv2
MAE
daizāvec
Micoopose
LDM
SPplSNAT
low
Image
Inception
Can-NeXi
SwAV
SimCI Rv?
CLIIP
DINOv2
MAF
daraávee
high
Projected GAN
ADM dripont
Figure 3: Heatmaps visualizing what the Fréchet distance "perceives" for each encoder. The sign of the heatmap is given by the activations of the saliency layer that visualized and does not reflect the sign of the gradient w.r.t. the FD - both bright yellow and deep blue can thus show an encoder's focus. Additional examples are shown in Appendix D.2.I
have from an encoder meant to evaluate images, as it emphasizes the important objects while still being able to pick up on elements elsewhere. MAE and data2vee, both trained using masked image modelling, have widespread focus textures and shapes, with an often smaller importance for the semantic information related to the main object. We exclude DreamSim from this analysis as it not straightforward to use Grad-CAM on its output, which is the concatenation of the output of multiple encoders. Appendix D.2.2 includes quantitative analyses, finding that masked models put more weight towards low-level image features rather than clustering classes by object semantics, while others (CLIP in particular) distill more object-focused representation space both in alignment with the qualitative analysis shown here.
In summary, we conclude that the representation spaces of sulf-supe methods are more ap- propriate for generative evaluation than supervised approaches, and that self-distillation (DINOv2) provides the best balance between focusing on important objects and holistie image structure.
4.2 The (mis)alignment of evaluation metrics and human assessment
In conjunction with our human evaluation baseline, we evaluated the ranking and diagnostic metrics outlined in Section 2 for each encoder and generated dataset. Figure (top) shows the relation of human error rate with FD and precision. We investigate diversity further in the following section. We include the correlation of 7 common metrics and human error rate (bottom, best viewed while zoomed in). Note that across encoders, FD, FD and KD are very highly correlated, resulting in essentially the same model rankings. This suggests that all these metrics provide sensible ways of quantifying distances between probability distributions, provided a good encoder is chosen. Overall, we find that despite some sample complexity issues, the FD metric when paired with an appropriate provides encoder provides a strong way of evaluating generative models, see Appendix D33 for details.
We find no strong correlation between human evaluation and any common metrics computed in the Inception representation space outside of the simplistic CIFAR 10 dataset, showing that Inception fails to encode perceptually relevant features for the larger, more diverse, and complex datasets that are the testbeds driving advancement of generative model development. In agreement with the previous section we find relatively poor performance of the AW model, which was proposed an alternative

---

Cansisessy
Dificiion
Flow
GAN
Tranffemer
YAE
Incophin
NeXi
SimCLRv2
DINOv2
CIAARO
0.3
Human brror Rate
2.22
Presiton
0.3
0.3
1SIU Redroom
0.3
191
Pre
Hon.
RD
Iun
ED
KD
Rec
Hun.
P
Ric
DP
Hun
KD
RAC
60
Deca
Ilmm
ED
Roc
P0 KD
Figure 4: Top: Fréchet distance, precision, and human error rate for each generative model as
measured by different encoders (columns) on different datasets (rows). Marker styles denote different generative techniques. Panels with a shaded background do not have strong (Irl 2 0.5) and significant (p S 0.05) correlations between FD and human error rate. Bottom: Pearson correlation of metrics over the three high-resolution datasets.
to the Inception model in 72 (in Appendix B.4.1 we also show poor alignment with a smaller CLIP-B/32 model investigated in a number of toy examples in 7 CLIP VIT-L/14, DINOv2, and MAE display far greater alignment with the human experiment baseline, as does DreamSim, except on ImageNet- which we believe to be a particularly important dataset, as it the most complex one being considered here, and it is thus used to train the most realistic models. Note that large precision values (which aims to quantify fidelity) do not consistently correspond high human error rate, even for encoders whose FD strongly correlates with human evaluation; precision is thus likely measuring more than just fidelity, and FD should be preferred over it. We include analogous results for recall in Appendix D.I. showing that it does not only capture diversity.
We find that diffusion models are often driving the discrepancy in alignment with human evaluation for the Inception network: FID prefers GANs over diffusion while FD determined by self-supervisg models trained on very large and diverse datasets does not. This discrepancy does not only oceur on non-ImageNer benchmarks, as previous works have shown for GANs, but is also true on ImageNet.
5 Alternative explanations: diversity, rarity, and memorization
We have found that FID does not correlate with human error rate, and have shown that replacing the Inception-V3 -3 network with an SSL encoder such as SINOv2-ViT-LAP both recovers the correlation of FD with human eтror rate, and results parts of images. While these are very promising characteristics for an evaluation metric, alternative explanations need to be ruled out before such a metric can be confidently adopted as a community standard. In this section we first verify that the lack of correlation between FID and human error rate not due to models with large human error rates lacking diversity, nor to humans wrongly classifying rare real images as fake. This justifies our use of human error rate as "ground truth" for fidelity, and confirms that lack of alignment with human error rate is a flaw of the encoder/metric pair. Then, we investigate whether the best performing generative models memorizing their training data.
metric which qualitatively focuses on the more relevant
7

---

GAN
Transformer
Diffusion
Train
RD Transa Transforme
ADMG-ADMM ADMU
DT--XL-2 guided
0.30
StyleGAN-XI
Human Ervor Rate
Vendi score
40
BigG
GjaOAN
Mask GI
ADM
ADND
Low
DIT-XL-2
Train
0.25
RQ- NQ-Dandeemee
ADMY
ADMO ADNAYY
ADMO-ADMILE
Vendi
20
GigaGAN
0.20
Mock-GrT
BiGGAN
OSNAGANAX
0.15
Lower FDDINOv2
Lower FDDINOV2
Figure 5: Left: Per-class Vendi scores of ImageNet models, in decreasing order of FDDINOV2 score. Right: FDDINOv2 on ImageNet, coloured by average per-class Vendi score (white corresponds to the train dataset).
5.1 Diversity and rarity
Diversity FD-based metries combine both fidelity and diversity into a single score, whereas human evaluation focuses only on the former. We must thus independently measure model diversity in order to confirm whether the lack of strong correlation between FID and human evaluation due to the FID score being flawed as an evaluation metric which focuses on fidelity, or if the discrepancy can simply be explained by high fidelity models having worse diversity. To decide between these two alternatives we explore the extent to which FDDINOV2 and FID align with diversity measures.
Our diversity analysis focuses on the Vendi score [33]. We justify this choice in Appendix D.4 and verify that the Vendi score meaning- fully quantifies diversity locally, but the same is not true globally. For example, Figure 6 dis- plays samples from DiT model on ImageNet with and without strong classifier-free guidance (cfg=4 and 1.5, respectively; we refer to the for- mer model as DiT-guided), where it is evident that DiT-guided exhibits much lower per-class semantic diversity. Yet, Appendix [D.4] shows that the overall Vendi score is higher for DiT- guided for almost all choices of encoders, while per-class Vendi scores are consistently lower for DiT--uudee across encoders. These results justify the use of the per-class Vendi score sensible diversity metrie; the overall Vendi score is mostly measuring inter-class diversity which is not particularly meaningful for class conditiani models such as the ones commonly used on ImageNet whereas the per-class scores focus on intra-class diversity consistently across encoders, and thus provide more meaningful quantification of semantic diversity.
DIT:XI-2-quided
DITXI -2
Figure 6: DiT-guided and DiT samples, labelled with per-class Vendi scores using DINOv2.
Equipped with the per-class Vendi scores, we evaluate the diversity of ImageNet models using the DINOv2 encoder in Figure 5) (left), where we can see that differences in diversity do not explain discrepancies between FID and human evaluations. For example, GigaGAN has diversity scores which are much farther away from those of the training data than most diffusion models, yet achieves a better FID (Figure 2 We see this as strong evidence of limitation of the use of the Inception network to measure fidelity with FID. We perform the same analysis using the FD metric with the DINOv2 encoder in Figure 5 (right): this evaluation metric is not only much more correlated to human evaluators, but diversity also better explains the few discrepancies between the two, c.g. the lack of diversity in DiT-guided results in a worse FDDINOv2 score than DiT despite having a better human error rate. Nonetheless, we highlight that the FDDINOv2 score emphasizes fidelity more than it does diversity (e.g. the DiT-guided model still obtains a very strong score).
Rarity To ensure that participants are not confusing "unrealism" with "unlikeliness" and assessing rare images as fake which would result in more diverse generative models ranking worse on human error rate we investigated whether the human error rate on each real image (individual images were evaluated by an average of 13 humans) was correlated with the image's "rarity score" [42]. Experiments are detailed in Appendix D.4 which show that human evaluators not confusing
8

---

"unrealism" with "unlikeliness", and thus human error rate is a sensible ground truth for image fidelity. Combined with the diversity analysis above, this rules out diversity as an alternative explanation for the lack of aligoment between human assessment and FID, and proves that alignment of FD metrics with human error rate is a desirable property for the studied generative models.
5.2 Memorization
Recent works have shown that diffusion models are particularly prone to memorization issues, in which models memorize individual images from their training data and emit them at generation time. Memorized samples may be near-pixel-wise identical, or semantically equivalent their source object while differing in termss of pixel-wise identity, the latter termed reconstructive memory [IO]]. Both predominantly occur either when the training set is small [13] or when there are a number of duplicate training samples [OO.] While it has been shown that diffusion and GAN models can memorize CIFARIO samples [TOT] a dataset that contains many duplicates c to our knowledge it is currently unknown whether this occurs on larger, more diverse, and higher resolution datasets such as ImageNet, FFHQ, or -UU-Beeoo and whether this affects any of the metrics we report. We set out to investigate this here.
Our experiments (refer to Appendix D.6p suggest that the larger datasets (e.g. ImageNett are not memorized by even the largest models we considered. This indicates that the models that are measured as superior in DINOv2 space are not capitalizing on memorization of the training data.
Collecting memorized samples We perform a direct check for pixel-wise memorization for each of the 100k images from each of our 41 generative models using the calibrated l2 distance proposed in 13 We find strong evidence on CIFARIO that most generative models exhibit exact memory and showcase set of memorized samples in Figure 7 We also report the memorization ratio of all models- - the proportion of generated samples that match source samples in the training set - to illustrate degrees of exact memorization across different models. We found no conclusive evidence of pixel-wise memorization by any model on ImageNet, FFHQ, or of models exhibiting reconstructive memory on ImageNet and LSUN-Bedroom [TOT. We find that less than 0.5% of DiT-XL-2 images exhibit close reconstructive memory. Although reconstructive memory on complex datasets is not necessarily a major concern, we recommend monitoring the memorization ratio, especially as models become more powerful and pixel-wise memorization becomes more likely. We include examples in Figure 7 (left) and additional visualizations and details in Appendix D.6 We note that our study was not explicitly designed to detect a more "copy- pppyy- approach wherein models memorize aspects of different training images and combine them when sampling, and thus we cannot rule out all forms of memorization. We leave such investigations for future work.
but found evidence
Evaluating memorization metries We evaluate the main memorization metrics in the literature in light of the memorized CIFARIO samples we uncovered: the percentage of authentic samples (AuthPct) [. the Or score (70), and the percentage of overfit Gaussians in FLS (FLS-POG) [53]. We first measure each metric's 's sensitivity
memorization in a controlled experiment, where we sample from the approximate posterior at different depths of a VDVAE's (2T) hierarchical structure to generate a collection of synthetic datasets that serve as increasingly less faithful reconstructions of the training set. Appendix I I contains the full description and results, and establishes that AuthPet, the Cr score, and FLS-POG are sensitive to memorization in an ideal scenario where all samples become increasingly memorized. In Figure] 7 (right) we investigate whether these automated metrics can differentiate models in practice, based on their measured CIFARIO memorization ratios. Here we used the DINOv2 representation space, while results for other encoders are shown in Appendix (D6.2 For most encoders, the Cr score trends in the correct direction, whereas AuthPet and FLS-POG trend inconsistently and fail to differentiate between models with different numbers of memorized samples.
By swapping the training set with a test ser that the model cannot possibly memorize, we check whether memorization metrics are sensitive to confounding properties other than memorization, such as fidelity or mode collapse. This experiment is not possible for FLS-POG as it does not use the training set, and for the Cr score we split the test set into two as it is already used. The results for Cr and AuthPet on test data are depicted with low opacity in Figure 7 (right). Surprisingly, both cases the results against the test set closely follow those against the training set, meaning that the CT score and AuthPet are dominated by some property other than memorization. Based on our analysis in Appendix D.6.3 we postulate that these metrics focus more on mode shrinkage and image fidelity

---

DDIM: CIFARIO
StyleGANZ-ADA: CIRARTO
Diffision
Flow
GAN
VAE
1.5
1.0
Memorization Ratio (%)
0.5
DIT-XL-2: ImageNet
90
55
60
AuthPet
1.5
ADM-dropout: LSUN-Bodroom
1.0
0.5
C7
modiiied
Figure 7: Left: Generated samples top with matched training samples the bottom showcasing exact memorization on CIFARIO and reconstructive memorization on ImageNet and Right: CIFAR10 models plotted by memorization ratio vs. metric. Memorization metrics against the test set instead of the training set are shown with low opacity.
(see Figure T.. respectively. From this analysis, we note that modifying the Cr score by swapping the roles the training and generated datasets makes insensitive to mode collapse. We include this modification of the Gr score in Figure (right).
In conclusion, find that none of AuthPet, the Cr score, FLS-POG is reliable metric for memorization. FLS-POG correlates poorly with our estimates of the percentage of memorized samples, while the CT score and AuthPct detect mode shrinking and image fidelity more than memorization. The reason behind these deficiencies is left to future work. Concurrent work [59] shows that in high dimensions moving the support of distribution and recall (measured using k-nearest neighbors), and the observed phenomenon here might have similar cause. Our recommended modification AuthPet, but still does not correlate well with the memorization ratio. Instead of using these metrics, we recommend researchers directly search for and collect memorized images using the calibrated 12-distance as described above, even though it is labour-intensive and requires tuning.
drastically change precision
the CT score improves on the Or score and
6 Conclusions
We carried out the largest and most comprehensive assessment of generative model evaluation metrics to date. We found that currently prevalent metries such FID not strongly predictive of human error rate, and that diffusion models achieve a higher human error rate than their GAN counterparts, yet often ranked worse according to FID. Our multiple investigations show that this discrepancy is not caused by model diversity, nor by humans assessing rare but real images as fake. Together, these findings imply that the differences between FID and human assessment are unfairly punitive towards diffusion models (in terms of assessing fidelity). We also showed that these deficiencies can be mostly addressed by replacing the Inception encoder by DINOv2 ViT-L/14. We include table with the FDDINOv2 score and many other metrics for all the models we considered in Appendix E which we hope will be useful as updated leaderboard of model performance. Finally, FD-based metrics are not designed to detect memorization, and we show that except on CIFAR1O. 10, the best performing models terms of FDDINOv2 are not memorizing their training data. In doing so, we also found that while the calibrated l2 distance proposed in [3] is reliable way to identify memorized samples, other metrics detect memorization are not ideal. We thus advocate for work proposing new generative models to report the ratio of memorized samples using the calibrated la metric alongside metrics computed with DINO-2-ViT-L/14 Because requires tuning, finding more automated metrics that reliably detect memorization will be productive avenue for future research.
10

---

References
[1] A. Alaa, Van Breugel. E. S. Saveliev, and M. der Schaar. How faithful is your synthetic data? Sample-level metrics for evaluating and auditing generative models. International Conference on Machine Learning, pages 290-306. PMLR, 2022.
[2] Azizi, S. Kornblith, C. Saharia, M. Norouzi, and D. Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.0886 084466 2023.
[3] A. Bacvski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli. Data2vee: A general framework for self- supervised learning in speech, vision and language. In International Conference on Machine Learning, pages 1298-1312. PMLR, 2022.
[4] R. Balestriero, M. Ihrahim, Sobal, Morcos, Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Tian, A. Schwarzschild, A. G. Wilson, J. Geiping, Q. Garrido, P. Fernandez, A. Bar, H. Pirsiavash, Y. LeCun, and M. Goldblum. A cookbook of self-supervised learning. WxXiv2234.122100 2023.
[5] A. Bardes, J. Ponce, and Y. LeCun. VICRegL: Self-supervised learning local visual features. In Advances in Neural Information Processing Systems, volume 35, pages $799-8810 10, 2022.
[6] B. Barz and J. Denzler. Do we train on test data? Purging CIFAR of near-duplicates. Journal of Imaging, 6(6):41, 2020.
[7] E. Betzalel, C. Penso, A. Navon, and E. Fetaya. A study on the evaluation of generative models. urXiv:2206.1092 2022.
[8] M. Binkkwski, D. Sutherland, M. Arbel, and A. Gretton. Demystifying MMD GANs. In International Conference on Learning Representations. 2018
[9] Bond-Tayles P. Hessey, Sasaki, Breckon, and G. Willcocks. Unleashing transformers: Parallel token prediction with discrete absorbing diffusion for fast high resolution image generation from vector-quantized codes. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XXIII, pages 170- 188. Springer, 2022.
[10] A. Borji. Pros and cons of GAN evaluation measures. Computer Vision and Image Understanding. 179:41-65, 2019.
[11] A. Borji. Pros and cons of GAN evaluation measures: New developments. Computer Vision and Image Understanding, 215:103329, 2022.
[12] A. Brock, J. Donahue, K. Simonyan. Large seale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[13] N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Schwag, F. Tramer, B, Balle, D. Ippolito, and E. Wallace. Extracting training data from diffusion models. arXi:2300.13188, 2023.
[14] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pages 9912-9924, 2020.
[15] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, Bojanowski, and A. Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEF/CVF international conference on computer vision, pages 9650 9660, 2021.
[16] G. Cazenavette, T. Wang, A. Torralba. A. A. Efros, and J.-Y. Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4759, June 2022.
[17] H. Chang. H. Zhang, L. Jiang, C. Liu, and W. T. Freeman. MaskGIT: Masked generative image transformer. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 1315-11325, 2022.
[18] T. Q. Chen, J. Behrmann, K. Duvenaud, and J.-H. Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, volume 32, 2019.
[19] T. Chen, S. Korblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong semi-suped supervuiee learners. In Advances in Neural Information Processing Systems, volume 33, pages 22243 22255, 2020. 2020.

---

[20] X. Chen. H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arX1v:2003.04297, 2020.
[21] R. Child. Very deep VAES generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2021.
[22] M. J. Chong and D. Forsyth. Effectively unbiased FID and inception score and where to find them. In Priceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 6070-6079, 2020.
[23] J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[24] E. L. Denton, S. Chintala, a. szlam, and R. Fergus. Deep Generative Image Models using Laplacian Pyramid of Adversarial Networks. In Advances in Neural Information Processing Systems, volume 28, 2015.
[25] Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pages 9780-8794, 2021.
[26] Dinh, Sohl-Dickstein, and S. Bengio. Density estimation using Real NVP. Conference on Learning Representations, 2017.
International
[27] C. Donahue, A. Balsubramani, J. McAuley, and Z. C. Lipton. Semantically decomposing the latent spaces of generative adversarial networks. In International Conference on Learning Representations, 2018.
[28] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Jakob Uszkoreit, and N. Houlsb. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
[29] D. Douglas, P. Ewell, and M. Brauer. Data quality online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA. PLOS ONE, 18(3)=e0279720, 2023.
[30] L. Ericsson, H. Gouk, and T. M. Hospedales. Why do self- supervised models transfer? Investigating the impact of invariance on downstream tasks. arXiv:2/1 1tt::2/11.113000 2021.
[31] P. Eyal, R. David, G. Andrew, E. Zak, and D. Ekaterina. Data quality of platforms and panels for online behavioral research. Behavior Research Methods, pages 1-20, 2021.
[32] J. Feydy. Geometric data analysis, beyond convolutions. PhD thesis, Université Paris-Saclay Gif-sur- Yvette, France, 2020.
[33] D. Friedman and A. B. Dieng, The Vendi Score: A diversity evaluation metric for machine learning. arXiv:2210.02410, 2022.
[34] Fu, N. Tamir, S. Sundaram, I.. Chai, R. Zhang. Dekel, and P. Isola. Dreamsim: Learning new dimensions human visual similarity using synthetic data, arXiv:2306.09344 09344,, 2023.
[35] Gadre, G. Ilharco, A. Fang, Hayase, G. Smymis, Nguyen, R. Marten, Wortsman, Ghosh, J. Zhang, E. Orgad, R. Entezari, S. Daras, Giannis adn Pratt, V. Ramanujan, Y. Bitton, K. Marathe, S. Mussmann, R. Vencu, M. Cherti, R. Krishna, P. W. Koh, O. Saukh, A. Ratner, S. Song, H. Hajishirzi, A. Farhadi, R. Beaumont, S. Oh, A. Dimakis, J. Jitsev, Y. Carmon, V. Shankar, and I. Schmidt. DataComp:
In search of the next generation of multimodal datasets. arXiv:2304. 14108, 2023.
[36] R. Geirhos, Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. ImageNetttrainedd CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. International Conference on Learning Representations, 2019.
[37] J. Gildenblat and contributors. Pytorch library for CAM methods. pytorch-grasd cam 2021.
[38] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 139414119--44, 2020.
[39] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf. and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
12

---

[40] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond. E. Buchatskaya, C. Doersch, B. Avila Pires. Z. Guo, M. Gheshlaghi Azar. B. Piot, k. kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent new approach to self-supervised learning. In Advances in Neural Information Processing Systems. volume 33, pages 21271 -12284, 2020.
[41] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and C. Courville. Improved Training of Wasserstein GANs, In Advances in Neural Information Processing Systems, volume 30, 2017.
[42] J. Han, H. Choi, Y. Choi, J. Kim, J.-W. Ha, and J. Choi. Rarity score: A new metric evaluate the uncommouness of synthesized images. In The Eleventh International Conference on Learning Representations, 2022.
[43] L. Hazami, R. Mama, and Thurairatnam, Pfficient-VDVAL Less is more. arXiv:2203. 2022.
[44] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[45] K. He. X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[46] D. Hendrycks, N. Mu, E, D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representations, 2020.
[47] K. Hermann, Chen, and S. Komblith. networks. In Advonces in Neural Information Processing Systems, volume 33, pages 19000-199999 1901,1915,,,,00 2020.
The origins and prevalence of texture bias in convolutional neural
[48] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge to local Nash equilibrium. In Advances in Neurul Information Processing Systems, volume 30, 2017.
[49] J. Ho, A. Jain, and P. Abheel. Denoising diffussin prohabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851, 2020.
[50] X. Huang, Y. Li, O. Poursaced, J. Hoperoft, and S. Belongic. Stacked generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, July 2017.
[51] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021.
[52] A. Jacot, Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018.
[53] M. Jiralerspong, A. J. Bose, and G. Gidel. Feature likelihood score: Evaluating generalization of generative models using samples. arXiv:2302.044: 02.00440, 2023.
[54] M. Kang, W. Shim, M. Cho, and J. Park. Rebooting ACGAN: Auxiliary classifier GANs with stable training. In Advances in Neural Information Processing Systems, volume 34, pages
2021.
[55] M. Kang, J. Shin, and J. Park. StudioGAN: A taxonomy and benchmark of GANs for image synthesis. XXiv:2206.09479) 2022.
[56] M. Kang. J.YY Zhu, R. Zhang, J. Park, E. Shechtman, S. Paris, and T. Park. Scaling up GANs for text-to- image synthesis. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, 2023.
[57] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data, In Advances in Neural Information Processing Systems, volume 33, pages 12104-12114 14, 2020.
[58] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 4401- 4410, 2019.
[59] M. Khayatkhoei and W. AbdAlmageed. Emergent asymmetry of precision and recall for measuring fidelity and diversity of generative models in high dimensions. MrXiv:2306.06618, 2023.
[60] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
13

---

[61] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[62] M. Kumar, N. Houlsby, N. Kalchbrenner, and E. D. Cubuk. Do better imagenet classifiers assess perceptual similarity better? Transactions on Machine Learning Research, 2022.
[63] Kynkäänniemi, Karras, M. Aittala, Aila, and Lehtinen. The role ImageNet classes in Fréchet Inception Distance. In International Conference on Learning Representations. 2023.
[64] T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, volume 32, 2019.
[65] D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1523-11532, 2022.
[66] Y. Liu, Z. Qin, Z. Luo, and H. Wang. Auto-painter: Cartoon image generation from sketch by using conditional generative adversarial networks, arXiv:1705.01903 2017.
[67] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A ConwNet for the 2020s. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11966- 11976, 2022.
[68] Lu, S. Wu, Y.W. Tai, and C.-K. Tang. Image Generation from Sketch Constraint Using Contextual GAN. In Proceedings of the European Conference on Computer Vision, September 2018.
[69] H. B. Mann and D. R. Whitney. On test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, pages 50 60. 1947.
[70] C. Mechan, K. Chaudhuri, and S. Dasgupta. A non-parametric test to detect data-copying in generative models, International Conference on Artificial Intelligence and Statistics, 2020.
[71] T. Mitra, C. Hutto, and E. Gilbert. Comparing Person- and Process- Centric Strategics for Obtaining Quality Data on Amazon Mechanical Turk. In Proceedings of the 33rd Anmual ACM Conference on Human Factors in Computing Systems, page 1345-1354, 2015.
[72] S. Morozov, A. Voynov, and A. Babenko. On supervied image representations for GAN evaluation. International Conference on Learning Representations, 2021.
[73] K. Nadjahi. Sliced- Wasserstein distance for large-scale machine learning: theory, methodology and extensions. PhD thesis, Institut polytechnique de Paris, 2021.
[74] K. Nadjahi, A. Durmus, P. E. Jacob, R. Badeau, and U. Simsekli. Fast approximation of the sliced- wasserstein distance using concentration of random projections. Advances in Neural Information Process- ing Systems, 34:1241 1-12424, 2021.
[75] M. Naeem, S. J. Oh, Y. Uh. Y. Choi, and J. Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pages 7176-7185. PMLR, 2020.
[76] C. Nash, J. Menick, S. Dieleman, and P. W. Battaglia. Generating images with sparse representations. arXiv:2103.038. 2021.
[77] J. Neter, M. H. Kutner, C. Nachtsheim, and W. Wasserman. Applied Linear Statistical Models. Irwin, 1996.
[78] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. International Conference on Machine Learning, volume 139, pages 8162-81 8171. PMLR, 2021.
[79] A. Odena, C. Olah, and J. Shlens. Conditional image synthesis with auxiliary classifier GANs. In International Conference on Machine Learning, volume 70, pages 2642-2651, 2017.
[80] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafranice, V. Khalidov, P. Fernandez, D. Haziza, Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnacve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. arXiv:2304.07 0003,0077...00 2023.
[81] M. Otani, R. Togashi, Y. Sawai, R. Ishigami. Y. Nakashima, E. Rahtu. J. Heikkilä, and S. Satoh. Toward verifiable and reproducible human evaluation for text-to-image generation. arXiv:2304.011 001661 2023.
14

---

[82] G. Parmar. R. Zhang, and J.-Y. Zhu. On aliased resizing and surprising subtleties in GAN evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11400- 11410, 2022.
[83] Pavlovia, http://www..arllovi.rrrg 2023.
[84] W. Peebles and S. Xie. Scalable diffusion models with transformers. arXiv:2212.09745 2022.
[85] J. Peirce, J. R. Gray, S. Simpson, M. MacAskill, R. Höchenberger, H. Sogo, E. Kastman, and J. K. Lindeløv. PsychoPy2: Experiments in behavior made easy. Behavior Research Methods, 51(1):195-203 Feb 2019.
[86] Prolific.
huts://www.prolfff..comm 2023.
[87] A. Radford, J. Kim. C. Hallacy. A. Ramesh, Goh, Agarwal, Sastry. A. Askell, Mishkin. J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-88 8763. PMLR, 2021.
[88] M. Raghu, T. Unterthiner, S. Korbllihh, C. Zhang, and A. Dosovitskiy. Do vision transformers see like convolutional neural networks? In Advances Neural Information Processing Syxtems, volume 34, pages 12116-11 12128,, 2021.
[89] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hicrarchical text-conditional image generation with CLIP latents. arXiv:2204.06 06, 25225, 2022.
[90] A. Ramesh, M. Paviov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot 2021.
text-to-iman image generation. In International Conference on Machine Learning, pages
[91] S. Ravuri and O. Vinyals. Classification accuracy score for conditional generative models. In Advances Neural Information Processing Systems, volume 32, 2019.
[92] D. Rezende and S. Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pages 1530-1538 1538. PMLR, 2015,
[93] R. Rombach, A. Blattmann, D. Lorenz, P. Esser. and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11684-16695, 2022.
[94] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, Ghasemipour, R. Gontijo Lopes. B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffuusion models with deep language understanding. In Advances in Neural Information Processing Systems, volume 35, pages 66479-36494 2022.
[95] M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, volume 31, 2018.
[96] Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, Improved techniques for training GANs. In Advances in Neural Information Processing Systems, volume 29, 2016.
[97] A. Sauer, K. Chitta, J. Müller, and A. Geiger. Projected GANs converge faster. In Advances in Neural Information Processing Systems, volume 34, pages 77480-17499 2021.
[98] A. Sauer, Schwarz, and A. Geiger. StylcGAN-XI: Scaling StyleGAN to large diverse datascts. In ACM SIGGRAPH 2022 Conference Proceedings, 2022.
[99] J. Snell, K. Ridgeway, R. Liao, B. D. Roads, M. C. Mozer, and R. S. Zemel. Learning to generate images with perceptual similarity metrics. In 2017 IEEE International Conference on Image Processing, pages 4277 4281, 2017.
[100] J. Sohl-Dickstein, E. Weiss. N. Maheswaranathaa. and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages 2256-2265 PMLR, 2015.
[101] G. Somepalli, V. Singla, M. Goldblum, J. Geiping, and T. Goldstein. Diffusion art or digital forgery? Investigating data replication in diffusion models. arXiv:22/2.03860 2022.
[102] Y. Song. P. Dhariwal, M. Chen, and I. Sutskever. Consistency models. arXiv:2303.01469, 2023.
15

---

[103] C. Szegedy, V. Vanhoucke. S. loffe. J. Shlens, and Z. Wojna. Rethinking the Inception architecture for computer vision. In Proceedings the IEEE Conference on Computer Vision and Pottern Recognition. pages 2818-2826, 2016.
[104] R. Turner, J. Hung. E. Frank, Y. Saatchi, and J. Yosinski. Satopolis-Hastting generative adversarial networks. In International Conference on Machine Learning, volume 97, pages 6353, 2019.
[105] P. Upchurch, Gardner, G. Pleiss, R. Pless, N. Snavely, K. Bala, and K. Weinberger. Deep feature interpolation for image content changes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, July 2017.
[106] A. Vahdat and Kautz. NVAE: Deep Hierarchical Variational Autoencoder. Advances Neural Information Processing Systems, volume 33, pages 19667-19679, 2020.
[107] A. Vahdat, Kreis. and Kautz. Score-based generative modeling latent space. In Advances in Neural Information Processing Systems, volume 34, pages 11287-11322 2021.
[108] van Breugel, T. Kyono, J. Berrevoets, and M. van der Schaar. DECAF: Generating fair synthetic data using causally-aware generative networks. In Advances in Neural Information Processing Systems, 2021.
[109] Walton. A. Hassani, X. Xu, Wang, and Shi. StyleNAT: Giving each head a new perspective. a.X1v:221..05770. 2022.
[110] Wang, H. Zheng, P. He, W. Chen, and M. Zhou. Diffusion-GAN Training GANs with diffusion. arXiv:2206.02262 2022.
[111] Wu, Donahue, Balduzzi, K. Simonyan, and Lillicrap. LOGAN: Latent optimisation for generative adversarial networks. arXiv:1912.0095,, 2019.
[112] Xu, Z. Liu, Y. Tian, S. Tong, M. Tegmark, and Jaakkola. PFGM+: Unlocking the potential of physies-inspired generative models. arXiv:2302.04255, 2023.
[113] Yang. Shen, Y. Xu, and Zhou. Data-efficient instance generation from instance discrimination. arXiv:2106.04566 2021.
[114] M. Yang, C. Yang, Y. Zhang, Q Bai, Y. Shen, and B. Dai. Revisiting the evaluation of image synthesis with GANs. arXiv:2304.01999 2023.
[115] Yu, A. Seff, Zhang. S. Song, T. Funkhouser, and Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv: 506.03365, 2015.
[116] Zhang, S. Gu, B. Zhang, Bao, D. Chen, F. Wen, Y. Wang, and B. Guo. StyleSwin: Transformer Based GAN for High- Resolution Image Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11304-11314, June 2022.
[117] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas. StackGAN: Text to Photo- Realistic Image Synthesis With Stacked Generative Adversarial Networks. In Proceedings of the IEFE International Conference Computer Vision, Oct 2017.
[118] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features a perceptual metrie. In Proceedings of the IEEE Conference Computer Vision and Pottern Recognition, pages 586-595, 2018
[119] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. iBOT: Image BERT pre-training with online tokenizer. International Conference on Learning Representations, 2022.
[120] S. 7hou, M. Gordon, R. Krishna, A. Narcomey, I.. Fei-Fei, and M. Bersstei. HYPE: A benchmark for Human eYe Perceptual Evaluation of generative models, Advances in Neural Information Processing Systems, volume 32, 2019.
16

---

Broader impact While we do not develop or release any generative models in this work, our work can be used to improve the quality generative models in the future. Such generative models, despite having number of beneficial uses, have the potential to be used for negative applications such as the generation of deepfakes. We note that all generative models, encoders, and datasets explored in this work are already publicly available assets.
Limitations We take human perceptions of image realism as a ground truth for measuring image fidelity, noting the expansive efforts of the community generate images that appear photo-realistic to humans, and will be used by humans. But, for certain generative applications, such as the use of generative models as augmentations to improve performance of classifiers (2), humans are not always the end users. In this case the end user is the downstream classifier, and models that achieve the best DINOO score may not directly translate to optimal improvements in downstream classification. A Additionally, while the image datasets (and hence generative models) in our study were chosen to span a wide range of natural images, they were all 3-channel RGB format of human-identifiable objects. is an open question whether our proposed methods of evaluating generative models translate to e.g. medical images or astronomical images, which are also RGB, or more broadly to scientific data of N-dimensions.
Compute Experiments were conducted without requirement for significant computational re- sources. Generating datasets was performed either on NVIDIA Titan V GPUs with 12GB of RAM or small local cluster, and required total of ~ 100 hours of GPU time, significant fraction of which was spent generating images from the DiT and L.DM models. The analysis was conducted on NVIDIA Titan V GPUs with 12GB of RAM, and computing all metrics for each encoder and generative model required 448 hours on a single GPU.
Assets and code As stated in the main text, to facilitate further development of generative models and their evaluation we have prepared a public release of all generated image datasets, human evaluation data, and modular library to compute 17 common metrics for 9 different encoders at btttps://gi tiggtbbb.cc.. com ham/layerbail labs/dgm--aae /gmddvaaa We provide detailed descriptions and links to the public generative models, metrics, and encoders used in Appendices A and B
A Generated datasets
For each generative model across the four datasets studied in this work we generated 100,000 images, unless otherwise stated. For class-conditional models generated equal number of samples from each class - 10,000 per class for CIFAR 10 and 100 per class for ImageNet. We chose the checkpoints and set parameters (if applicable) to the that achieved the lowest FID stated in the paper, or on the project's github if not otherwise stated, For GANs we set truncation=1. 1.0 throughout, and for class-conditional diffusion models we set the guidance parameter to the reported values. To validate that our set of images for each model was generated correctly we compare the FID calculated with our code (using 50k samples) to the reported value in the paper in the code release. Any discrepancies were investigated by contacting the authors or raising issues on the project github.
When downsampling images from their native sizes the ones used in this work note that significant differences in FID can result from different choices of interpolation methods. As such, report the interpolation techniques we used to downsample ImageNet and FFHQ to 256 256 pixels, and note that these are required for reproducing our results when using the original images (the resized datasets are provided along with our code). Differences in FIDs and the reported values in various papers could partially be attributed to this effect- for example using Box filtering instead of Lanczos increases the FID of StyleGAN2-ada on FFHQ by 1.0.
Below we list (i) the generative models used for each dataset, (ii) links to the project pages and checkpoints used, (iit) the reported FID in comparison to the value determined from our generated images, and (iv) any special notes.
A.1 CIFAR10
StudioGAN
models
5 httpa://ggitiub..om//PSSSSCCC
-Sturch-Stu
checkpoints are in folder httpp::/hhuginngaacc..
17

---

CIFAR10_tailored/ We used current weights from each folder.
ACGAN 79
Reported
FID=33.39 ours=35.47
BigGAN-Deep ott.aacciaaaaaaggg..aaaaa0000220 Re-
ported FID N/A, ours=3.91.
LOGAN
|11
Reported
FID=20.65, ours=17.87.
ReACGAN (FA.apo....aca..................
Reported
FID=3.87, ours= 4.40.
MHGAN 104
Reported
FID=3.95, 4urs=4.22.
WGAN-GP 41
Reported
FID=53.98, ours=26.25.
LSGM-ODE [077
https: http:s//ggtttub.coom/////a///L We used the FID checkpoint,
sampling with ODE framework. R2pommead110022221000,00112211.
iDDPM-DDIM 78
http:///gghhllocccogrrraaa/ppppa Reported
FID=2.94, ours=3.27.
PFGM++ 112
http:://gghhuu..co//////bbbeebrp
We used
the following checkpoint
http::/dtive.ggggl... http://ooll...crrriieeeeoe htt.v//oovveee/////////ooooa/s// /tdp
[tl.::/coo..c...gg.....aaaaaaaa1 our9=1.79.
Reported FID 1.74,
RESFLOW
18
Reported
FID-46.37, ours= 29.
NVAE [16]. Reported FID N/A, ours=32.53.
StyleGAN2-ada 57
https://github.co http://shhhhaa.cooaaaaaaaaattatp
We used the following checkpoint ottp:://rrlaaa--ficcd....ddddd com/
StyleGAN-XL
98
We used the following checkpoint
http:://....aac...aaa111aaooooom com
http:////..aaaaaeoooooaaooaaaar/ Reported ou.s
A.2
ImageNet
Four models used sets of 50k publicly available images provided athttps://github com/
ADM 25
Reported FID=10.94, ours=11.84
ADMG 25
Reported FID=4.59, ours=5.58.
ADMG-ADMU 25
Reported FID=3.94, ours=4.75.
BigGAN 12
Reported FID=6.95, ours=7.94.
DiT-XL-2
84
http://gttaat.cccaccccooooreeere
Reported FID=2.27,
ours=2.80.
DiT-XL-2-guin 84] This model was equivalent to the above, but used stronger classifier free guidance term cfg 4.0 instead of cfg 1.5. As described in Section [.1 this model was included to study the effects of model that produces very realistic images, but sampled with a lower intra-class diversity, Reported FID=N/A, ours=17.24.
GigaGAN (56] with 100k images provided privately by authors. Reported FID==.45,,
ours=4.16
LDM http:://ggttaaacooooo//////atttt Reported FID=3.60,
0ur9=4.29.
Mask-GIT httpep///gthhnnooooooooeerrrrerr We used
the following checkpoint http:::/stoorgeeggoo// htw.:/toorrrr.gggooogeaaaa..oo.o http://pppp.....mm/mmmmiiaappppa
chw.spoinccchhcccninccco Reported FID=6.06, ours=5.63.
18

---

RQ-TTaanQtae T--ransformer
65
We used the 1.4B model from the htw.ooooooogggghoooooooooppp///a
net/
4B_rqvae__500 50e. tar .g htt.:aaa__.gaaalaii.ooaaaa......
StylcGAN-XL
98
We used the following checkpoint httpp::/s33 h
central- www.:a...aaaaass....mmm com
wWg-projectss/tylegaa http:///gaaa..emmmmme/////////aa pk1
Reported FI1)=2.26,
ours=2.91.
A.2.1 A note on the curation of ImageNet
The treatment of the raw ImageNet dataset 23 often not explicitly described in the litera- ture particularly for building generative models on ImageNet and we have found some in- consistencies on how FID is reported. Therefore we describe our exact approach to calculating FID on ImageNet, and provide parsing scripts with the goal of standardizing the process across papers. The full dataset was obtained from https::/pp///. hww.aeeebbbbeee----olliiiiiaoooo
//ww: kaggle com/ competitions the data no longer available from the ImageNet website (https: / image- org. index. php).
Raw ImageNet is dataset of 1,000 classes, each with roughly 1,300 images. The images themselves are generally non-square, with image height and width both always at least 256. To get from the raw rectangular ImageNet images to the 256 x 256 resolution of ImageNet256 we perform the following two operations:
1. Center crop along the long edge. This results in square image of side length N min H, W). given individual image height and width W.
2. Downsample from x N to 256 x 256 using bicubic interpolation, as suggested by [2 We also experimented with Lanezos interpolation with only minimal change the results.
This procedure results in roughly 1.3 x 10th images of size 256 x 256. We construct reference batch of 100,000 training images matching our generated sets by sampling 100 images from each class without replacement. We then calculate FD using 50,000 images drawn without replacement each from the training set and the generated set. These choices result in a slight increase in FID values for most of the ImageNet models reported above, as most determine the FID) of 50k generated samples and 1.3MM training samples, but do not affect any model rankings.
A.3 FFHQ
Following the examples set in the StyleGAN repositories, we downsampled the original 1024 1024 FFHQ dataset 256 x 256 using Lanezos interpolation.
Efficient-vd VAE 43
We used the 8-bit version
from the following checkpoint latpps//storage
googleapis com fitessa-ppbli---leeeffeeeeeeeeee httpa///eeeeaaaaaaaaaa.ooooo/aar
2 Reporte N/A, ours=34.88.
Insgen
BB
http:://githuu..cc//eeeffrrr/// http://le..ommiee///////////////
We used the
following
checkpoint
ES60EtiSddR16-ggann 2500Ef/acccsggoo0000/aaa//eennnn Repp FID=3.31. 30r ours=3.46.
LDM 93
Reported FID=4.98,
111. To our knowledge the settings we used generate the images were consistent
with the codebase and with those used in the paper. We contacted the authors about this via
email (as the GitHub page is full of unresolved issues) but received no response.
Projected-GAN
Re-
ported FID=3.39, ours=3.46.
StyleGAN2-ada 57 httte///sshhh.ooo////////////sae
We used the following
checkpoint http::/wwbbs--ficch...waa.
ffhg
1llg700-pape2266 wwwppp5pppp-... -dw-555--da---0 pkl
Reported FID=4.30, ours=5.30. The discrepancy
19

---

is purely due to different choices for downsampling: We used Lanczos filtering downsample the original 1024x1024 pixel FFHQ images to 256x256, while their work used box filtering. Box filtering exactly reproduces the results reported in the StyleGAN2-ada paper, and we thank the authors for their work help solve this discrepancy. Please see the following github issue for more details httpp://giithb. com http:///wwa///..aaaaaaaaaaaaaaaa
StyleGAN-XL. [98 http:://gggghhi...ooooooooooooaa
We used the following checkpoint httpp::
http::::://.3....ctntteee centrall www..a.aaaanaas... com/ FID=2.19, ours=2.26. 1.0=22.99.
StyleNAT http:://gtthab.coo//llllaaaabbbp
We used the following
940k_flip. pt. Reported FID=2.046,,
StyleSwin
116
We
used the 10D2222E0WuaVVRRE////aaaa2222DD2
following
checkpoint ftttps://drive.goggee. com file/d/
Reported FI1)=2881,
ours=2.89
Unleashing-TTansffrrmer
https: com/:/mb-mbababbaa. t/ We used the following checkpoint httpss
srleessinggtranstff tran ormers
//github. com/ NVlabs wabs//fhq--a fatttffaggddaetttt dataset
Reported FID=7.12, ours=9.02. We note
that other discrepancies with the reported FID on FFHQ have previously been raised via the issues function GitHub.
A.4 LSUN-Bedroom
Four models used sets of 50k publicly available images provided atthttps://githtii com/ openai/ spenai/guided wtt.aaiiii///eeee//aaaaaaaaaaaaa
[25]
ADMMNctdrrppuut 25 Reported FID=1.90, ours=2.20.
DDPM
49
Reported FID=4.89, ours=5.
iDDPM Reported FID=4.24, ours=4.54.
78
SSyleGAN 58
Reported FID=2.35, ours=2.65.
Consistency 70)
We used the consistency training model provided at htttts:///ttthttt comm/ppena//coniiseenng. cicteenncymooeelsssst models Reported FID=7.85, ours=8.27.
Diffusionn-rroeectedd GAN [10]. We used the pretrained projected GAN model pro- GAN Reported FID=1.43,
vided at httttps://gtthubb com/ Sachenddnng-aag/DDieee ours=1.79.
Projected GAN 977. We used the pretrained projected GAN model provided at https //github. cwm/
gan Reported FID=1.52, ours=2.23. The settings we used to generate the images were consistent with the codebase defaults, which were, as far as we could tell, consistent with those used in the paper. We contacted the authors both via email and the issues function of GitHub but received response.
Unleashing Transformers [ We used the pretrained model provided by the authors at htttp:///githu: com/ saaa-tmmmmunaa Baanb--//unlasshnng.. tengg-trrasfformee. Reported FID=3.64, ours=3.58.
B Metrics and encoders
To assess all aspects of generative models we include number diagnostic metries that designed to directly quantify the fidelity, diversity, rarity, or memorization of our 100k image samples. Such metrics are valuable for diagnostic purposes but difficult use to directly rank generative models. Ranking requires metrics which group these concepts into single value without clear tradeoff. In total we assess each generative model across 17 metrics, both ranking and diagnostic. These metrics are computed in a number of supervised and self-supervised representation spaces. In total we include 9 encoders. Throughout this section, we denote generated samples as {a. = and real samples as {x [=
20

---

The following appendix (i) details each of the metrics, (ii) presents controlled experiments and additional checks for diversity and memorization metrics, and (iit) describes the encoders and motivates the choice of ViT model size used.
B.1 Metries for ranking generative models
The Fréchet distance (FD) uses (Jr, E,,) and (Ug. Eg)) the sample mean and covariance of the real and generated representations, respectively, and determined through:
FD
FD Er: Mg 2g) II,, My + Tr (E,(E,, (E, -ig - 2(E,,,)*)
(I)
The FD corresponds to the Wasserstein-2 distance between Gaussians with the corresponding means and covariances, and is thus a valid metric between the first two moments of real and generated distributions. If these two distributions happen be characterized by their first two moments (c.g. they are both Gaussian), then the FD becomes a metric between distributions, not just their first two moments. We use 50,000 images for both the real and generated images. To move away from the standard Inception- V3 network which [T03]] used as a feature extractor we drop the "I" from this metric and the following, where appropriate.
FDoc FDxe 22 aims remove the inherent bias of the FD due to the finite number of samples.
is determined by evaluating the FD at 15 regular intervals over the number of samples N, from N 5,000 to N 50,000, fitting linear trend the 15 data points, and using the trend to infer FD value at N oo, FD...
The spatial FID (sFID) 76 is the FID computed using a representation from the intermediate mixed 6/conv layer of the Inception-Ve V3 network [103] trained for ImageNet1k (23), rather than the standard (poo13, 2048 dimensional) layer. The sFID relies on the Inception- V3 network, and thus we do report it for other architectures.
sFID
The Kernel distance (KD) 8 aims replace the FD with something that is proper distance between distributions regardless of whether the distributions are characterized by their first two moments. In order to this, unbiased estimate of the maximum mean discrepancy [39 is used:
KD
KD ((a)) Nan (-))==)) 11 1)
Σ
m(n
(2)
where k is a positive definite kernel that chosen as a hyperparameter. We use the standard 3rd degree polynomial kernel.
IS The Inception score (IS) 196] is maximized when the entropy of the distribution of labels predicted by the Inception- model is minimized for every input (i.e. generated image) and when the predictions are evenly distributed across all 1000 possible labels. It is given by:
IS (f=:)1 = = Σr_ (=(pppl=)))ll<())
33)
where p(y[z) denotes the output of the Inception-V2. V3 network when I is the input, and p(y) the observed frequencies of labels in the training data. We use a refactored scoreoon-ocrr--ytorrc.e
version from https:: /ttps:://itthin giggtu....... com/ wtw.uuu..o//bbbrraaaaaaaaaaaaeae inception_ www_sscore.. py to compute this metric. The IS relies on the Inception V3 network, and thus we do not report for other architectures.
The feature likelihood score (FLS) 53 is recently proposed density estimation method that requires training, generated, and test samples. The goal FLS is be as close to likelihood evaluation even when there is no likelihood available. In order to do this, the FLS fits kernel density estimate (KDE) to the generated samples (after having been transformed by the encoder). The bandwidths of the KDE are chosen as to maximize the log-likelihood of training data, and the FLS is then given by (an affine transformation of) the test likeliiel obtained by the KDE. We the default implementation from httph:ttp:///etm htmm/ttps://ggthiie gigggthubb.... httphs///gghhbbb..omaarrooooiii htt.a//cooomaarroir///ll.....
FLS
21

---

B.2 Metrics for fidelity, diversity, and rarity
Precision, recall, density, coverage As proxies for sample fidelity we use precision 99
64
and density (75. Both precision and density are based upon nearest neighbours computed in representation space. Precision counts the binary decision of whether a generated sample contained in any neighbourhood sphere of training points, while density counts how many real- sample neighbourhood spheres contain the sample. To quantify sample diversity we use recall [95] 64] and coverage (75], which similarly to their counterparts precision and density, are based upon nearest neighbours computed a representation space. Formally, precision given by:
precision ([=))= -
Σ
(x) S({)))=1)),
(4)
where 1-) denotes the indicator function, 5(12))7=1) - B NND, (4f)), where (xx,r) denotes Fuclidean ball centered with radius r, and NND,, (xt)) is the distance between 23 and its htth nearest neighbour {af) }= excluding itself. Similarly, recall given by:
Σ recall ((x)) li {= = m
1 (2) ((f2))=)))
(5)
3=1
density is given by:
density ((=))==1, (x5))==
(of e B(D,,
(6)
kn
and coverage is given by:
coverage ((=9)=n
max Ilonn
(of B(g,, (ND,(
m
Precision, recall, density, and coverage are determined with the code provided at https://tttt tiub com/ www.ooaaaaaageeeattrrreoooeo evaluation -prde associated with the work [75] We use nearest neighbors (k- = 5) and 10,000 samples throughout, as standard.
Rarity score To determine the "rarity" of an individual image use the rarity score 42 to precision, recall, density, and coverage, the rarity score uses k-nearest neighbours:
Similar
rarity (ar, {-)))=)))
min
NND (47),
(8)
(=)
where [ax (=)))
m E B(cJ, NND
Vendi score To study inter- intra-class diversity for class-conditional image generation we utilise the Vendi score [3 This score does not require a reference dataset. Formally, the Vendi score is given by:
VS (5((=))/=1 = -i À.
where X; is the 7t eigenvalue of the n X matrix K/n, where Ki (3) for positive semi- definite kernel I (chosen as a hyperparameter) with k(x, 1. The convention that Olog = is used to compute the Vendi score. The Vendi score over the whole dataset reports the overall sample diversity, and when conditioned on class quantifies the intra-class diversity of samples. We separate the in order to study mode concentration of class conditional models we find some models have high inter-class diversity, which is encouraged through the class conditioning, while lacking intra-class diversity. While recall and coverage can also be determined per class, the small number of generated samples available for each class (e.g. 100 for each ImageNet class) results in difficulty constructing robust seaarettnnighhouu-bbeeeeee linear kernel to compute the score. We also tried using the same polynomial kernel that we used for computing KD, but found no clear benefit from doing so.
estimates. As per the original paper, we use
B.3 Metrics for memorization
We also investigate form of overfitting that we term memorization, which models memorize individual images from their training data and emit them at generation time.
22

---

AuthPet AuthPet [ deems each generated sample either as authentic or inauthentic and returns the fraction of authentic samples. Inauthentic samples are those for which the distance the nearest point in the training set is less than the distance between that training sample and its nearest neighbour the training set.
CT The OT score 70 is a global three sample non-parametric test explicitly designed to detect memorization. It uses the training set, the test set, and the generated samples, and summarizes how often training samples are closer to generated samples than they are to samples in the test set through Mann-Winne Whitney hypothesis test [69]. The data is split into number of cells using k-means clustering. the score is determined in each cell. then the per-cell score is averaged into the final one. We use the default implementation from https: //gittbb com/ casey- rasey-meehan -eehann dataa copying namely k 3 and preprocessing the representation space using 64 component PCA.
FLS-POG We denote the percentage of overfit Gaussians determined in FLS as FLS-POG (53) For each generated sample we compute the difference between the train and test log-likelihoods and report percentage of generated samples that had a higher ligelioolo under the training set than the test set. We use the default implementation from httmmtt/t///ttttt
commmarcojiral
Memorization ratio with calibrated l2 distance sample of is given by:
The calibrated l2 distance 101 of generated
NN (2 INN - NNN
Uzg, (-
(10)
k
k
where N (x)) denotes the hth nearest neighbour (in Euclidean distance) to x in the training data without i.e. ) {z). Note that this metric is computed directly on pixel space, with no encoder being used. Low values of (=f. {-5) (=)))=)) indicate that 2 memorized a training sample. A threshold r is and the generated sample 2. is then considered as "memorized" if its calibrated l2 distance is smaller than 7. The memorization ratio is given by the percentage of memorized samples:
" memorization ratio ((=))== (=,]),, = Σ 1 (M(=) (af))))) <T)
(11)
While the memorization ratio can provide good assessment of how much a model memorized its training samples (Section [.2), no default configuration of the k and hyperparameters achieves this across all models and datasets: these values need to be hand-tuned in order to meaningfully flag memorization. Thus, despite our recommendation use this metric detect memorization, do not believe that it should be used as a basis to compare models.
B.4 Encoders
With the aim of finding more generalized perceptual representation space across the span of natural images we employed a number of alternative encoders. Here we outline the specific implementations used, and provide evidence for the ViT-L/14 architecture choice. For self-supervised methods include short descriptions on the family training employed, grouping into categories based on those of
ConvNeXt As a more modern supervised benchmark we use a ConvNe architecture trained on ImageNer22k (67. We use the timm implementation provided here https: g thub com/huggingface/ppree pttp/ bataggemodeess/lllle htt//mainn/ttt http://ainiiimmoooeee///////// lect the convnext aaaaneet_largee.aoteeee large_ in22k model.
py and se-
SwAV SwAV [4] featured heavily in previous work advocating for the use of self- supervised feature extractors for generative model evaluation [72]. as at the time it was a state-of-the-art self-supervised image representation model. SwAV is trained on Ima- geNerlk using canonical correlation analysis method, and learns by clustering data while enforcing consistency between cluster assignments produced for different views of an im- age. We use the ResNet50 architecture and weights from (72), with code provided
23

---

at http::/.w................oooooop BeIL.BpperrBB0..aa...aaarreeeee
is contrastive method which learns visual representations by encour- aging the similarity between two often heavily augmented views of an image, while discouraging the similarity between distinct training images. SimCLRv2 is CNN trained on ImageNetlk. We used the PyTorch implementation available at hhttps://githhi com
SimCLRy2 SimCLRv2
[9]
CLIP CLIP is a multi-modal language-image model 87 which trains an image encoder and text encoder to predict which images were paired with which texts. We use CLIP ViT 87.. ELIP-B/32 was studied for generative evaluation in a controlled experiment in (7] (while they do not state the architecture used in the paper we found it at lattps://gite main/fcc ffdd py). In the next section we show that this choice is sub-optimal, and that a better choice is the OpenCI.IP ViT-L/114 implementation [5] trained on DataComp-1B 35 Thus, we use OpenCIIIP
com eyee eyyalbetzalle ell /dd/////////// blob/
ViT-L/14 trained on DataComp-13
DINOv2 DINOv2 [80] is from the self-diiilllaiiondiiciinnaaaiii family, which learns representa- tions by passing two different views of an image to two encoders and mapping one to the other by means of predictor. DINOv2 is ViT-L/14 architecture trained large custom dataset of 142M images built by combining common datasets for classification, segmentation, depth estimation, and retrieval tasks, plus additional images from the internet. We justify the architecture choice alongside CLIP in the following section.
MAE Masked autoencoder (MAE) 44 is from the masked image modelling family, and learns to directly reconstruct masked image patches. We use the ViT-I PyTTorch implementation from which was trained on ImageNetIk.
is from the masked image modelling family, and learns by predicting latent representations of the full input data based on a masked view of the input. We use the Hugging Face implementation found here tettps://uugiiin
data2vec data2vec B]
wtt//ocss/taassoorerrseseeee/eee data2vec We use ViT-L, which was trained on ImageNet1k.
DreamSim The DreamSim method (34] uses dataset of human similarity judgments over image pairs in order to fine-tune encoders to better align with human perception of image similarity. DreamSim was publicly released after the initial submission this paper. We also highlight that the focus of our work is different than DreamSim: our human trials focus on image quality rather than similarity, and our subsequent analysis uses these human judgements to evaluate current metrics for generative models, while DreamSim creates a dataset of human-peat pereeeivd image-to-image similarities and designs an encoder that reflects these human judgements. In other words, DreamSim focuses on obtaining an encoder which maps images that humans assess similar to nearby points in latent space, whereas we focus on finding an encoder where distances between probability distributions on its latent space, such FD, correlate with human judgment of fidelity. Our experiments the full DreamSim ensemble, the concatenation of the representations of three different self-supervised models (DINO [15. CLIP, and OpenCLIP) that have each been fine-tuned the DreamSim dataset.
We note that while separating our encoders into such self-supervised families allows for a broad understanding of the different methodologies employed during training and hints potential charac- teristics at their learned representation spaces, models can in practice employ a mix of approaches. In particular, DINOv2 also employs masked modeling objective, however it not masking images but rather the latent-space with a teacher network used to provide targets [8].
B.4.1 Determining ViT model size for generative evaluation
To determine which model size provides good perceptual space for generative evaluation with ViT-based models we calculated FD and precision of the 11 ImageNet generative models under varying model sizes of CLIP and DINOv2 encoders. For CLIP we also studied models trained using different sets of training data (OpenAI LATON-40MM, LIAON-2B, and DataComp-11 while DINOv2 encoders all used an identical training set. CLIP models not denoted with OpenAl were from OpenCLIP. While larger models achieve better accuracy under linear evaluation protocol
24

---

CLIP-B92 OpenAi
CLIP 8:32
CLIP-B/IS
CLIP-L114 LAION-JB
CLIP-/144 Duscomppll IB
CLIP-GF14 LAION-JB
LAION-20 28
LAION-IB
Hunas Eaver Rute
0.30
0.25
03
0.29
FD
FD
FD
Canditaa EalgeW TD
DINDV2-8/14
DIN0-2-8/14
DINOv2-L/14
DINOx2-g/14
V
Thmun Erroe Rate
0.50
1.0
100
0.25
0.20
w0000
0.15
10000
Figure 8: ImageNet results function of VIT model size for CLIP (top) and DINOv2 (bottom).
on ImageNetIk, which may be evidence towards using them for generative evaluation, argue that the computational cost of such a large model restrictive to the development generative models. Often, metrics need to be tracked during model development for numerous training setups, hyperparameter settings, and epochs, which motivates networks with a lower computational cost. For reference, workstation with single NVIDIA Titan V GPU, determining the representations of 50k samples takes ~4 minutes with ViT-B/14 B/14, ~ minutes with ViT-L/14, and ~55 minutes with ViT-G/14. Thus, smaller encoders that exhibit representation spaces matching the utility of larger encoders should be preferred.
Figure 8 shows the results of the model size sweep. Models are listed by ascending size. For CLIP we find that CLIP-B/32-00 0/32-OpenAI, -LPP-B/3--LIION2B, and CLIPBB/I6-LAAOOOO 1/------II show much smaller correlation between FD and human error rate than the ViT-L/14 and ViT-G/14 models. We see that the model best aligned with the expensive ViT-G/14 the highest performing CLIP model under linear evaluation protocols is correlation matrix shown on the right (the precision matrix, while not shown displays the same trends), and find very high correlation between CLIP CLIP-L/14DattaComp IB and CLIP-G/14-B G/14-LAION2B.
1B. We quantify this agreement with the FD
For DINOv2 we see lower alignment of FD and human error rate when using the VIT-S/14 model in comparison to the larger three, and also find good agreement between the ViT-B/14, ViT-L/14, and ViT-G/14 models. We again quantify this agreement with the FD correlation matrix shown on the right (the precision matrix, while not shown displays the same trends), and find that the ViT-L/14 shows the highest correlation with ViT-G/14, although ViT-B/14 is highly correlated as well
As evidenced by both the CLIP and DINOv2 results, ViT-L/14 provides a perceptual space nearing the ImageNet qualities of the largest encoders a much lower computational cost. Thus we believe that ViT-L/14 provides a good tradeoff between representation quality and computation cost, and use it as the basis for all ViT-based encoders in this work. While we propose DINOv2-L/14 as the encoder that best improves upon Inception-Ve V3, we suggest that the strong correlation of DINOv2-B/14 yet factor of ~4 lower computational cost, will provide useful during generative model development metrics can be tracked using this model size during training, and final leaderboard values can be reported after training has concluded using DINOv2-L/14, Our codebase allows to easily do this.
C
Human subject experiments
C.1
Test description
As noted Section 3 the goal of our human subject experiments was evaluate the fidelity of generated images in a scientifically grounded way across many models and datasets. The design of our tests was therefore specifically tailored to address the realism of generated images, and not other aspects of the modeled distributions, such as diversity. We further emphasize that our focus is on the realism of images that are generated for human use, not other uses (for example generating synthetic data to improve classifier performance [46), to debias datasets [108), or for dataset distillation [16]).
25

---

de
leaniig ariiical genca
Cheaayy will
and lite
li sall, se ve the monem mon servent
Sore the moge vey raing cohora spcies
This grout tie pipiiarrs stin marry mtgee
llage canate AuA..
pudiiie dess sel basten cnti coli
Faam sond mito
prowe you than a slude perts I.
Preas SPRLEEE to sua rose or mis ragee tress
Freastaain der complayy by dartacd
Dany Enenar
preccaaa wis you lede
cut
200
paastic make sica
Ale in laye wition now carercanee
fades
Figure 9: Screens displayed participants during our StyleGAN-X XL evaluation for ImageNet. Other experiments followed the same template. The bottom left shows the format of the 40 practice and 200 test trials, with the correct answer text shown only after a participant entered their response.
We found that the experiments from [120] for the HYPE, metric supported these goals, and thus used them as template. We first describe our method in detail, compare the differences from 120 and then present our full results.
In each test, generated images from a single model are directly compared to images from the training dataset an absolute comparison [8]. Previous human experiments have used comparative ratings
rather than absolute, for example by directly comparing pairs of models [99] 105 117
94
or
grouping together several models with baseline and comparing all at once 24 99 The absolute approach has several advantages:
96
50
97
Models tested against single, consistent baseline.
The number of experiments scales linearly with the number of models to be tested. Pairwise testing requires quadratic number of experiments.
Experiments be extended with new models in a consistent manner additional experi- ments do not depend on previously used models.
Comparisons of new results against previously- gathered results are analytically simple using between-subjects comparisons. The comparative paradigm would require mixing between-sune subjectss and within-subjects designs.
The performance of any given model does not depend which other models were tested in the same experiment.
Scores can indicate whether models generate images indistinguishable from the baseline, if there is room for improvement (see Figure 2 where the optimal error rate of 0.5 is approached only for CIFAR10).
Experiment results will not be outdated when researchers make progress with new models.
In total, 41 distinct experiments were conducted, each consisting of an instruction and training phase, followed by a testing phase. Research on how to collect high quality data from crowd sourcing platforms [1]] shows that providing training on the task to be completed is the best way to improve
26

---

data quality, while providing financial incentives can also work, but pre-screenig individuals by their on- on-platform reputation is ineffective. Hence, in the first phase we provide participants with background information, request for consent, and clear instructions on their task, including the fact that exactly half of the presented images would be real. We then conducted 40 practice trials which were not used in the analysis of our results. Each trial was a two-alternative forced choice task, in which the participant was shown a single image which was either a "real" image from the training dataset, or "fake" image generated by the model being tested. The participant was simply asked to judge if the image was real or fake with no alternatives, and they were provided feedback on the correct answer directly after they responded. In the testing phase, subjects 200 trials of the exact same format. Participants were unconstrained in their time to view and judge each image. There was a 200 ms inter-trial interval before presentation of the subsequent image, and a delay of 100 ms after image presentation before participants could enter a response to prevent accidentally advancing through trials. For reproducibility, in Figure 9 we show screenshots from an example test, the StyleGAN-XI X-I, model on ImageNet, as they were displayed to the participants. The experiments were created using PsychoPy [85] and hosted on Pavlovia [83]. The experiments were only administered on desktop or laptop computers, not on mobile devices or tablets, and image size was scaled to the user's display.
shown an additional
The choice to reveal correct answers to participants after they guess is not completely standard across the literature. Some authors have provided feedback [96] 120 50). while others have avoided it [117 [6], and often no justification is provided. Ultimately, we want participants to make the most accurate decisions they are capable of, which is why we use long training phase, and provide feedback throughout. The main argument against providing feedback during testing is that participants may "learn" and adjust their responses, which may in effect change the distribution of experimental data throughout the test. We do not find this to be concerning for the following reasons. First, our long training period is meant for participants learn and improve their performance on the real task; most learning will occur early during the training phase, less so during the testing phase. Second, even without feedback, participants can still learn and change their approach to the task as they see more examples. Third, with or without feedback all participants have the same opportunity to learn, and so our experimental datapoints, the error rates of individual participants on 200 trials, are drawn from the same hypothetical distribution. In preliminary experiments we observed no sigmificant difference between the error rates on the first and second halves of the test trials, showing that learning during the test phase does not create large effect.
Ethical considerations Our institution does not have an internal review board (IRB) approval requirement for running crowd-sourced experiments, and thus has no formal approval process for experiments such as ours. In place of an IRB approval we followed an informal process internally prior to running the experiments, as recommended in the NeurlPS 2023 guidelines. The process involved: confirming what internal review processes existed and were required of us at our institution, ensuring that none of the images that would be shown to participants had any explicit or offensive content, informing participants that they may be shown images that could trigger common phobias, taking the test ourselves, and paying the participants above minimum wage.
Participant recruitment and compensation Human subject experiments for generative model evaluation are quite common, but often are ad-hoc and cannot be considered as randomized, controlled experiments. Sometimes authors themselves are the participants [48] 68], or volunteers known the
authors are used [24] 117 (66 97], raising questions of bias. Luckily, there are several crowd-sourcing
platforms recruit willing and unbiased participants from the general population. For our experiment, participants were recruited from Prolific [86] which has been shown to provide the highest quality data in scientific studies about behavioural research platforms [3] 29]. While we did not filter eligible participants by their on-platform reputation, we did require participants to be fluent in English, the language used throughout the experiment, and to have completed college, or university degree at least at the Bachelors level (we used this filter to maximize the probability of fluency in English, and thus of instructional obedience during the trials). No other filters were used, and statistics on the demographics about consenting participants are shown in Table Comparisons between models and datasets are therefore netween-subjects, For each experiment, which took a median of 15 minutes to complete, participants were paid at median rate of E7.60/hr, including base rate for completing the study which complied with the payment standards of the Prolific platform, and an incentive of f0.001 for each correct answer.
27

---

Table Demographics and performance for the 897 of 1036 participants who voluntarily consented to release their demographics through Prolific. The normalized error rate accounts for the varying difficulty of tasks over different combinations of (dataset, model). A negative value means better performance on the task, with a value of meaning 1 standard deviation above the mean within their task cohort of ~2 participants. As intuitively expected, there is no significant performance difference based on demographic group.
Group
# Participants
Normalized error rate
19-29
574
-0.91=0.93
30-39
200
-0.18=0095
Age group
40-49
77
-0.01=1.07
50+
0.52
Female
389
0.01
Sex
Male Prefer not to say
507
-0.05-
Doctorate degree
19
-0.3
Graduate degree
294
0.10++0.98
Highest education level
Technical/community college
121
0.02+1.02
Undergraduate degree
463
-0.09=00996
Africa
127
0.21
Asia
40
0.19
Europe
557
00.0
Continent of birth
North America
129
-0.19=0.99
Oceania
13
-0.46=0.99
South America
29
-0.36=0.87
Unknown
2
0.42+
Asian
44
0.34
Black
103
0.35+
Ethnicity
Mixed
86
-0.10=0.96
Other
44
-0.26=0085
White
620
-0.08=0.96
The experiments for each of the 41 models were completed by at least 25 unique recruits (see Tables through 77 totalling 1,036 different people, and providing over 207k individual test responses (along with more than 41k practice responses not included in our statistics), making this the largest human study evaluating generated image quality that we know of A small number of experiments were completed by more than 25 participants due to unstable connections to the Prolific and Pavlovia servers causing logging issues.
Image selection Within each experiment, the images shown a given participant were selected and randomized as follows. First, 2,000 images from the training dataset were randomly selected without replacement with classes in proportion the dataset if applicable. The same of training images were used for each model corresponding to that dataset. In a similar fashion, 2,000 model samples were selected (see Section and Appendix A in proportion the dataset's class distribution if the model was class-conditional. For each participant, 100 real and 100 generated images were selected random from these pools, again respecting class distributions where applicable. The number of times each generated image was viewed by a human therefore follows a binomial distribution with randomized
25 and p 0.05. The 200 selected images were presented the participant order. The 40 practice images were chosen in a similar fashion, but had no overlap with the 4,000 test images, and were reused for all participants. Appendix D shows image samples for each model.
"The next largest similar experiment that we know of collected about 138k individual responses [120] Human evaluations have also been conducted on text-to- tittt--mmage generation quality, where (90) collected about 40k individual responses, (94] collected about 25k, and more recently [8] collected about 33k responses.
28

---

C.2 Comparison to HYPE,0
Our experimental design was inspired by the HYPE&c score [20], but we made several changes to improve the quality of collected data. We summarize the differences in Table 3 We doubled the number images shown to each participant for more accurate estimation of each participant' emor rate, while slightly lowering the number of participants per model. We used Prolific rather than MTurk because research shows that Prolific can provide better data quality due their participant screening practices [31 29]. Prolific also requires fair base pay rates based on time, compensating participants solely on their number of correct responses disallowed. The pre-screening test from [20] used mix of data from all models they tested, which does not align with our preference for absolute comparisons and extensible experiments given in Appendix C.I so we did not use it.
C.3 Results
Fach of the 41 models were evaluated based on human error rate 120]. the fraction of the 200 test images which were incorrectly classified. This is simple metric with the intuition that models with better fidelity produce images that are more difficult for humans to distinguish from real images. Detailed results by dataset that were summarized in Figure and Table T
from the main text are given
in Tables 4 through 7 Along with the overall error rate, was also provide the error rates the real and fake images separately. As [120], we observe that the error rate on real images increases for models with higher fidelity, even though real images are drawn from the same pool of 2000 examples
across models. We also provide the mean rate that participants guessed "real", noting that they were instructed that exactly half of presented images would be real. Finally, give the mean time spent on the 200 test examples.
Table 3: Comparison between design of HYPE. experiments 120 and ours.
Ours
HYPE
Correct answer revealed to participant after their response
Yes
Not specified
Number of real/fake images in selection pools
2000
5000
Number of images shown each participant
200
100
Number of participants for each experiment
> 25
30
Number of experiments conducted
41
13
Number of responses collected
207k
39k
Recruitment platform Payment
Prolific
MTurk
Base pay f1.80 #0.001 per correct answer
SI for passing qualifying test + $0.02 per correct an- swer on main test
Qualification
Fluent in English, minimum education
Achieve 265% accuracy on test using mix of data from all models tested
requirement
29

---

Table 4: Detailed human experiment results CIFARIO
Model
# Tests
Error Rate f R Error Rate Error Rate f R Answer Rate Time (s)
ACGAN
25
0.148 + 0.013 0.181 + 0.021 0.116 + 0.016
0.467
217
BigGAN
25
0.387 + 0.014 0.381 + 0.021 0.393 = 0.022
0.506
284
IDPM--DDIM
26
0.400 = 0.013 0.390 + 0.019 0.410 = 0.022
0.510
268
LOGAN
25
0.206 = 0.020 0.204 = 0.021 0.208 = 0.026
0,502
220
LSGM-ODE
27
0.437 0.010 0.417 = 0.021 0.456 = 0.018
0.519
232
MHGAN
25
0.336 = 0.015 0.331 = 0.016 0.341 = 0.021
0,505
267
NVAE
25
= 0.128 = 0.015 0.134 = 0.025
0.503
240
PFGM+
25
0.436 0.011 0.422 = 0.024 0.450 = 0.025
0,514
243
ReÁCGAN
25
0.336 0.016 0.326 0.017 0.345 = 0.026
0.510
247
RESFLOW
25
0.088 + 0.008 0.090 0.010 0.086 = 0.011
0.498
199
StyleGAN2-ada
25
0.393 + 0.012 0.387 0.020 0.399 = 0.022
0.020
0.506
328
StyleGAN-XI.
25
0.399 + 0.012 0.378 + 0.018 0.420 = 0.022
0.521
251
WGAN-GP
25
0.170 = 0.013 0.158 = 0.013 0.181 0.022
0.511
221
Table 5:
Detailed human experiment results on ImageNet
Model
Tests
Error Rate
Error Rate
Error Rate
Answer Rate Time (s)
ADM
25
0.266 = 0.016 0.254 = 0.021 0.279 + 0.018
0.513
369
ADMG
25
0.248 = 0.012 0.236 = 0.021 0.259 = 0.016
0.512
396
ADMG-ADMU
25
0.269 = 0.016 0.240 = 0.017 0.298 = 0.024
0.529
344
BigGAN
25
0.158 + 0.014 0.155 + 0.016 0.161 + 0.019
0.503
326
DiT-XL-2
25
0.286 + 0.016 0.266 + 0.026 0.307 + 0.024
0.521
341
DiT-XI--2-uuied
27
0.330 + 0.013 0.329 + 0.019 0.330 0.017
0.500
352
GigaGAN
25
0.212 # 0.015 0.212
0.500
398
LDM
26
0.309 + 0.017 0.294 + 0.023 0.323 0.022
0.515
369
Mask-GIT
25
0.183 + 0.016 0.164 0.0188 0.203 + 0.025
0.520
278
RQ-Transformer
26
0.223 1 0.012 0.205 1 0.014 0.242 1 0.017
0.518
308
StyleGAN-XL
25
0.153 + 0.013 0.145 = 0.014 0.162 + 0.018
0.509
315
Table 6: Detailed human experiment results on
Model
# Tests
Error Rate 1 R Error Rate F Error Rate t R Answer Rate Time (s)
ADM-dropout
25
0.387 +: 0.020 0.334 +: 0.027 0.440 + 0.022
0.553
306
Consistency
25
0.155 + 0.022 0.131 + 0.026 0.179 + 0.027
0.524
325
Diff-ProjGAN
26
0.148 + 0.019 0.130 + 0.018 0.167 = 0.023
0.519
335
DDPM
25
0.267 = 0.021 0.259 + 0.026 0.276 = 0.021
0.509
354
iDDPM
25
0.297 0.021 0.263 = 0.025 0.331 0.028
0.534
336
Projected-GAN
25
0.142 + 0.018 0.144 0.022 0.140 + 0.017
0.498
266
StyleGAN
26
0.217 0.018 0.179 0.017 0.254 0.025
0.537
427
Unleash-Tres -rann
25
0.209 + 0.022 0.178 + 0.024 0.240 = 0.027
0.531
305
Table 7:
Detailed human experiment results on FFHQ
Model
# Tests
Error Rate R Error Rate ? F Error Rate 1 R Answer Rate Time (s)
Efficient-vdVAE
25
0.088 + 0.010 0.081 + 0.013 0.096 + 0.010
0.507
282
InsGen
25
0.255 + 0,018 0.275 + 0.025 0.236 + 0,016
0,480
326
LDM
25
0.186 0.195 0.016
0.016 0.178 0.022
0.492
429
Projected-GAN
25
0.120 0.012 0.107 0.011 0,134 + 0.018
0,514
318
StyleGAN2-ada
25
0.215 0.017 0.215 0.019 0.216 0.021
0.500
359
StyleGAN-XI.
25
0.160 + 0.012 0.163
0.158 + 0.021
0.498
340
StyleNAT
25
0.299
+ 0.019 0.314
0.283
0.485
387
StyleSwin
25
0.180
+ 0.198 + 0.023 0.161
+ 0.010
0.481
288
Unleash-Traad
27
0.173 + 0.015 0.142 0.015 0.203 0.022
0.530
330
30

---

Table 8: Layer used for GradCAM for each encoder.
Model
Layer Name
Inception
blocks.3.2
ConvNeXt
sogess3.blocks..
SWAV
layer4.2
SimCLRv2
net.4.blocks.2.nee
CLIP
visual. hest..rrannfommeeeelllessssssss
DINOv2
blocks.23.nom
MAE
blocks.23.norml
data2vec
model.encoddrrlaeee...... before
D Additional experimental results
D.1
Additional scatter plots and correlations
Figure didisplays additional views of Figure showing trends as function of other popular metrics (recall, precision, and density), while Figures and T2 display the correlation of additional metrics.
D.2
Representation spaces for generative evaluation
D.2.1 Experimental details GradCAM and additional heatmaps
We follow the gradient based visualization technique of 63 As the FD is defined only between large sets of images (50k), the method pre-computes the representations of 50k images from the real dataset, and for set of 49,999 generated images, then adds the additional image of interest using Grad-CAM to visualize the parts of the image that have the largest influence on FD. For ViT-based encoders, we use the Grad-CAM variation introduced by (37] The layer GradCAM was applied to for each encoder can be found in Table
To unify the color scheme across encoders, changed the sign on the heatmap for CLIP, DINOv2, ConvNeXt and SimCLRv2. Indeed, the sign of the heatmap is given by the activations of the saliency layer that visualized and does not reflect the sign of the gradient w.r.t the FD. Both bright yellow and deep blue can thus show an encoder's 's focus. The sign was changed for the listed encoders as it seemed to make better sense with the semantics of the images observed.
Additional heatmaps for all encoders on Imagenet, FFHQ and L.SUN-Berrood can be seen in Figures 13 and [ 14 On those heatmaps, the "generated" set is taken to be the real dataset, and not from any specific generative model.
D.2.2 Quantitative representation spaces
In conjunction with the preceding qualitative analysis we performed two quantitative analyses of the representation spaces, and display the results in Figure [5] First, measured the perceptual score in representation space using the BAPPS 2AFC dataset 118), data from a two alternative forced choice (2AFC) test that asked human evaluators which of two distorted views is more similar reference. The perceptual score measures the alignment of the human preference with the relative distances between the reference and patches in representation space. In the "traditional" experiment patches were distorted using traditional photometric, noise, blur, spatial, and compression distortions, while another experiment distorted patches using "CNN-based" distortions such as autoeneoding, denoising, colorization, and super-resolution. Interestingly, better classifiers achieve worse perceptual scores [62]. We find that SimCLR ranks worst on the perceptual score while the masked methods of MAE and data2vec score the highest. These results are perhaps unsurprising given their respective self-supervised training procedures the contrastive SimCLR relies on a of heavy augmentations while MAE and data2vec work closer to pixel space and do not use augmentations. but such results point to clear differences between perceptual spaces as direct result of the self-supervised training objective. We note that for generative evaluation, distortions such jitter do not necessarily change the semantic information of the image and thus should not change the representation. Such distortions minor fraction of the total dataset.
31

---

Condstency
Diftuson
How CL.IP
GAN
Itandomer
VAB
Inception
CoevNeXi
SwAV
SinCTLRv2
DreamSim
DINOv2
MAE
date2vee
CHEARIO
0.4
Imatee
0.5
Human Laror Rate
0.2
Socall
11110
LSUN Bedroom
0.2
Lowv:: FD
Lower FD
Lower FD
Lower FD
Lawer FD
Lower FD
Lower FD
Lowel FD
Lower FD
Consistency
Difluios
Flow
GAN
Trunsormer
a VAE
Inception
ConyNiei
SwAV
SinCLRv2
CLIP
DreamSim
DINOw2
MAE
data2voc
CHEARIO
94
InageNer
0.3
Human Livor Rato
0.2
Douity
LSUN Bedrevm
0.2
Lower FD
Lower FD
Lower FD
Lower FD
Lower FD
Lower HD -
Lower FD
Lower FD
Lower FD
Consistency
Dicfusion
Floww
GAN
Truusformer a VAIl
Inception
CoevNeXi
SwAV
SimC1.Rv2
CLIP
DreamSin
DINOV2
MAE
daraZvoc
CIFARIO
0.
ImagoNer
0.3
Humar laroe Rats 1
0.22
Coverage
LSUN Bedroom
0.2
Lowve: FD
Lower FD
Lower FD
Lower FD + Lowes FD
Lower
Lowe FD
Lower FD
Lower FD
Figure 10: We show analogous results to Figure with points coloured by additional metrics. Fréchet distance and human enror for each generative model measured by different encoders (columns) on different datasets (rows). Panels with shaded background do not have strong (|rl 2 0.5) and significant (p 0.05) correlations between FD and human error
32

---

100
Figure 1I: Pearson correlation for all metrics using different encoders (rows) on different datasets (columns). Missing metrics require validation test set, which datasets do not have.
CIFAR10
ImageNet
FD
FD SEID KD
FLS
Prec.
Rec.
Dens.
Cov.
AuthPet
CJ mod ELS-POG
Hum. CZ.
FFHQ
LSUN Bedroom
FD
FD SFID KD
Free.
Rec.
Dens.
Cov.
AuthPet
mod
FLS-POG
Hum.
Rec.
Dens
Cow.
AuthPet
C
C mode
PL FLS- POG
AutaPet
PL PLS-POO POG Hum.
Hum.
Figure 12: Pearson correlation for additional metrics (including IS and sFID) for the Inception- V3 network. Supporting the conclusions of the main text, we find that these additional metrics are not strongly correlated with human evaluation on datasets more complex than the simplistic CIFAR10. Missing metrics require validation or test set, which some datasets not have.
33

---

Inception
ConvNeXi
SWAV
SimCLRv2 Rv2
Figure 13: Additional GradCAM visualisations (1/2).
34

---

CLIP
DINOV2
MAE
data2vec
Figure 14: Additional GradCAM visualisations (2/2).
35

---

Correlation intra-class ImageNet diversity
Panch
Rehermes
Pach
Pixel
0.14
0.03
0.42
0.00
Inception
0.14
1.00
0.80
0.74
0.73
0.66
0.31
0.20
0.07
ConvNeXt
0.15
0.80
1.00
0.76
0.63
0.79
0.18
0.07
0.01
CLIP
0.03
0.74
0.76
1.00
0.54
0.69
0.13
0.08
0.09
SwAV
0.32
0.73
0.63
0.54
1.00
0:41
0.60
0.53
0.23
DINOv2
0.69
0.02
0.03
Percepnall Score
0.8
MAE
0.42
0.31
0.18
0.13
0.60
0.02
1.00
0.68
0.22
0,7
dala2vec
0.41
0.20
0.07
0.08
0.53
0.08
0.68
1.00
0.40
Trdiiional
CNN
SimCLR
0.00
0.07 0.01
0.09
0.23
0.03
0.22
0.40
1.00
SimCLR
ConvNeXt
DINO
Sway
Inception
CLIP
dataDyoc
MAE
Pixel
Inception
ConvNeXi
CLIP
SWAy
DINOv2
MAE
data2vee
SimC
Figure 15: Left: Perceptual score
622 for each encoder measured using the BAPPS 18 dataset.
Samples from the dataset are shown above, where v denotes the patch that humans judged to be closer to the reference. Right: correlation of intra-class diversity on ImageNet.
To probe the level of semantic and pixel-wise diversity captured by each encoder we calculate the intra-class Vendi score over each class of the ImageNet training set (including in pixel space). This score is proportional to the effective number of independent samples within the class after transforming the representation space. Section D.4 provides more details. low score reflects a low diversity within the class, which is an indicator of feature space that overemphasizes small set of class specific features rather than the wide diversity of semantic in pixel-space. The correlation of the score over all classes across our set of encoders in shown in Figure [5] Sorting the encoders by their correlation with Inception we find two distinct groupings. The masked models, MAE and data2vec, have intra-class diversity that is highly correlated with each other, but also with the diversity as measured in image-space, This indicates that such representation spaces put more weight towards low level image features rather than clustering classes by object semantics, which is quantitative agreement with the characteristics of the heatmap visualizations. Interestingly MAE achieves high clasiiiiasiiiication accuracy but performs very poorly on k-nearest neighbours evaluation protocols 44) perhaps also indicating a lack of clustering based on class-specific semantic information. ConvNeXt, CLIP, SwAV, and DINOv2 show high correlation with Inception yet low correlation with image-space, indicating that their feature spaces distill a more object-based semantic information. Of the ViTs, CLIIP has the smallest alignment with pixel space diversity and the largest. with Inception space.
D.3
FD bias
It is known that using finite datasets for computing FID (and similarly for FD in general) results in biased estimate the quantity that one would obtain if an infinitely large amount of data was observed. As mentioned in Appendix B. FID, (22) aims to remove this inherent bias. Here we measure the biases of FD and FD in realistic setting with a number of encoders used in our work in order to test whether the default 50k generated samples are sufficient for our proposed replacement to the Inception-V V3 feature extractor.
We use the 100k samples generated with DiT-XL-2 84 trained on ImageNet-256, and calculate the relevant statistics as function of the number of training samples used by sub-sampling both the DiT generated set and the set of training images. Figure T6] shows the results across 4 encoders Inception, CLIP, DINOv2, and MAE, We observe that while FDx has less bias compared to FD, does not completely solve the problem, and continues to decrease with the number of samples used. We also find that the self-supervised encoders exhibit less biased FD than the Inception network. For example, DINOv2 with 20k samples has the same relative error compared to the full 100k set
36

---

Convergance FD FD. Accoes Eacedles
Convergence Using Inception
CLIPED
FD
CLIP Fi,
10
DINOK2 FD
DINOV2 FD.
Relanse Error
MAGTD
Relative Erur
10
MAE FD.
10"
Ineggcon FiN.
10
10
10
5
Samples 10000
Samples 1000
Figure 16: Left: Convergence of FD and FD using different representation spaces. Right: Convergence of FD and ASW using Inception as the encoder.
as Inception does using 50k samples. While one could argue that this means only 20k samples are required for DINOv2, we retain consistency with FID and use 50k samples for FD with all encoders.
Naïvely, the FD thus has two issues: the aforementioned bias, and the fact that it only compares the first two moments of its corresponding distributions (see Appendix [B.I]. Since the FD can be understood as approximating its distributions with Gaussians of matching moments and then computing the Wasserstein-2 distance, W2, between them, it sensible to consider other metrics based on Wasserstein distance as potential evaluation metrics. Unfortunately, computing this distance is in general computationally expensive 32
In an attempt to address these issues, we considered using the sliced Wasserstein distance, SW2, 2. instead. Let IP and Q be distributions distributions, respectively. Let u (-) denote the push-forward of measures under the mapping (u, z). Then SW2 is defined through:
Rd, which should think of as the generated and real
SW (I,, Q) E,, [W uy Q]]
(12)
where the expectation is taken with respect to the uniform measure on the unit sphere in TR...
can be shown that SW2 2 has lower sample complexity compared both W2 and its relaxation Calculating optimal transport in one dimension requires significantly less computation than
773.
higher dimensions as it can be solved via sorting. Therefore, the RHS of Equation 12 can be approximated well using Monte-Carlo methods. However, [74] introduces closed form approximate sliced Wasserstein (ASW) distance with convergence guarantees that does not require Carlo, making the final result a formula based on the first two moments of IP and Q
1 As (=) W2 (N (N .N (0. Mo((r)))a1))
Mallal)(m))
+ +1110 Mr 1r112.
d
(13)
where
M2((2]H1
Σ || Hal12 2
+ 111g
(14)
(and analogously for M2 M2(12117mm)) and where By and par denote the sample mean of generated {x: ( in and real samples {x - respectively. Note that the W₂ term required to compute the ASW can be trivially computed through the I-dimensional FD formula. The ASW resembles FD, but with the advantages of having both convergence and sample complexity guarantees.
To test the sample stability we use the same setting that we did for FD and FD Figure 16 (right) depicts the relative error of ASW and compares it with FD using Inception. It is clear that the ASW has less dependence on sample size, indicating that even small batches of around 10k samples may be sufficient (note that the plot is log-scale).
Despite theoretical guarantees, we found that ASW does not correlate well with human perception across various encoders and does not seem to be affected by mode shrinkage. Since it is not widely
37

---

ConvNeltt
CUP
SiMOLAR2
7.5
10
50
5.0
25
2,5
DIT-IL2
SIT-M-2- quulad
DITXL22
pTT-L22 2-giddd
nan
DIFXL2
DiTAIL 2rSML---quide
DIT-L-2
p gtance
DiNDu
Swaty
NAE
data2vec
60
1.2
20
11
1,02
1.0
Taim
DITAUL2
Train
pITxU22
DITXLL---giide 2-guiided
Train
D/TML2
DITXL-2 guiiedd
Trin
DiFat/2 quileed
Figure 17: Per-class Vendi score (higher is more diverse) distributions with different encoders. By visual inspection, DiT-XL-2 more diverse than DiT-XL-2 XL----uuiddd.
used in the literature, we chose to not include it in the main text. These results do however raise the question: why does the FD perform so well practice compared to the ASW, despite the latter being seemingly more principled? We hypothesize that there are two factors at play: (i) the encodings
are likely approximately Gaussian (e.g. [52]), meaning that comparing the first two moments enough; and (it) while the computation of the FD is biased, if the bias behaves similarly across generated datasets, it will have no impact in model rankings. While we leave a formal exploration of these hypotheses for future work, we point out that the strong correlation of FD with KD (Figure suggests that the FD only comparing the first two moments and having sample complexity issues are not problems that are borne out in practice, and thus recommend using FD as-is given that its use already widespread.
D.4 Diversity
In this section, we outline the rationale behind our decision to utilize the per-class Vendi score with DINOv2 as the metric for our diversity analysis.
Based on an analysis of the DiT-XL-2 and DiT-xL-2-guided datasets, the strong classifier guidance value of DiT-guided in comparison to DiT resulted in samples that lacked diversity within each class (examples were shown in Figure 6). We pose that a robust diversity metric should flag such a lack of intra-class diversity, and thus these two models are the basis of the following analysis. As mentioned in the main manuscript, much like precision measures more than just fidelity (Figure E and similarly density, see Figure 1O), we find that recall and coverage also quantify more than just diversity, which can be clearly seen in Figure TO (as both of these metrics have relatively strong correlations with human evaluation when using good encoders exhibit). On the other hand, the per-class Vendi score. which will further justify shortly, has the intended property of being uncorrelated with human error rate (Figure 5 right).
property that diversity metrics should not
We computed the Vendi score for the DiT-XL-2 and DiT-XL-2-guided models over each individual ImageNet class (100 images per class), and averaged the results, which are presented in Table The distributions of the per-class Vendi scores are plotted in Figure The per-class Vendi scores consistently ranked DiT-XL-2 higher than DiT-XL-2- -ugided for every encoder, indicating that DiT- XL-2 exhibits higher intra-class diversity. This observation aligns with our inspection, where it is apparent that the guided model produces less diverse intra-class samples. Hence, we adopted per-class Vendi score for our analysis.
We next report the overall (not conditioned on class) Vendi score Table 10 using 50k images from each set. It is important to note that Vendi scores represent the effective number of unique samples in dataset. Therefore, is counterintuitive obtain very low values, such those observed in MAE, data2vec, SimCLRv2, and even with simple resizing, where there are only a few effective samples among the 50k images. Moreover, not all encoders ranked DÝT-XL-2 above DYT-XL-2-guicke based on the overall Vendi scores, except for DINOv2, SimCLRv2, and ConvNext. However, SimCLRv2 exhibited minimal differences across ImageNet train, DiT-XL-2, and DiT-XL---guided. Additionally, neither SimCLRv2 nor ConvNext showed FD rankings that aligned well with human perception. Consequently, we pose that this diversity analysis provides additional support for the use of DINOv2 for generative evaluation.
38

---

Table Mean Vendi score for different encoders on ImageNet train and generated samples.
Encoder
ImageNet train DiT-XL-2
Diu-xl-2-guided
Inception ConvNeXt
6.04
5.50
3,25
28.08
23.52
8.73
CLIP
13.74
10.17
5.72
SimCLRv2
5.17
4.99
4.15
DINOv2
15.93
14.17
5.61
SWAV
8.72
6.42
4.40
MAE
1.10
1.08
1.07
data2vec
1.03
1.02
1.02
Resize (to 32x32)
1.09
1.09
1.09
Table 10: Overall diversity for ImageNet training and generated images.
Encoder
ImageNet train DiT-XL-2 DiT-XL-2-guuded
Inception
88.57
89.03
129.92
ConvNext
951.83
824.33
687.58
CLIP
36.56
27.61
31.90
SimCLRv2
6.74
6.77
6.38
DINOv2
762.52
688.53
671.54
SWAV
55,61
43.51
59.77
MAE
1.15
1.14
1.17
data2vec
1.04
1.04
1.04
Resize(to 32x32)
1.12
1.13
1.18
D.5 Rarity
Here we perform experiments investigate whether participants confuse fakeness with unlikeliness. Such effect would result in images generated from more diverse models being more likely to be assessed as "fake" and hence diverse model receiving lower human error rate compared to model of lower quality but less diversity. The following experiments rule out such flaw in our experimental setup and ensure that our human error rate dataset is not misleading.
To isolate this effect, we on focus comparing the error rate on "rare" real samples versus "common" real samples. For each of the 2000 real images used for each dataset (individual images were evaluated by an average of 13 humans), we determined the fraction of humans that labeled it as fake, while measure of image rarity/unlikeliness we calculate the "rarity score" (RS) (42). The RS is determined on image representations, so we performed the calculation using both Inception-v3 and DINOv2 quantify the dependency of RS on the embedding space.
Table 11: Correlation of the fraction of humans that labelled real image fake and the rarity score 42 of that image. The rarity score can only be determined for images that fall "on manifold".
Dataset
Encoder % manifold
p-value
(94%)
p-value (94%)
CIFARTO
Inception
76.3
0.278
0.000
0.216
0.000
CIFAR10
DINOv2
88.3
0.062
0.009
0.0004
0.859
ImageNet ImageNet
Inception DINOv2
82.2
-0.034
0.163
-0.002
0.922
92.9
0.008
0.726
-0.012
0.624
Inception
70.3
0.110
0.000
0.045
0.106
LSUN- -ed-roodrom
DINOv2
90.2
0.102
0.000
0.021
0.386
FFHQ
Inception DINOv2
79.3
-0.027
0.286
-0.024
0.357
FFHQ
89.7
-0.019
0.431
-0.040
0.103
39

---

CIFARIO
ImageNet
FFHQ
LSUN Bedroom
20
Inception
Rarity score
10
60
DINOV2
40
20
Fraction of humans that labelled real image fake
C.5
hule
Figure 18: Top: Human error rate and rarity score for each of the 2000 training images. Individual images were evaluated by average of 13 humans. Bottom: Examples on LSUN Bedroom (left) and CIFARIO (right), with the top rows displaying training images correctly judged by humans as real, and the bottom rows displaying real images "incorrectly" judged as fake. The fraction of participants who judged the image as fake denoted on the top right of each image.
As seen in Figure T8 and Table TT we find no correlation between the human error rate and RS on ImageNet and FFHQ. On CIFAR-TO and LSUN-Bedroom we find a small but statistically significant correlation, which we identify as driven by dataset issues: the non-zero correlation is caused by very small percentage of "real" images which are clearly taken from 3D-generated scenes (instead of bedroom photographs (in CIFAR-10), These results show that (i) humans are more likely to label generated scenes as fake/generated (LSUN-Bedron) CIFAR-101 10), and (it) humans are more likely to label low-quality images fake//enerated (CIFAR-10) 10). Such images have higher than average RS, and hence the small correlation between human evaluation and RS on CIFAR1O and LSSNN-Bedood
or from 2D-generated scenes or low quality images
is due to humans properly identifying these dataset issues. We find that removing just 6% of the "fakest" (as measured by humans) real images on LSUN- UU-Beddoom removes the correlation of RS and human rate quantitative proof that the small correlation is driven by dataset issues, and not due to humans associating diversity with fakeness. Rare defects in the training set are not enough to affect our results: being so rare 6%% means they barely affect the average error rate; and this training-set effect is the same for every generative model evaluation and thus does not change their rankings.
This analysis provides additional validation that experimental design and human error rate metric are accurately measuring the fidelity of images, and are not effected by extraneous factors such as image unlikeliness/rarity
D.6 Memorization
In this subsection we include detailed description of our procedure to (i) collect memorized samples for generative models trained on CIFAR 10, ImageNet, FFHQ, and a number of controlled experiments to probe the effectiveness of memorization metrics in ideal scenarios, and (iii) show additional results for memorization metries in different representation spaces.
(Wi) summarize
40

---

Memorization check with calibrated lg distance In Figure 19 we show the memorized ratio for each of the 13 generative models we evaluated on CIFARIO using the metries described in Section 5.2 We also tailored k - which is the hyperparameter specifying the number of nearest neighbours used to compute the metric to each dataset. We use k = 50 for CIFAR1O, FFHQ. and LSUN, while for ImageNet with its limited examples per class in the training set, we set k = 3. In addition, we exclusively conduct intra-class nearest neighbour search on ImageNet, as opposed to performing the search across the entire training set. We find that more complex models are more likely to memorize on CIFARIO, whereas less performant models such as ResFLOW and NVAE are less likely memorize.
Moreover, we provide additional visualization results further showcase the exact memory exhibited from DDIM on CIFAR 10 in Figure [0] and the reconstructive memory exhibited from DiT-XL-2
on ImageNet in Figure 21 and from ADM-dropout on L.SUN in Figure 22 Based on Figure 20 the presence of exact memorization is evident, raising concerns that need to be considered on CIFAR10 10. However, each "memorized" sample found with the above metric Figure [2T on ImageNet and Figure 22 on LSUN while not being pixel-wise identical retains higher degree of semantic similarity compared to the three alternative matches in the training set. We can also observe that, for some cases, the generated image does not exhibit significantly closer similarity to the matched training sample compared to the training sample's ' nearest neighbors. While seems that the model tends to show reconstructive memory samples that have similar duplicates in the training we note that this marks potential failures of the memorization metric, and thus do not report a total number of samples without further work to understand the reasons behind this. This memory be interpreted not necessairily as memorization, but as a form of generalization, as the model captures and reproduces common patterns characteristics present in the training data. Our analysis leads us to conclude that reconstructive memory on complex datasets not currently a major concern, as out of millions of training images found small number of matches. Nevertheless, we re-iterate our stance from the main text that reconstructive memory should continue to be monitored as models themselves become increasingly more complex.
also
Other than the pixel-wise check, we also revisit the nearest neighbour search in DINOv2 representation space in small scale and find decently- decentty-matched matched results for complex datasets; qualitatively, the results are similar to the pixel-wise check from Figure 21 This could even further explored future work. Yet it is still unclear whether or we have exact memorization the matched cases mostly oceur within classes exhibiting low intra-class diversity, where the samples in training set are extremely similar with each other.
1.4
1.2
Menoriatiss Ratio %
1.0
0.8
0.6
0.4
0.2
0.0
ReACGAN
P
Sy
LSOM
LOGAN
MEGAA
ResFLOw
WGAN
ACGAN
NVAB
SydeeANN-ADA
Figure 19: Memorization ratio for 13 generative models on CIFAR10.
D.6.1 Controlled experiment for memorization metrics
To study the response of metrics to memorization we first constructed controlled experiment where generated samples become increasingly memorized from realistic set of training images.
41

---

Figure 20: Memorized samples from DDIM on CIFARIO. Images are arranged vertical pairs, where the upper image is generated, and the lower image is training set image.
We generate the synthetic datasets using the VDVAE architecture 21 which is hierarchical VAE. Specifically, we used an FFHQ-256 pretrained model, which has 62 layers, and for each training example we sampled from the approximate posterior until certain layer, and sampled from the prior in later layers. this manner, going deeper with the approximate posterior corresponds to more closely memorizing the training example. In particular, using depth 0 means sampling from the model's prior, while using the posterior until depth 62 corresponds to a reconstruction that a very close reproduction of the original image. For qualitative example of depth and memorization refer to Figure 23
Formally. a VAE consists of three functions: (2) the generator p=(z]2). (it) the approximate posterior (=]x), and (ii1) the prior po(2), where the latent variable. In the hierarchical setting, assumed that = (2o, 21, =n.) and the prior and the approximate posterior are factorized as:
Pa (=) = po (= (2) ... pp ==)))
(15)
as [ [x) (2)l)))) |., 20) 9s (2n x z.n)
(16)
The distribution of the image conditioned on this is po (x|2) i.e. generator. sample from this model, pe(2) is used to sample and then to sample æ conditioned on 2. This corresponds to
42

---

Figure 21: Reconstructive memorized samples from DiT-XL-2 on ImageNet. Each set of images in row includes generated sample, matched training sample, and three nearest neighbors from the training set associated with the matched training sample.
43

---

Figure 22: Reconstructive memorized samples from ADM-dropout LSUN. Each set of 5 images in row includes a generated sample, matched training sample, and three nearest neighbors from the training set associated with the matched training sample.
CT t
80
F18-POG
AVAECT
60
40
Value
20
-20
-40
Depth
Figure 23: Left: Qualitative example of synthetic data for the controlled memorization experiment. Top left image is training sample and the next 5 images correspond to using the approximate posterior up to depth 41, 15, 5, 3, and 0. Right: Values of the CT score, AuthPct, and FLS-POG at each depth. The DINOv2 encoder used for these metrics.
depth 0 setting. To make samples resemble the training set, we can use (2(2)x) instead po(2). The distribution is modified follows
p(2|z)= = (20l)) ... (z I, 2.m (tt.)))e@@mmmmmmmm (2m+11
(17)
In terms of sampling, this corresponds to using the training image and the approximate posterior to generate 20, Ø1, ... Am and then using the prior to generate the rest z. As m (which we refer to as depth) increases, the mutual information between and = increases, which in turn results in more faithful reconstruction.
Using the described procedure, 30,000 samples are generated at each of the following depths: (meaning no conditioning). 1, 3, 5, 9, 15, 25, and 41. We use 63,000 images as the training set and 7,000 as test images; these are the same sets that are used to train and test the VDVAF, We calculate AuthPet, Cr, and FI-S-POG on each set of images and display the results Figure 23 We find that both CT score and Auth have characteristics that indicate they can correctly identify memorization
44

---

in this experiment. At depths less than 15, Cr is positive value, indicating the training samples are no closer to the generated samples than they are to ones from the test set (are not memorized), while at depths greater than 15, Cr becomes increasingly negative, indicating stronger detection of memorization. AuthPet starts from close to 100%, meaning all samples are authentic, and decreases to 20% at a depth of 41, indicating that generated samples are closer to training samples than expected. Since a depth of zero means no conditioning on individual training images (except for the training of the original model), the result confirms the effectiveness of the proposed method for this particular point. As the depth increases, AuthPet correctly decreases the percentage of authentic samples in rate that is almost equal to the decrease in Cr. FLS-POG has the correct trend as it increases as samples become more memorized. However, the rate of the increase is minimal, does not reflect the level of memorization seen at depth of 41 (as seen in the images training samples which are almost exactly reproduced at this depth), and is not comparable to the change in the other two metrics. From this controlled experiment we conclude that, in the absence of mode collapse/mode shrinkage when training samples are uniformly memorized, CT and AuthPct can flag memorization in practice while FLS-POG does not flag memorization strongly enough.
D.6.2 Memorization metric quality
With a better understanding of AuthPet, FLS-POG, and the Cr score, we calculate them on the sets of CIFAR1O generated images and compare to the measured memorization ratios determined through the analysis above. Results for the DINOv2 representation space were shown in Section 5.2 while here we show the scores for every encoder in Figure 24 We also include our modification the Cr score which swaps the roles of the training and test samples (details and motivation for the modified CT score follow in Appendix D.6.3). We remind the reader that higher AuthPet, Cr score, and modified Cr score are meant to indicate less memorization, whereas lower FLS-POG is meant to indicate less memorization.
Here we detail the characteristics of each encoder. For Inception, we see that a higher memorization ratio seems to correctly correlate with lower AuthPet and Cr score. However there is no discernible pattern for FLS-POC and CT modified. Meanwhile, the SWAV encoder seems to provide the correct directionality for all memorization metrics. CLIP produces no discernible trend for AuthPet and FLS-POG, a slightly accurate trend for Cr, and strongly accurate trend for modified Cr. MAE is similar CLIP except with the roles of FLS-POG and CT perhaps flipped. data2vec is somewhat similar to SwAV with the correct directionality particularly for CT score but with a slightly weaker trend for the other metries. Lastly. DINOv2 shows a correct correlation for the modified Cr score, but no strong trends for the others. While the performance of each encoder shows a large variation, is important to note that we illustrate clear issues with the memorization metrics in the next section (D.6.3). We find that modified Or score performs the best in this section, with Cr score next, then AuthPet, then FLS-POG last. This agrees with our previous experiment that shows FLS-POG is the weakest detector of memorization.
D.6.3 Low-dimensional experiments for memorization metries
Table 12:
Memorization metrics on synthetic datasets
Metric
True Distribution Shrinkage Memorized Underfit 1 Underfit 2
Underfit 3
AuthPet
41.20
20.00
0.00
46.60
67.20
77.30
Cr Score
-0.23
-16. 14
-25.26
7.16
15.22
18.42
CT Mod.
0.05
-0.86
-16.71
-0.62
0.23
1.37
FLS-POG
60.10
60.00
89.70
59.00
56.30
55.20
While the previous subsections have confirmed that Cr and AuthPet can flag memorization in ideal scenarios (Section D. and seemingly show appropriate trends in some representation spaces (Section D.6.2). is not clear that they are explicitly flagging memorization over other forms of overfitting. Curiously, the metrics computed against the test set resemble those against the training set, which the models cannot have memorized outside of data-leakages between the construction of the train and test sets. While we note that such leakages do occur for CIFARIA 10 (6. in the following toy scenario we demonstrate the inability of the metrics to separate types of overfitting from memorization.
45

---

Diliiain Fuow
GAN
VAE
Zaito
Incestion
0.10
005
90.0
AuthPer
VLS-POG
Cr
Cy modified f
Memeriatiin (5)
SwAY
0.0
83
46
50
0
AuthPPt
ELS-POG
Cr
Cy modified f
0.1.
Menoriiation (%)
0.10
CLI
0.08
90
90
90
40 FLS--POG
50
20
40
60
-1.0
-0.5
0.0
AuthPPr
Cr
Cr modified
0.15
Memoniation Ratio
MAU
0.10
0.05
70
25
100
50
-19
10
AuthPer
ULS-POG
modified
0.15
Memusiaiion Ratio
D.10
n.s-
16
10
10
20
1.5
1.0
0.5
00
Authfet
TTS--POG
Cy
Cr medified
0.15
Mettrrialion (%)
DINOK2
0.10
0.08
AuthPPr
FLS.PIG
Cr
Cr modified
Figure 24: Memorization ratio metric value for each metric out of AuthPet, FLS-POG, Cr, and the modified Cr, evaluated on the models from Figure 19. The same metric values computed against the test set are shown with lower opacity.
To understand the impact of various modelling phenomena on memorization metrics we design simple 2-dimensional experiment. As ground truth distribution, we take uniform mixture of 5 Gaussian components. Each component's mean sampled from a unit diagonal Gaussian distribution, while each component's covariance matrix is diagonal with diagonal elements sampled from uniform distribution over the interval [0.01, 0.09). From this distribution we sample 1000 training and 1000 test points.
46

---

The Dist Dusion
Shridlgge to Mares
Mesocied Traiing Tuas
1.5
10-
05-
05-
0.3
2.-
00
-0.5
Uskkti Modd
Undofi Mold
Koda
Counad
Figure 25: Toy datasets generated for the purpose of investigating memorization metrics.
We then generate 1000 points from various "models" from which to measure memorization:
A perfect model with zero memorization, represented by independent samples from the same ground truth distribution.
A model experiencing total mode shrinkage: it uniformly samples a mean from each Gaussian in the mixture.
A model that has perfectly memorized the training data: it amounts to randomly sampling from the training data with replacement.
Three "underfit" models, which in fact have learned the true distribution with upscaled standard deviation for each component of the Gaussian mixture: 1.5o, 3a, and 4.5g
The training, test, and generated distributions are shown in Figure 25 We evaluate metries on these datasets and report the values in Table 12
FLS-POG seemingly performs correctly for detecting memorization in these regimes- with the memorized model receiving the highest score of 80% and the other models receiving score hovering around 60%. However, a value of 60% on both the true and mode shrinkage examples indicates that it poorly flags mode shrinkage, one of its claimed usages (53), and the relative change of 5% from underfit to the true distributions is on the order of the change seen in the VDVAE controlled experiment, indicating that such relative changes can have multiple meanings. AuthPet indicates memorization (< 50%) for the mode shrinkage model, the perfect model, and the slightly underfit model. From this, we gather that AuthPet incorrectly flags well-trained models that produce samples on the true manifold.
We also find that the Cr score performs incorrectly here: the score of 16.14 it assigns to the mode shrinkage model well below 0- has incorrectly flagged memorization. We see why by looking closer at the metric: the CT score compares how much more often (i) the distance from a generated data point training data point is lower than (ii) the distance from a test data point to a training data point, and reports a highly negative score if (i) is often lower than (ii). In this case, we would expect our generated, mode-shrunken samples to be much closer to the training data than the test data is to the training data, which agrees with what we observe. The issue here boils down to a lack of symmetry between test and training set: the test set only used for one of the comparisons, whereas the training set is used in both distance checks. The generated dataset is also only used once. We address this asymmetry by proposing a modification to the CT score - referred to as the modified CT score wherein we essentially swap the roles of the training and the generated data within the base Cr score. The modified Cr score thus compares how much more often (i) the distance from generated data point to training data point is lower than (ii) the distance from a generated data
47

---

RD ELA!
DESL4
Figure 26: ImageNet samples for generative models (rows) sorted by error rate from our human subject experiments. Classes from left to right: great white shark, English Springer, comic book, and volcano.
point to a test data point, reporting a highly negative score if (i) is often lower than (di). To fully summarize and perhaps visualize our change, consider the Cr score function from training data Ti, generated data G, and test data 72 to the real numbers, represented as (T G, T2). Then, the modified Cr score call it C .M - can be represented Cr. aM (Tı, G, = Cr, T2,)). We see that our change has produced the desired behaviour, as now the mode shrinkage dataset is not detected as memorized by Cr.M.
From this discussion we gather that the Or score incorrectly flags mode shrinkage, and AuthPet incorrectly flags well-trained models. Recall that of the models except for the memorized one are actually derived from the test set. These experiments explain why AuthPet and Cy seem to score models in way that is almost invariant the training set Figure
D.7
Model samples
Figures 26 27 datascts.
29 and
28 show image samples from the 41 generative models across the four
48

---

Secad
RD 2.
Eas Hoao
RD LD-
Eas Bas A04
Figure 27: CIFARIO samples for generative models (rows) sorted by error rate from our human subject experiments. Non class-conditional models show a random selection of generated samples while class-conditional models show classes airplane, automobile, bird, and dog.
ZERM
Ri -
Eae Bal 4009
Figure 28: LSUN Bedroom samples for generative models (rows) sorted by error rate from our human subject experiments.
49

---

Tixe
Kiw Hout TDD-502
di dehees
Foe RD--500 5.30
SUGNAT 2.11
Figure 29: FFHQ samples for generative models (rows) sorted by error rate from our human subject experiments.
50

---

E 2DDINOV2 scores leaderboard
Table 13 shows our considered metrics across models and datasets.
51

---

FLS-POG
10.13
13.93
11.46
0.87
3.17
0.31
-0.02
-0.76
1.23
0.57
0.90
-0.14
0.66
2.86
1.58
2.99
1.52
2.71
0.30
2.23
3.43
1.56
5.66
3.20
Cr Cr mod.
0.02
0.56
-0.12
0.12
0.10
-0.64
-0.46
-0.21
-0.63
-0.89
-1.12
-1.56
-0.73
-1.56
-1.61
-2.33
-1.52
-1.58
-2.03
-1.73
-1.74
-2.87
-516
-3.25
49.17
59.90
61.46
65.01
69.40
63.28
62.87
62.64
57.98
48.64
51.93
45.60
43,23
121.94
124.25
106.24
103.29
97.40
104.43
70.19
51.33
59.56
43.29
34.32
Precision Recall Density Coverage AuthPet
76.72
82.38
87.45
86.91
91.51
93.56
93.49
94.04
93.22
92.13
93.03
91.56
91.96
72.23
92.11
94.30
93.77
95.02
92.60
93.34
92.35
89.56
90.91
61.46
85
69.83
93.60
93.85
94.25
91.72
90.21
92.65
91.01
87.60
96.91
96.70
90.74
96.04
94.43
93.75
93.10
91.91
91.30
0.05
0.07
0.08
0.11
0.12
0.41
0.42
0.44
0.52
0.64
0.60
0.71
0.72
0.97
0.54
0.58
0.68
0.68
0.75
0.72
0.85
0.89
0.86
0.95
0.94
0.96
0.24
0.29
0.44
0.45
0.71
0.68
0.76
0.90
0.30
0.39
0.54
0.51
0.53
0.64
0.63
0.71
0.74
1.91
2.06
1.60
1.27
0.98
0.69
0.66
0.66
0.70
0.78
0.69
0.76
0.80
1.01
0.95
0.69
0.73
0.76
0.77
0.67
0.80
1.00
0.86
1.31
1.01
1.00
0.78
0.74
0.67
0.75
0.87
0.68
0.77
0.93
0.31
0.36
1.04
0.46
0.61
0.71
0.68
0.77
0.83
Table 13: DINOV2 VII--/I11 L/14 leaderboard.
0.01
0.01
0.04
0.27
0.25
0.47
0.46
0.44
0.51
0.66
0.55
0.74
0.71
0.88
0.39
0.76
0.77
0.72
0.76
0.85
0.84
0.76
0.84
0.60
0.84
0.94
0.23
0.28
0.41
0.23
0.41
0.61
0.64
0.75
0.07
0.06
0.14
0.13
0.24
0.28
0.43
0.41
0.44
0.87
0.84
0.82
0.76
0.71
0.70
0.68
0.69
0.71
0.74
0.72
0.74
0.73
0.88
0.81
0.76
0.81
0.82
0.84
0.80
0.87
0.92
0.88
0.99
0.93
0.95
0.80
0.79
0.78
0.83
0.85
0.79
0.83
0.85
0.57
0.59
0.86
0.64
0.76
0.79
0.77
0.79
0.81
FLS
49.15 15
52.31
54.62
58.85
59.38
76.84
76.73
77.31
81.55
85.37
84.10
87.95
88.13
324.78
66.48
70.43
74.61
75.16
78.89
77.41
83.56
87.92
85.07
97.23
92.20
124.59
KD
2.64
2.41
2.04
1.57
1.51
0.49
0.45
0.39
0.45
0.26
0.24
0.18
0.17
-0.00
0.52
0.34
0.24
0.22
0.24
0.19
0.09
0.09
0.08
0.03
0.06
0.00
2.43
2.05
1.48
1.66
0.72
0.65
0.45
0.13
2.03
1.64
1.57
1.33
1.06
0.79
0.56
0.56
0.55
FD
1296.52
1205 42
1083.16
16
916.05
876.35
353.90
353.25 25 349.76
320.71
300.96
.6
207.69
198.78
143.14
137.50
1.03
393.13
294.75
219.27
207.09
206.36
195.01
114.69
103.73
102.63
90.04
70.32
12.38
634.58
544.73
437.68
425.76
236.64
227.69
163.85
57.73
589.20
511.68
511.19
432.68
390.60
300.18
240.07 07. 236.98
226.04
223.64
FD
1301 38
1208. 26
1088.56 56
921.34
881. 381.73
358.07
326.66
305.92
212.30
204.60
60
148.27
141.65
31.07
401 .2
304.05
228.37 37
214.88
214.45
203.45
123.27
112.40
111.24
99.16
79.36
21.04
636.35
547.61
440.04
04
428.9 99
239.79
229.76
166.19 19
59.64
592.26
514.78
514.16
436.26
393.45 45
303.21
229.42 42
226.72
Human emor rate
0.09=0.01
0.15
0.17=0..0
0.13:0..2
0.212
0.34+00.1
0.34-10.22
0..01
0.39=10.11
0 0.40=0.01
0.40=0.01
0.44=0.01
0.4410.01
0.16=0.01
0.22+0.01
0.21=0.01
0.15
1..02
0.27:10.22
0.25
0.31
0.27=0.02
0. 0.33
0.29:00.22
0.14=00.2
0.15=0.02
0..120.02
0.16:10.2
0.22
0.27++0.2
0.30+=0.02
0.39-10.2
0.12+00.1
0.20110.2
0.09
0.25
0.17=
0.1811.01
0.101
0.30
0.19::0.02
StyleGAN2 ada
iDDPM-DDIM
StyleGAN-XL
LSGM-ODE
RQ- Te-Trasfformer GigaGAN
StyleGAN-XI
ADMG-ADMU
DIT-XL-2-guide 2-gudded
Validation Set
Projected- GAN
GAN
Diff- Pro-PrroGAAN
Unleash- Trans
ADM-dropout
Projected-
StyleGAN2-ada
Efficient-vdVAr VVVE
Unleash- Trans
StyleGAN-X -L
RESFLOW
ACGAN
WGAN-GP
LOGAN
MHGAN
ReACGAN
BigGAN
PFGM++ Test Set
BigGAN
Mask-GIT
DiT-XL-2
Consistency
StyleGAN
StyleSwin
StyleNAT
Model
NVAE
ADM
ADMG
LDM
DDPM
iDDPM
InsGen
LDM
GIFARD
et 256 100x/10 2266 256 6
SSUA Cediromn
FHHO x56x28 256
$2

---

Author contributions
Table 14 lists individual contributions of the authors (in alphabetical order).
Table 14: Author contributions
AC BR
ET
GL
GS
JC
RH
VV
YS ZL
Conceived of ideas for project initially
X
X
Conceived human experiments
XX
X
XX
Created, performed, & analyzed human exp.
X
XX
Trialed human experiments
X
X
X
X
X
X
X
X
X
X
Generated CIFAR 10 datasets
X
X
X
X
Generated ImageNet datasets
X
X
X
Generated FFHQ datasets
X
X
X
X
Generated LSUN-Bedrood Bedroom datasets
XX
X
Implemented encoders
X
X
X
X
X
Grad-CAM encoder understanding
X
XX
X
Designed metries
X
X
X
Implemented metries
X
X
X
X
X
XX
Memorization design
X
X
X
X
X
X
Memorization implementation
X
X
X
X
Diversity
X
X
X
Other experiments
X
X
Structured Codebase
XX
X
XX
Prepared figures
X
X
X
X
Analyzed data
X
X
X
X
X
X
X
X
Wrote paper
X
X
X
XX
XX
XX
X
X
X
Managed the project
X
XX
53


Reviews:
[{"review_id": "jr0RFsmgmv", "reviewer": "Reviewer_xpCe", "summary": "This paper constructs an image dataset sampled from various generative models and scored by human participants in terms of their fidelity, and argues that existing metrics do not correlate well with this notion of fidelity. Then, it investigates how different choices of embedding space (i.e. different encoders) affect the metrics in terms of their ability to measure diversity and fidelity, and concludes that DINOv2 can be a superior choice of encoder. Finally, the paper studies whether existing memorization metrics can effectively measure memorization in high resolution datasets, and argues that none are reliable.", "questions": "See the weaknesses section.", "limitations": "See the weaknesses section, particularly on the collection of the dataset.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper asks interesting and important questions regarding the reliability of existing metrics of generative performance, and provides many valuable experiments with unique insights on the limitation of existing metrics for measuring fidelity and diversity of generative models. The proposed dataset of human evaluation can be a useful addition to the existing datasets for measuring and improving generative performance.", "weaknesses": "My main criticism of this paper is that it is not one coherent paper, rather parts of three different papers: one paper on constructing a dataset of human evaluation of generative models, a second paper on how the choice of encoders affects existing metrics, and a third paper on measuring memorization in high resolution datasets. As a result of this, many important results are shifted to Appendix (Appendices B and D in particular), and several observations are poorly explored and studied. I think each of these three directions deserves its own focused paper, carefully considering the caveats, and avoiding broad unjustified claims. I’ll elaborate on my specific concerns below.\n\n***Regarding the dataset:***\n\n1. I think the human experiment is flawed in the sense that “choose the fake sample” could result in choosing the unlikely samples as fake (confusing fakeness with unlikeliness). For example, a contorted low quality image of the common crow can be rated as “real” whereas a high quality image of an exotic rare parrot can be rated as “fake”. Therefore, a diverse model can receive a lower human error rate compared to a model of lower quality but less diversity. The experiments must propose a way to control for the effect of rarity, otherwise the dataset can be misleading.\n\n2. Another issue with the human experiment is that it is unclear how the participants were selected. For example, if participants are mostly of one ethnicity, it is possible that they penalize diversity in place of fakeness. I understand that this issue is not easy to solve, but at the very least, I expect to see some effort in making sure the human participants are of sufficiently diverse backgrounds and origins.\n\n3. For the results in Table 1, the difference being statistically significant alone is not enough, the amount of difference is itself of value. For example, a 1 percent difference is not as important, albeit statistically significant, as a 10 percent difference in human error rates. Reporting the actual numbers in a table similar to Table 1 will clarify this matter.\n\n4. I am not sure how the main claim of “unfair treatment of diffusion models” in the title is justified. The brief explanation in L198-199 is too broad: “Coupling the results in Table 1 with the FID rankings in Figure 2, we conclude that current diffusion models produce the most realistic images, but are unfairly downranked by FID”. Could you be more specific about the connection from evidence to this claim?\n\n***Regarding the metrics and encoders***\n\n5. L225 claims that “We find that Inception does not perceive a holistic view of images even on its ImageNet training set”, however I don’t think the qualitative results in Fig 3 can support such a strong statement. A quantitative experiment is required to be able to claim this finding (I don’t see how the quantitative results in Appendix D.0.2 can support this claim, if so, please elaborate.)\n\n6. L299 claims that: “We see this as strong evidence of a fundamental limitation of the use of the Inception network when computing FID”. I don’t see how this shows a fundamental limitation. To me, it only shows that for some generative models, FID seems to not correlate well with per class Vendi score. If the intention is to show that FID does not correlate with Vendi in general, why not explicitly report correlation? But even then, can you elaborate what “fundamental” means in this context? I do understand that different embeddings focus on different features, that is given by design, but that one embedding is fundamentally weaker than another needs a more formal argument.\n\n7. The conclusion that DINOv2 is better than Inception is not well justified to me. The main evidence for this strong claim seems to be Figure 6, which is at best a motivating observation (reporting the correlation coefficients could help make Figure 6 more substantial). A systematic ablation study for the encoder on the effect of a) the number of training classes, b) various loss functions, and c) various architectures (only changing one factor at a time and fixing the other factors), on the performance of metrics can be more conclusive regarding which encoder is more reliable.\n\n8. For a paper about metrics, the definition of considered metrics should be restated to facilitate the clarity and readability of the paper. This can be done for the main metrics in the body of the paper (FD, P/R, Vendi) and for others in the appendix. The current definitions in Appendix B lack the exact mathematical definitions except for FD.\n\n***Regarding memorization***\n\n9. The paper does not discuss why each of the considered memorization metrics fail, that is, what assumptions in the definition of each metric deviates from practice and potentially causes the observations. The lack of such discussions and followup experiments to pinpoint the cause, makes the empirical results inconclusive in general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We are happy our paper “provides many valuable experiments with unique insights on the limitation of existing metrics”. We believe the replies below address your concerns, and kindly ask that if you agree, to consider raising your score.\n\nOn our work being “three different papers”: Given the long history of generative evaluation, any proposed metric must be demonstrably superior on a wide range of quantitative and qualitative evaluations. The three components of our paper are a cohesive whole to accomplish this goal, and removing any would make the paper insufficient to build community trust and support. \n\nThe first part – constructing the dataset – would be an incomplete contribution in terms of novelty, as we follow (yet improve) on the methodology of HYPE [91]. The resulting novel insight that FID does not correlate well with human judgment demands a subsequent investigation of alternate encoders and metrics. This second part is the most natural way to search for a new evaluation metric that correlates better with human evaluators. \n\nThe third part resolves two critical caveats: Section 4.2.1 dismisses the concern that FID fails due to humans not measuring diversity, and Section 4.3 establishes whether the best generative models “cheated” their human alignment through memorization (a valid concern since FD does not detect it). We thank you for pointing out an area where we can improve our writing and will refine this narrative in the final version of our paper.\n\n1\\. We respectfully disagree that this is a flaw in our experimental design - our methodology accounts for and negates such an effect: each participant sees a total of 250 images from ImageNet (which has 1000 distinct classes), making it essentially impossible to learn any diversity within any class, and thus forcing a focus on fidelity. Using your crow/exotic parrot example, *in ImageNet these would be different classes, and generative models synthesized class-conditional images that closely resemble the semantic information of a class* (see Figure 20 for proof). Therefore, the crows/parrots from different generative models will have varying degrees of fidelity, which will be measured by the human error rate. Additionally, if participants were confusing fakeness with unlikeliness, *this effect would be consistent across models and therefore would not alter rankings*. Another control for any such effects is the training phase (Figure 18), in which participants gain a sense for the diversity of images before starting the test.  Finally, we highlight that our experimental design is highly similar to the widely adopted HYPE design, and was aided by experts in psychophysics precisely to avoid these types of flaws and measure human perception of realism accurately.\n\n2\\. & 8. See general rebuttal.\n\n3\\. & 4. We believe that the titular claim of “unfair treatment of diffusion models” is justified. Figure 2 clearly shows that diffusion models score the highest human error rates, and that GANs often score lower human error rates yet achieve a better FID ranking. Table 1 summarizes the statistical significance, and we report the exact values and error bars in Table 11. We agree that the word “unfair” in lines 198-199 is not yet justified at that part of the paper though, and will remove it. Nonetheless, the remainder of the paper investigates potential causes for this misalignment of FID and human evaluation, and ultimately builds up sufficient evidence to support the “unfair” conclusion. \n\n5\\. Thank you - we agree that this is a qualitative claim, which we will soften in the final version. We point out however that it is consistent with our quantitative analysis and that in Appendix C of [47], which we will also make clear in the paper.\n\n6\\. We believe this viewpoint stems from a misunderstanding of the goal of our diversity experiments, which we will clarify in the final version of our paper. The lack of correlation between FID and human evaluators could be due to either (a) FID being fundamentally flawed, or (b) FID accounting for diversity where human evaluators do not - perhaps diffusion models trade diversity for fidelity. The goal of our diversity experiments is to rule out option (b) (i.e. checking that differences in Vendi score do not explain flips in ordering between FID and human evaluators – this is not equivalent to correlation).  Our experiments show that images from diffusion models have good diversity, which, combined with results from our human experiments, allows us to rule out option (b) and conclude option (a).\n\n7\\. On Fig 6 please see the point above. On an ablation study: what you describe would require retraining self-supervised foundational models on internet-scale data, which would be prohibitively expensive (DINOv2 alone required 200k GPU-days [58]) and out of scope for this work. We also note that the self-supervised foundational encoders that we use were trained with a variety of objectives and architectures.\n\n9\\. Given the increasing attention placed on memorization of generative models [11, 58] we included our results which show that memorization metrics can not be relied on at this stage, which presents a natural call for further study. Recent work [C] (made publicly available after the NeurIPS submission deadline) has shown that various pathologies associated with precision and recall are due to the curse of dimensionality as a byproduct of nearest neighbors. These findings are consistent with our own observations of precision and recall, and we hypothesize that this could cause the failure of memorization metrics. We will include this discussion in the final version of our manuscript.\n\n[C] Emergent Asymmetry of Precision and Recall for Measuring Fidelity and Diversity of Generative Models in High Dimensions. ICML 2023."}, {"review_id": "XOUw7avhNf", "reviewer": "Reviewer_uifn", "summary": "This paper initially demonstrates which embedding space is similar to human evaluation criteria by utilizing various datasets and image generation models. It reveals that the embedding space of DINOv2 aligns most closely with the tendencies identified through a large-scale human survey. Moreover, it highlights the fact that most of the commonly used metrics do not align with human preferences. Additionally, the paper proposes approaches to address the issue of memorization, assessing whether the models are simply reproducing existing images.", "questions": "Is there an intention to make all the features publicly available? \n\nAdditionally, what distinguishes this paper from the next paper?\nhttps://dreamsim-nights.github.io/\n", "limitations": "The novelty may not be significant; however, I believe the experiments conducted in this paper hold sufficient value.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The limitations of the traditional FID metric were widely recognized, and several papers were already trending towards using CLIP FD as the primary metric. While FID captures typical trends well, its clear limitations prompted this paper to explicitly demonstrate these constraints and advocate for the use of specific models' embedding spaces. The paper made significant contributions by conducting large-scale surveys to create metrics for human evaluation and by publicly sharing the generated images used in subsequent analyses, which is particularly valuable considering the time-consuming nature of Diffusion models. Furthermore, the paper addressed the issue of memorization and provided a quantitative representation of it.", "weaknesses": "The setting of using DINOv2's gradCAM is inevitably beneficial. It would be advantageous to include comparisons with metrics such as LPIPS as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review and helpful feedback! To address your questions:\n1. *Is there an intention to make all the features publicly available?*\n\n   Yes, we will be making all the features of the code publicly available, and all of the datasets. Hopefully this will help facilitate further research into evaluating deep generative models. (If you meant feature representations of the images, we did not intend to release these, but they are easy to compute using the code and images we will provide).\n\n2. *What distinguishes this paper from https://dreamsim-nights.github.io/?*\n\n   In short, our paper is about evaluating deep generative models, while theirs focuses on image-to-image similarity similar to LPIPS. To be specific, they create a dataset of human-perceived image-to-image similarities and design an encoder that reflects these human judgements. On the other hand, our human trials focus on image quality rather than similarity, and our subsequent analysis uses these human judgements to evaluate current metrics for generative models. In other words, DreamSim focuses on obtaining an encoder which maps images that humans assess as similar to nearby points in latent space, whereas we focus on finding an encoder where distances between probability distributions on its latent space, such as FD, correlate with human judgment.\n\n    It is natural to wonder whether their improved DreamSim encoder might provide a better representation space for evaluating generative models than current self-supervised models such as DINOv2; however, when the paper came out, we ran a preliminary comparison and found it did not. For example, We found that the OpenCLIP-DreamSim fine-tuned encoder had nearly identical human/FD alignment as the original OpenCLIP encoder, and that their ensemble of encoders did not align with our human experiments as well as DINOv2 does. Unfortunately the DreamSim paper was first publicly released June 15, well after NeurIPS submissions had closed, so we were unable to describe these results in the paper (but will in the final version).\n"}, {"review_id": "5FRqKwP4rb", "reviewer": "Reviewer_6HZb", "summary": "In this paper, the authors conduct a thorough investigation into the limitations of the Frechet Inception Distance (FID) metric for evaluating generative models. They address this issue by performing human evaluation and proposing a superior alternative for automatic generative model evaluation. Through dedicatedly designed experiments, the authors empirically demonstrate that FID, which relies on a pre-trained InceptionV3 model, exhibits a weak correlation with human evaluation. However, by replacing the InceptionV3 with a self-supervised model like DINO, the automatic evaluation becomes more closely aligned with human evaluation. The insights presented by the authors offer valuable guidance on the appropriate approach for evaluating image generative models.", "questions": "please refer to the weaknesses.", "limitations": "I believe this paper delivers a crucial message to the community by emphasizing the inadequacy of FID as a metric for evaluating image generative models. Additionally, the authors propose a substantial improvement by changing from InceptionV3 to DINO as the backbone, which significantly mitigates this issue. While it may not completely resolve the problem, it remains a commendable contribution. Considering these contributions, I would give a score of 7 to this paper.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "[1] This paper addresses a highly significant issue within the generative modeling community. The authors' findings, demonstrating the lack of correlation between FID and human fidelity judgment, are particularly intriguing to me.\n\n[2] The proposed alternative, FD_{DINO}, appears to be a sensible solution and holds promise for future evaluations of generative models.\n\n[3] I think the provided benchmark tables are helpful for researchers trying to evaluate their generative models.\n\n[4] In my opinion, the paper is well-written and makes a noteworthy contribution to the community.\n\n", "weaknesses": "[1] One potential concern is that all interpretations in the paper are based on the assumption that human evaluation is entirely correct, which may introduce some inherent risks.\n\n[2] Providing further explanations of the evaluation metrics would enhance the paper's accessibility for individuals who are not experts in this particular field.\n\n[3] Adding more detailed explanations about the differences between the authors' paper and closely related work [R1] would be beneficial for better understanding the novelty and contributions of the presented work.\n\n[R1] M. Yang, C. Yang, Y. Zhang, Q. Bai, Y. Shen, and B. Dai. Revisiting the evaluation of image synthesis with GANs. arXiv preprint arXiv:2304.01999, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and helpful review. We are thrilled to hear you frame our insights as “valuable guidance on the appropriate approach for evaluating image generative models.” You rightly mentioned that including more details on the metrics in our paper will make it more accessible - please see the general response where we address that. We address your other concerns below:\n\n- “All interpretations in the paper are based on the assumption that human evaluation is entirely correct”:\n\n   This is essentially true. We carried out this study with the premise that human alignment is superior because humans are the users of the generated output. If a particular neural net architecture were the intended viewer, for instance, we would need to evaluate images differently. This is the premise of GANs, and as we demonstrate in our work, generative models that can fool a neural net classifier need not look the most realistic to humans.\n\n   Taking a practical perspective, we do believe that human perception is the best metric available for image fidelity. We highlight that we did our best to capture it with the cleanest crowd-sourced data possible through our experimental design informed by best practices from the literature [20, 19], namely by including a long training period and financial incentives for correct answers. This was also the reason why we structured our experiment to provide direct comparison between generated and training images, and only compared generative models that were trained on identical sets of images, as we wanted to separate out notions of fidelity to those of “style”. We will include this discussion in the final version of our paper.\n\n- \"Further explaining the differences between our work and Yang et al. (2023) would be helpful\":\n\n \n   We believe lines 172 to 181 of our manuscript do contain the fundamental differences between our work and Yang et al. (2023), which was concurrent with our study (the first public version was released April 4, less than two months before the NeurIPS submission deadline). Namely, this work’s trials targeted much more ambiguous tasks than our two alternative forced choice task. Rather than a direct measurement on the ability of humans to distinguish real and fake images, their main user study instead scored whether each generated image was “photorealistic”, yet participants had no knowledge or familiarity with data samples from the training set, and received no training or compensation. This lack of alignment between their human evaluation study and the goal of FID and similar generative metrics of quantifying distributional distances between real and generated images makes their analysis difficult to interpret. This difficult interpretation coupled with a much smaller and less rigorous human evaluation setup (they used fewer models and collected responses from 10 times fewer participants) are fundamental differences. Nevertheless, we are excited to see that others see value in evaluating generative metrics.\n"}, {"review_id": "bz2DvI0MYG", "reviewer": "Reviewer_uTCz", "summary": "This paper try to get rid of the limitations of current evaluation metrics for generative models and focuses on the perceptual fidelity of diffusion models. The authors conduct an extensive study using a wide range of image-based generative models across diverse datasets. They employ psychophysics to measure human perception of image realism and compare it with existing evaluation metrics. The paper reveals that commonly used metrics, such as FID, do not align with human evaluations of perceptual realism in diffusion models. The authors attribute this discrepancy to over-reliance on the Inception-V3 network. They propose using alternative self-supervised feature extractors to improve the evaluation of generative models. Additionally, the paper explores the issue of data memorization in generative models and highlights the limitations of current metrics in detecting memorization accurately. The authors release the generated image datasets, human evaluation data, and a library to compute 15 common metrics for 8 different encoders, facilitating further research in this area.", "questions": "see weakness.", "limitations": "Despite the considerable experimental analysis and insights presented by the authors in this paper, I suggest that they provide a detailed discussion of the limitations of the content being discussed. It would be preferable to dedicate a separate section at the end of the article specifically for this purpose.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The author comprehensively points out the issues with the existing evaluation metrics for generative models.\n- The article presents a considerable number of insights and involves a substantial amount of work.", "weaknesses": "- Figure 1 lacks intuitiveness, and it is recommended to further improve it so that readers can grasp its meaning within a short period of time.\n\n- The conclusion of the entire article is extensive, making it challenging to grasp the main points. It is suggested to enhance the writing by highlighting key conclusions, particularly in the introduction section.\n\n- If possible, I suggest conducting a bias analysis on the 1000 paid participants, such as examining whether these participants are all students or individuals involved in the field of artificial intelligence, or if the age distribution is concentrated within a specific range, like 20-22 years old, and so on. This preliminary examination of bias can enhance the reliability of the experimental results. Of course, privacy concerns should be carefully addressed during the analysis.\n\nOverall, I greatly appreciate this work and recommend acceptance of the article. Furthermore, I strongly suggest the authors share the code for analyzing the figures and charts in the article to facilitate further analysis by other researchers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, we appreciate the time and effort that went into evaluating our work, and we are glad you found it “extensive” and “comprehensive”. Please see our general rebuttal for an answer to your question about potential bias of the participants, where we provide additional examination of demographics. In short, the paid participants were very diverse in terms of age, gender, and ethnicity, but there is no significant connection between the demographic attributes and performance on our tests. We plan to release the demographic data along with the performance data, where we have the participants consent to do so, to instill confidence in the data we collected.\n\nAs to your other points:\n\n- On Figure 1 lacking intuitiveness: Could you please elaborate what in the figure was unclear? We strive to provide our results in an easily-to-understand format and will happily update this figure accordingly to improve the clarity of the final paper.\n\n- On the main takeaways not being highlighted enough: Thank you for pointing this out, we will reformat the introduction in the final version of our paper so as to better summarize the key results of our paper.\n\n\n"}, {"review_id": "SGRHI9AGow", "reviewer": "Reviewer_SaoU", "summary": "The authors performed extensive experimental study on various image-based generative models. Based on the study, it showed that no existing metric strongly correlated with human evaluations. The authors also included alternative self-supervised features extractors for evaluation. Additionally, data memorization was investigated. The experiments revealed limitations of existing evaluation metrics for generative models.", "questions": ". Human error rate: this metric is calculated as the fraction of images which were incorrectly classified. The incorrectly classified cases include both real-->fake and fake-->real. From the reviewer's perspective, these two cases should be separately treated, and the fake-->real case should be more important for evaluation. For example, if there are a lot of real-->fake cases from one human participant, this probably indicates that this human participant has some judgment issues and his/her results cannot be trusted. One thing we probably can try is to use the real-->fake case to evaluate whether this human participant can be trusted or not, and then use the fake-->real case as the human error rate. Of course, there are definitely other ways to improve that.  \n\n. Memorization: how to evaluate pixel-wise memorization and reconstructive memorization mathematically? Based on these two metrics, how to set the threshold values to determine memorized samples?", "limitations": "The authors have discussed the limitations of their work, and probably will address them in the future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors implemented extensive experiments to evaluate various image-based generative models from different perspectives, eg., encoders, human evaluations, diversity, and memorization. The evaluation was performed using several state-of-the-art evaluation metrics including humans. The comparison results are expected to have a high impact on evaluating existing generative models. ", "weaknesses": "The experiments showed that no existing metric strongly correlated with human evaluations. There is some concern about how the human error rate is calculated. The reviewer wonders whether the conclusion would change by improving the calculation of the human error rate. See detailed comments under Questions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, we appreciate the time and effort that went into evaluating our work, and we are glad you found that the “results are expected to have a high impact on evaluating existing generative models”.\n\n- On concerns with the “human error rate” metric: The human error rate metric was established and explored in [91] which we build on in our work. We agree that the individual real-->fake and fake-->real metrics are useful for various diagnostic purposes, but there are a number of conceptual reasons why we prefer the full human error rate as in [91] for ranking generative models by human preference. Still, we included the three metrics for all generative models in Appendix C.3 as the “Error Rate”, “R Error Rate“, “F Error Rate”, respectively, and they are all very highly correlated.\n \n    - On a conceptual basis, the average of R Error Rate and F Error Rate, what we simply call Error Rate, is our preferred choice to rank models as it has a few properties that neither R Error Rate nor F Error Rate have in isolation. Focusing on the suggestion that “the fake-->real case should be more important for evaluation”, we note that F Error Rate does not detect a scenario of “hyper-realistic generation”, where fake images look more realistic than the dataset on which a model is trained. For example, on our ImageNet and LSUN-bedroom datasets, some images from the training set look “less realistic” due to effects of data curation (lower resolution, aspect ratios, etc.). Nevertheless, these are real images that the model was trained on, and a generative model which has learned the true data distribution should be capable of generating images with these features. The F Error Rate metric does not capture this, but R Error Rate does, and hence Error Rate does as well. \n    - On a quantitative basis, we note that R Error Rate, F Error Rate, and Error Rate are very highly correlated with each other - *Pearson’s correlation coefficient between F Error Rate and Error Rate over all models is 0.99, and replacing the Error Rate with the F Error Rate results in no change to the trends seen in Figure 4 or the rest of the results*. One implication of the high correlation is that R Error Rate is not constant across experiments using the same training dataset, but different generated datasets, as one might suppose *a priori*. Instead, when the generated images are more realistic, humans make more mistakes on the real images as well. This phenomenon was previously discussed in [91], so we only discussed it in Appendix C.3.\n    - On a diagnostic basis, when analyzing individual participant trials, we found no evidence of a significant divergence of the measures (i.e. we saw no evidence of a high R Error Rate and low F Error Rate, which would be evidence of a participant simply selecting “fake” for every image). With the observation above that R Error Rate is higher when generated images are more realistic, we cannot filter out participants simply for having a high R Error Rate, because this is a normal outcome on the more challenging tasks. Thus, using this metric as a diagnostic to rule out poor performing participants would introduce a bias into the final results, as we would have to set a model-dependent threshold on the R Error Rate guided by prior knowledge.\n\n- On memorization: The questions you raise on defining the concepts of reconstructive memorization mathematically are very interesting, but to our knowledge there is no consistent formal definition for this in the literature and it is an open-ended question that deserves further study. While pixel-wise memorization is more direct to evaluate, it still requires a choice of threshold values to determine a sample as memorized. We followed the examples set by [11] and [58] for our analyses in Section 4.3, and included full details for reproducibility in Appendix B.2. We performed the memorization analysis as we felt it was necessary to determine if models with good FD_DINOv2 scores (i.e. mostly diffusion models) are just memorizing the training data, and thus “cheated” their human alignment; indeed we found that this is not the case. To our knowledge a memorization analysis has not been performed on the datasets in our paper. "}]

Author rebuttal:
We thank all the reviewers for their thoughtful feedback and the time they spent assessing our work. We are very encouraged by the largely positive feedback, including that our paper provides “significant contributions” (uifn), “presents a considerable number of insights” (uTCz), and “delivers a crucial message to the community” (6HZb). Below we address themes raised by more than one reviewer, and otherwise reply to each reviewer individually.

1. On selection, diversity, and potential bias of participants (xpCe, uTCz): Thank you for bringing this up. We agree that it is relevant to discuss, and that doing so will enhance our paper. Our attached rebuttal pdf includes demographic information about the participants (only those who explicitly agreed to provide and to allow us to use this information are included). It is clear that the set of participants is diverse, and that the results of the human evaluation experiment exhibited no biases from any of the diverse groups of participants (as the difference in means of normalized error rates between demographic groups is much smaller than the corresponding standard deviations). While our participants were recruited from a global crowd-sourcing platform, we did not explicitly select participants based on diversity due to various reasons:
    - In the first experiments we ran (on CIFAR-10), we found no systematic difference in participant’s performance based on demographics, and we thus decided to not filter based on demographics on subsequent experiments. This is indeed confirmed once again in the post-hoc analysis (on all datasets) included on the rebuttal pdf.
    - Not only do we not observe an empirical difference in responses based on demographics, but there is also no theoretical reason to believe there might be one. The dominant perspective in visual cognition is that vision is impenetrable, which is another way of saying bottom-up, or not influenced by cognition [A,B]. Another way of thinking about this would be to imagine the architecture of the visual system as a CNN. The processing is predominantly in the direction of the forward pass, or, from the retinae to the primary visual cortex to areas of higher cognition. Backward connections exist but serve the stability of perception rather than the content. In other words, a person’s culture or ethnicity is unlikely to affect the forward pass.
        - [A] Pylyshyn, Z. (1999). Is vision continuous with cognition?: The case for cognitive impenetrability of visual perception. Behavioral and brain sciences, 22(3), 341-365.
        - [B] Firestone, C., & Scholl, B. J. (2016). Cognition does not affect perception: Evaluating the evidence for “top-down” effects. Behavioral and brain sciences, 39, e229.
    - Conventionally, studies in psychophysics do not report demographic information on the participant pool beyond age and gender. This largely reflects widespread belief in the views expressed in the point above. As of August 4th 2023, a cursory examination of the 20 most recent open access articles in *Visual Cognition*, a popular journal dedicated to studies in visual psychophysics, confirms this. 17 of the 20 were empirical studies. Of these, all report age and gender. Only two report any demographic information on the participants’ ethnicity. Both were studies on face perception, which is a subfield where exposure to faces of different ethnicities has known effects on performance. In other words, unless there is an *a priori* reason to expect an effect of ethnicity or other demographic variable (other than age and gender), it would be unusual to report them. Nonetheless, we are happy to report additional demographic information anyway in the final version of our paper.
    
    In summary, our participants are diverse in spite of diversity not being a source of concern for bias in our study, and we find no systematic difference in participant’s performance based on demographics.

2. On including more detailed definitions of the metrics used in our paper (xpCe, 6HZb): Thank you for pointing out this area of improvement. In the final paper we will expand upon the current descriptions to add the full mathematical descriptions and explanations for each of the metrics used throughout the paper. 


Meta-review: This submission was reviewed by five knowledgeable referees. The reviewers raised concerns w.r.t. (1) the number of topics covered by the paper, which made the presentation slightly confusing (xpCe, uTCz); (2) the effect of rarity in the user study (xpCe); (3) the user's study participant selection (xpCe, uTCz); (4) missing comparisons with LPIPS (uifn); and (5) the positioning of the contribution w.r.t. prior work (uifn, 6HZb). The reviewers found the claims of the paper too strong given the evidence (xpCe), missed an in-depth limitations discussion (uTCz), and raised questions about the computation of the human error (SaoU). The rebuttal addressed most of the reviewers concerns by extensively discussing the participant selection process, the participant diversity, and the human error rate metric; by adding results to strengthen the claims; and by positioning their work w.r.t. the papers shared by the reviewers. The reviewers acknowledge the effort put by the authors in the rebuttal and discussion, the AC also acknowledges this effort. After discussion, most concerns appear addressed and 4/5 reviewers lean towards acceptance. Reviewer xpCe still has concerns w.r.t. the overstated subjective claims, and the human rating which they find not necessarily a good ground truth. Although the AC understands the arguments put forward by reviewer xpCe (humans are not the only end users of generative models), the AC also agrees with the argument of assessing the realism of images. This paper also received ethics reviews. Ethics reviewers raised a concern that the paper had not gone through IRB. The authors stated that their institution does not have an IRB process and appealed to the NeurIPS 2023 guidelines making authors at institutions with no IRB process exempt. The AC and SAC discussed this issue and side with the authors.

The AC recommends to accept and expects the authors to reword the overstated claims. In particular, the AC expect the authors to tone down the claims w.r.t. unfair treatment of diffusion models, Dino-v2, and Inception-v3, as suggested by the reviewers. The AC also encourages the authors to make the conclusions more concise in the final version of their manuscript.
Decision: Accept (poster)