SlowGuess/NeurIPS_2023_OCR huggingface dataset:
on ImageNet1k, which may be evidence towards using them for generative evaluation, we argue that the computational cost of such a large model is restrictive to the development of generative models. Often, metrics need to be tracked during model development for numerous training setups, hyperparameter settings, and epochs, which motivates networks with a lower computational cost. For reference, on a workstation with a single NVIDIA Titan V GPU, determining the representations of 50k samples takes  $\sim 4$  minutes with ViT-B/14,  $\sim 15$  minutes with ViT-L/14, and  $\sim 85$  minutes with ViT-G/14. Thus, smaller encoders that exhibit representation spaces matching the utility of larger encoders should be preferred.
Figure 8 shows the results of the model size sweep. Models are listed by ascending size. For CLIP we find that CLIP-B/32-OpenAI, CLIP-B/32-LAION2B, and CLIP-B/16-LAION2B show a much smaller correlation between FD and human error rate than the ViT-L/14 and ViT-G/14 models. We see that the model best aligned with the expensive ViT-G/14 - the highest performing CLIP model under linear evaluation protocols - is CLIP-L/14-DataComp-1B. We quantify this agreement with the FD correlation matrix shown on the right (the precision matrix, while not shown displays the same trends), and find a very high correlation between CLIP-L/14-DataComp-1B and CLIP-G/14-LAION2B.
For DINOv2 we see lower alignment of FD and human error rate when using the ViT-S/14 model in comparison to the larger three, and also find good agreement between the ViT-B/14, ViT-L/14, and ViT-G/14 models. We again quantify this agreement with the FD correlation matrix shown on the right (the precision matrix, while not shown displays the same trends), and find that the ViT-L/14 shows the highest correlation with ViT-G/14, although ViT-B/14 is highly correlated as well.
As evidenced by both the CLIP and DINOv2 results, ViT-L/14 provides a perceptual space nearing the ImageNet qualities of the largest encoders at a much lower computational cost. Thus we believe that ViT-L/14 provides a good tradeoff between representation quality and computation cost, and use it as the basis for all ViT-based encoders in this work. While we propose DINOv2-L/14 as the encoder that best improves upon Inception-V3, we suggest that the strong correlation of DINOv2-B/14, yet a factor of  $\sim 4$  lower computational cost, will provide useful during generative model development - metrics can be tracked using this model size during training, and final leaderboard values can be reported after training has concluded using DINOv2-L/14. Our codebase allows to easily do this.
C Human subject experiments
C.1 Test description
As noted in Section 3, the goal of our human subject experiments was to evaluate the fidelity of generated images in a scientifically grounded way across many models and datasets. The design of our tests was therefore specifically tailored to address the realism of generated images, and not other aspects of the modeled distributions, such as diversity. We further emphasize that our focus is on the realism of images that are generated for human use, not other uses (for example generating synthetic data to improve classifier performance [46], to debias datasets [108], or for dataset distillation [16]).
Figure 9: Screens displayed to participants during our StyleGAN-XL evaluation for ImageNet. Other experiments followed the same template. The bottom left shows the format of the 40 practice and 200 test trials, with the correct answer text shown only after a participant entered their response.
We found that the experiments from [120] for the  $\mathrm{HYPE}_{\infty}$  metric supported these goals, and thus we used them as a template. We first describe our method in detail, compare the differences from [120], and then present our full results.
In each test, generated images from a single model are directly compared to images from the training dataset - an absolute comparison [81]. Previous human experiments have used comparative ratings rather than absolute, for example by directly comparing pairs of models [99, 105, 117, 90, 94], or grouping together several models with a baseline and comparing all at once [24, 99, 96, 50, 27, 97]. The absolute approach has several advantages:
- Models are tested against a single, consistent baseline.
- The number of experiments scales linearly with the number of models to be tested. Pairwise testing requires a quadratic number of experiments.
- Experiments can be extended with new models in a consistent manner - additional experiments do not depend on previously used models.
- Comparisons of new results against previously-gathered results are analytically simple using between-subjects comparisons. The comparative paradigm would require mixing between-subjects and within-subjects designs.
- The performance of any given model does not depend on which other models were tested in the same experiment.
- Scores can indicate whether models generate images indistinguishable from the baseline, or if there is room for improvement (see Figure 2 where the optimal error rate of 0.5 is approached only for CIFAR10).
- Experiment results will not be outdated when researchers make progress with new models.
In total, 41 distinct experiments were conducted, each consisting of an instruction and training phase, followed by a testing phase. Research on how to collect high quality data from crowd sourcing platforms [71] shows that providing training on the task to be completed is the best way to improve
data quality, while providing financial incentives can also work, but pre-screening individuals by their on-platform reputation is ineffective. Hence, in the first phase we provide participants with background information, a request for consent, and clear instructions on their task, including the fact that exactly half of the presented images would be real. We then conducted 40 practice trials which were not used in the analysis of our results. Each trial was a two-alternative forced choice task, in which the participant was shown a single image which was either a "real" image from the training dataset, or a "fake" image generated by the model being tested. The participant was simply asked to judge if the image was real or fake with no alternatives, and they were provided feedback on the correct answer directly after they responded. In the testing phase, subjects are shown an additional 200 trials of the exact same format. Participants were unconstrained in their time to view and judge each image. There was a  $200\mathrm{ms}$  inter-trial interval before presentation of the subsequent image, and a delay of  $100\mathrm{ms}$  after image presentation before participants could enter a response to prevent accidentally advancing through trials. For reproducibility, in Figure 9 we show screenshots from an example test, the StyleGAN-XL model on ImageNet, as they were displayed to the participants. The experiments were created using PsychoPy [85] and hosted on Pavlovia [83]. The experiments were only administered on desktop or laptop computers, not on mobile devices or tablets, and image size was scaled to the user's display.
The choice to reveal correct answers to participants after they guess is not completely standard across the literature. Some authors have provided feedback [96, 120, 50], while others have avoided it [117, 66], and often no justification is provided. Ultimately, we want participants to make the most accurate decisions they are capable of, which is why we use a long training phase, and provide feedback throughout. The main argument against providing feedback during testing is that participants may "learn" and adjust their responses, which may in effect change the distribution of experimental data throughout the test. We do not find this to be concerning for the following reasons. First, our long training period is meant for participants to learn and improve their performance on the real task; most learning will occur early during the training phase, less so during the testing phase. Second, even without feedback, participants can still learn and change their approach to the task as they see more examples. Third, with or without feedback all participants have the same opportunity to learn, and so our experimental datapoints, the error rates of individual participants on 200 trials, are drawn from the same hypothetical distribution. In preliminary experiments we observed no significant difference between the error rates on the first and second halves of the test trials, showing that learning during the test phase does not create a large effect.
Ethical considerations Our institution does not have an internal review board (IRB) approval requirement for running crowd-sourced experiments, and thus has no formal approval process for experiments such as ours. In place of an IRB approval we followed an informal process internally prior to running the experiments, as recommended in the NeurIPS 2023 guidelines. The process involved: confirming what internal review processes existed and were required of us at our institution, ensuring that none of the images that would be shown to participants had any explicit or offensive content, informing participants that they may be shown images that could trigger common phobias, taking the test ourselves, and paying the participants above minimum wage.
Participant recruitment and compensation Human subject experiments for generative model evaluation are quite common, but often are ad-hoc and cannot be considered as randomized, controlled experiments. Sometimes authors themselves are the participants [48, 68], or volunteers known to the authors are used [24, 117, 66, 97], raising questions of bias. Luckily, there are several crowd-sourcing platforms to recruit willing and unbiased participants from the general population. For our experiment, participants were recruited from Prolific [86] which has been shown to provide the highest quality data in scientific studies about behavioural research platforms [31, 29]. While we did not filter eligible participants by their on-platform reputation, we did require participants to be fluent in English, the language used throughout the experiment, and to have completed college, or a university degree at least at the Bachelors level (we used this filter to maximize the probability of fluency in English, and thus of instructional obedience during the trials). No other filters were used, and statistics on the demographics about consenting participants are shown in Table 2. Comparisons between models and datasets are therefore between-subjects. For each experiment, which took a median of 15 minutes to complete, participants were paid at a median rate of £7.60/hr, including a base rate for completing the study which complied with the payment standards of the Prolific platform, and an incentive of £0.001 for each correct answer.
Table 2: Demographics and performance for the 897 of 1036 participants who voluntarily consented to release their demographics through Prolific. The normalized error rate accounts for the varying difficulty of tasks over different combinations of (dataset, model). A negative value means better performance on the task, with a value of 1 meaning 1 standard deviation above the mean within their task cohort of  $\sim 25$  participants. As intuitively expected, there is no significant performance difference based on demographic group.
Group	# Participants	Normalized error rate
Age group	19-29	574	-0.01±0.93
30-39	200	-0.18±0.95
40-49	77	-0.01±1.07
50+	46	0.52±1.21
Sex	Female	389	0.01±0.95
Male	507	-0.05±0.99
Prefer not to say	1	-
Highest education level	Doctorate degree	19	-0.32±0.70
Graduate degree	294	0.10±0.98
Technical/community college	121	0.02±1.02
Undergraduate degree	463	-0.09±0.96
Continent of birth	Africa	127	0.21±0.94
Asia	40	0.19±1.07
Europe	557	-0.02±0.96
North America	129	-0.19±0.99
Oceania	13	-0.46±0.99
South America	29	-0.36±0.87
Unknown	2	0.42±1.61
Ethnicity	Asian	44	0.34±1.12
Black	103	0.35±0.89
Mixed	86	-0.10±0.96
Other	44	-0.26±0.85
White	620	-0.08±0.96
The experiments for each of the 41 models were completed by at least 25 unique recruits (see Tables 4 through 7), totalling 1,036 different people, and providing over 207k individual test responses (along with more than 41k practice responses not included in our statistics), making this the largest human study evaluating generated image quality that we know of. A small number of experiments were completed by more than 25 participants due to unstable connections to the Prolific and Pavlovia servers causing logging issues.
Image selection Within each experiment, the images shown to a given participant were selected and randomized as follows. First, 2,000 images from the training dataset were randomly selected without replacement with classes in proportion to the dataset if applicable. The same set of training images were used for each model corresponding to that dataset. In a similar fashion, 2,000 model samples were selected (see Section 2 and Appendix A), in proportion to the dataset's class distribution if the model was class-conditional. For each participant, 100 real and 100 generated images were selected at random from these pools, again respecting class distributions where applicable. The number of times each generated image was viewed by a human therefore follows a binomial distribution with  $n \geq 25$  and  $p = 0.05$ . The 200 selected images were presented to the participant in a randomized order. The 40 practice images were chosen in a similar fashion, but had no overlap with the 4,000 test images, and were reused for all participants. Appendix D shows image samples for each model.
2The next largest similar experiment that we know of collected about 138k individual responses [120]. Human evaluations have also been conducted on text-to-image generation quality, where [90] collected about 40k individual responses, [94] collected about 25k, and more recently [81] collected about 33k responses.
C.2 Comparison to  $\mathbf{HYPE}_{\infty}$
Our experimental design was inspired by the  $\mathrm{HYPE}_{\infty}$  score [120], but we made several changes to improve the quality of collected data. We summarize the differences in Table 3. We doubled the number of images shown to each participant for more accurate estimation of each participant's error rate, while slightly lowering the number of participants per model. We used Prolific rather than MTurk because research shows that Prolific can provide better data quality due to their participant screening practices [31, 29]. Prolific also requires fair base pay rates based on time, so compensating participants solely on their number of correct responses is disallowed. The pre-screening test from [120] used a mix of data from all models they tested, which does not align with our preference for absolute comparisons and extensible experiments given in Appendix C.1, so we did not use it.
Each of the 41 models were evaluated based on human error rate [120], the fraction of the 200 test images which were incorrectly classified. This is a simple metric with the intuition that models with better fidelity produce images that are more difficult for humans to distinguish from real images. Detailed results by dataset that were summarized in Figure 2 and Table 1 from the main text are given in Tables 4 through 7. Along with the overall error rate, was also provide the error rates on the real and fake images separately. As in [120], we observe that the error rate on real images increases for models with higher fidelity, even though real images are drawn from the same pool of 2000 examples across models. We also provide the mean rate that participants guessed "real", noting that they were instructed that exactly half of presented images would be real. Finally, we give the mean time spent on the 200 test examples.
Table 3: Comparison between design of  $\mathrm{HYPE}_{\infty}$  experiments [120] and ours.
Ours	HYPE∞
Correct answer revealed to participant after their response	Yes	Not specified
Number of real/fake images in selection pools	2000	5000
Number of images shown to each participant	200	100
Number of participants for each experiment	≥ 25	30
Number of experiments conducted	41	13
Number of responses collected	207k	39k
Recruitment platform	Prolific	MTurk
Payment	Base pay £1.80 + £0.001 per correct answer	$1 for passing qualifying test + $0.02 per correct answer on main test
Qualification	Fluent in English, minimum education requirement	Achieve ≥65% accuracy on test using a mix of data from all models tested
Table 4: Detailed human experiment results on CIFAR10
Model	# Tests	Error Rate ↑	R Error Rate ↑	F Error Rate ↑	R Answer Rate	Time (s)
ACGAN	25	0.148 ± 0.013	0.181 ± 0.021	0.116 ± 0.016	0.467	217
BigGAN	25	0.387 ± 0.014	0.381 ± 0.021	0.393 ± 0.022	0.506	284
iDDPM-DDIM	26	0.400 ± 0.013	0.390 ± 0.019	0.410 ± 0.022	0.510	268
LOGAN	25	0.206 ± 0.020	0.204 ± 0.021	0.208 ± 0.026	0.502	220
LSGM-ODE	27	0.437 ± 0.010	0.417 ± 0.021	0.456 ± 0.018	0.519	232
MHGAN	25	0.336 ± 0.015	0.331 ± 0.016	0.341 ± 0.021	0.505	267
NVAE	25	0.131 ± 0.018	0.128 ± 0.015	0.134 ± 0.025	0.503	240
PFGM++	25	0.436 ± 0.011	0.422 ± 0.024	0.450 ± 0.025	0.514	243
ReACGAN	25	0.336 ± 0.016	0.326 ± 0.017	0.345 ± 0.026	0.510	247
RESFLOW	25	0.088 ± 0.008	0.090 ± 0.010	0.086 ± 0.011	0.498	199
StyleGAN2-ada	25	0.393 ± 0.012	0.387 ± 0.020	0.399 ± 0.022	0.506	328
StyleGAN-XL	25	0.399 ± 0.012	0.378 ± 0.018	0.420 ± 0.022	0.521	251
WGAN-GP	25	0.170 ± 0.013	0.158 ± 0.013	0.181 ± 0.022	0.511	221
Table 5: Detailed human experiment results on ImageNet
Model	# Tests	Error Rate ↑	R Error Rate ↑	F Error Rate ↑	R Answer Rate	Time (s)
ADM	25	0.266 ± 0.016	0.254 ± 0.021	0.279 ± 0.018	0.513	369
ADMG	25	0.248 ± 0.012	0.236 ± 0.021	0.259 ± 0.016	0.512	396
ADMG-ADMU	25	0.269 ± 0.016	0.240 ± 0.017	0.298 ± 0.024	0.529	344
BigGAN	25	0.158 ± 0.014	0.155 ± 0.016	0.161 ± 0.019	0.503	326
DiT-XL-2	25	0.286 ± 0.016	0.266 ± 0.026	0.307 ± 0.024	0.521	341
DiT-XL-2-guided	27	0.330 ± 0.013	0.329 ± 0.019	0.330 ± 0.017	0.500	352
GigaGAN	25	0.212 ± 0.015	0.212 ± 0.019	0.212 ± 0.015	0.500	398
LDM	26	0.309 ± 0.017	0.294 ± 0.023	0.323 ± 0.022	0.515	369
Mask-GIT	25	0.183 ± 0.016	0.164 ± 0.018	0.203 ± 0.025	0.520	278
RQ-Transformer	26	0.223 ± 0.012	0.205 ± 0.014	0.242 ± 0.017	0.518	308
StyleGAN-XL	25	0.153 ± 0.013	0.145 ± 0.014	0.162 ± 0.018	0.509	315
Table 6: Detailed human experiment results on LSUN-Bedroom
Model	# Tests	Error Rate ↑	R Error Rate ↑	F Error Rate ↑	R Answer Rate	Time (s)
ADM-dropout	25	0.387 ± 0.020	0.334 ± 0.027	0.440 ± 0.022	0.553	306
Consistency	25	0.155 ± 0.022	0.131 ± 0.026	0.179 ± 0.027	0.524	325
Diff-ProjGAN	26	0.148 ± 0.019	0.130 ± 0.018	0.167 ± 0.023	0.519	335
DDPM	25	0.267 ± 0.021	0.259 ± 0.026	0.276 ± 0.021	0.509	354
iDDPM	25	0.297 ± 0.021	0.263 ± 0.025	0.331 ± 0.028	0.534	336
Projected-GAN	25	0.142 ± 0.018	0.144 ± 0.022	0.140 ± 0.017	0.498	266
StyleGAN	26	0.217 ± 0.018	0.179 ± 0.017	0.254 ± 0.025	0.537	427
Unleash-Trans	25	0.209 ± 0.022	0.178 ± 0.024	0.240 ± 0.027	0.531	305
Table 7: Detailed human experiment results on FFHQ
Model	# Tests	Error Rate ↑	R Error Rate ↑	F Error Rate ↑	R Answer Rate	Time (s)
Efficient-vdVAE	25	0.088 ± 0.010	0.081 ± 0.013	0.096 ± 0.010	0.507	282
InsGen	25	0.255 ± 0.018	0.275 ± 0.025	0.236 ± 0.016	0.480	326
LDM	25	0.186 ± 0.016	0.195 ± 0.016	0.178 ± 0.022	0.492	429
Projected-GAN	25	0.120 ± 0.012	0.107 ± 0.011	0.134 ± 0.018	0.514	318
StyleGAN2-ada	25	0.215 ± 0.017	0.215 ± 0.019	0.216 ± 0.021	0.500	359
StyleGAN-XL	25	0.160 ± 0.012	0.163 ± 0.008	0.158 ± 0.021	0.498	340
StyleNAT	25	0.299 ± 0.019	0.314 ± 0.023	0.283 ± 0.022	0.485	387
StyleSwin	25	0.180 ± 0.012	0.198 ± 0.023	0.161 ± 0.010	0.481	288
Unleash-Trans	27	0.173 ± 0.015	0.142 ± 0.015	0.203 ± 0.022	0.530	330
Table 8: Layer used for GradCAM for each encoder.
Model	Layer Name
Inception	blocks.3.2
ConvNeXt	stages.3.blocks.2
SwAV	layer4.2
SimCLRv2	net.4.blocks.2.net.3
CLIP	visual.transformer.resblocks.11.ln_1
DINOv2	blocks.23(norm1
MAE	blocks.23(norm1
data2vec	model encoder(layer.23(layernorm_before
D Additional experimental results
D.1 Additional scatter plots and correlations
Figure 10 displays additional views of Figure 4, showing trends as a function of other popular metrics (recall, precision, and density), while Figures 11 and 12 display the correlation of additional metrics.
D.2 Representation spaces for generative evaluation
D.2.1 Experimental details on GradCAM and additional heatmaps
We follow the gradient based visualization technique of [63]. As the FD is defined only between large sets of images (50k), the method precomputes the representations of 50k images from the real dataset, and for a set of 49,999 generated images, then adds the additional image of interest using Grad-CAM to visualize the parts of the image that have the largest influence on FD. For ViT-based encoders, we use the Grad-CAM variation introduced by [37]. The layer GradCAM was applied to for each encoder can be found in Table 8.
To unify the color scheme across encoders, we changed the sign on the heatmap for CLIP, DINOv2, ConvNeXt and SimCLRv2. Indeed, the sign of the heatmap is given by the activations of the saliency layer that is visualized and does not reflect the sign of the gradient w.r.t the FD. Both bright yellow and deep blue can thus show an encoder's focus. The sign was changed for the listed encoders as it seemed to make better sense with the semantics of the images observed.
Additional heatmaps for all encoders on Imagenet, FFHQ and LSUN-Bedroom can be seen in Figures 13 and 14. On those heatmaps, the "generated" set is taken to be the real dataset, and not from any specific generative model.
D.2.2 Quantitative representation spaces
In conjunction with the preceding qualitative analysis we performed two quantitative analyses of the representation spaces, and display the results in Figure 15. First, we measured the perceptual score in representation space using the BAPPS 2AFC dataset [118], data from a two alternative forced choice (2AFC) test that asked human evaluators which of two distorted views is more similar to a reference. The perceptual score measures the alignment of the human preference with the relative distances between the reference and patches in representation space. In the "traditional" experiment patches were distorted using traditional photometric, noise, blur, spatial, and compression distortions, while another experiment distorted patches using "CNN-based" distortions such as autoencoding, denoising, colorization, and super-resolution. Interestingly, better classifiers achieve worse perceptual scores [62]. We find that SimCLR ranks worst on the perceptual score while the masked methods of MAE and data2vec score the highest. These results are perhaps unsurprising given their respective self-supervised training procedures - the contrastive SimCLR relies on a set of heavy augmentations while MAE and data2vec work closer to pixel space and do not use augmentations - but such results point to clear differences between perceptual spaces as a direct result of the self-supervised training objective. We note that for generative evaluation, distortions such as jitter do not necessarily change the semantic information of the image and thus should not change the representation. Such distortions are a minor fraction of the total dataset.
Figure 10: We show analogous results to Figure 4, with points coloured by additional metrics. Fréchet distance and human error rate for each generative model as measured by different encoders (columns) on different datasets (rows). Panels with a shaded background do not have strong  $|r| \geq 0.5$  and significant  $(p \leq 0.05)$  correlations between FD and human error rate.
Figure 11: Pearson correlation for all metrics using different encoders (rows) on different datasets (columns). Missing metrics require a validation or test set, which some datasets do not have.
Figure 12: Pearson correlation for additional metrics (including IS and sFID) for the Inception-V3 network. Supporting the conclusions of the main text, we find that these additional metrics are not strongly correlated with human evaluation on datasets more complex than the simplistic CIFAR10. Missing metrics require a validation or test set, which some datasets do not have.
ConvNeXt
SwAV
SimCLRv2