SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This procedure results in roughly  $1.3 \times 10^{6}$  images of size  $256 \times 256$ . We construct a reference batch of 100,000 training images matching our generated sets by sampling 100 images from each class without replacement. We then calculate FD using 50,000 images drawn without replacement each from the training set and the generated set. These choices result in a slight increase in FID values for most of the ImageNet models reported above, as most determine the FID of 50k generated samples and all 1.3MM training samples, but do not affect any model rankings.
A.3 FFHQ
Following the examples set in the StyleGAN repositories, we downsampled the original  $1024 \times 1024$  FFHQ dataset to  $256 \times 256$  using Lanczos interpolation.
- Efficient-vdVAE [43] https://github.com/Rayhane-mamah/Efficient-VDVAE. We used the 8-bit version from the following checkpoint https://storage.googleapis.com/dessa-public-files/efficient_vdvae/Pytorch/ffdq256_8bits基礎_checkpoints.zip. Reported FID N/A, ours=34.88.
- Insgen [113] https://github.com/genforce/insgen. We used the following checkpoint https://drive.google.com/file/d/10tSwESM_8S60EtisddR16-gzo6QW7YBM/view?usp=sharing. Reported FID=3.31,ours=3.46.
- LDM [93] https://github.com/CompVis/latent-diffusion. Reported FID=4.98, ours=8.11. To our knowledge the settings we used to generate the images were consistent with the codebase and with those used in the paper. We contacted the authors about this via email (as the GitHub page is full of unresolved issues) but received no response.
- Projected-GAN [97] https://github.com/autonomousvision/projected-gan. Reported FID=3.39, ours=3.46.
- StyleGAN2-ada [57] https://github.com/NVlabs/stylegan2-ada-pytorch. We used the following checkpoint https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada/pretrained/paper-fig7c-training-set-sweeps/ ffhq70k-paper256-ada.pkl. Reported FID=4.30,ours=5.30.The discrepancy
is purely due to different choices for downsampling: We used Lanczos filtering to downsample the original 1024x1024 pixel FFHQ images to 256x256, while their work used box filtering. Box filtering exactly reproduces the results reported in the StyleGAN2-ada paper, and we thank the authors for their work to help us solve this discrepancy. Please see the following github issue for more details https://github.com/NVlabs/stylegan2-ada-pytorch/issues/283.
- StyleGAN-XL [98] https://github.com/autonomousvision/stylegan-x1. We used the following checkpoint https://s3.eu-central-1.amazonaws.com/avg-projects/stylegan_x1/models/ffhq256.pkl. Reported FID=2.19, ours=2.26.
- StyleNAT [109] https://github.com/SHI-Labs/StyleNAT. We used the following checkpoint https://shi-labs.com/projects/stylenat/checkpoints/FFHQ256_940kflip.pt, Reported FID=2.046, ours=2.11.
- StyleSwin [116] https://github.com/microsoft/StyleSwin. We used the following checkpoint https://drive.google.com/file/d/10jYZ1zEWGNdivORFKv7KhXRmYko72Lj0/view?usp=sharing. Reported FID=2.81, ours=2.89.
- Unleashing-Transformers [9]: https://github.com/samb-t/unleashing-transformers. We used the following checkpoint https://github.com/NVlabs/ffhq-dataset. Reported FID=7.12, ours=9.02. We note that other discrepancies with the reported FID on FFHQ have previously been raised via the issues function of GitHub.
A.4 LSUN-Bedroom
- Four models used sets of 50k publicly available images provided at https://github.com/openai/guided-diffusion/tree/main/evaluations [25].
- ADMNet-dropout [25]. Reported FID=1.90, ours=2.20.
- DDPM [49]. Reported FID=4.89, ours=5.18.
- iDDPM [78]. Reported FID=4.24, ours=4.54.
- StyleGAN [58]. Reported FID=2.35, ours=2.65.
- Consistency [70]. We used the consistency training model provided at https://github.com/openai/consistency_models. Reported FID=7.85, ours=8.27.
- Diffusion-Projected GAN [110]. We used the pretrained projected GAN model provided at https://github.com/Zhendong-Wang/Diffusion-GAN. Reported FID=1.43, ours=1.79.
- Projected GAN [97]. We used the pretrained projected GAN model provided at https://github.com/autonomousvision/projected-gan. Reported FID=1.52, ours=2.23. The settings we used to generate the images were consistent with the codebase defaults, which were, as far as we could tell, consistent with those used in the paper. We contacted the authors both via email and the issues function of GitHub but received no response.
- Unleashing Transformers [9]. We used the pretrained model provided by the authors at https://github.com/samb-t/unleashing-transformers. Reported FID=3.64, ours=3.58.
B Metrics and encoders
To assess all aspects of generative models we include a number of diagnostic metrics that are designed to directly quantify the fidelity, diversity, rarity, or memorization of our 100k image samples. Such metrics are valuable for diagnostic purposes but difficult to use to directly rank generative models. Ranking requires metrics which group these concepts into a single value without a clear tradeoff. In total we assess each generative model across 17 metrics, both ranking and diagnostic. These metrics are computed in a number of supervised and self-supervised representation spaces. In total we include 9 encoders. Throughout this section, we denote generated samples as  $\{x_i^g\}_{i=1}^n$ , and real samples as  $\{x_j^r\}_{j=1}^m$ .
The following appendix (i) details each of the metrics,  $(ii)$  presents controlled experiments and additional checks for diversity and memorization metrics, and  $(iii)$  describes the encoders and motivates the choice of ViT model size used.
B.1 Metrics for ranking generative models
FD The Fréchet distance (FD) uses  $(\mu_r, \Sigma_r)$  and  $(\mu_g, \Sigma_g)$ , the sample mean and covariance of the real and generated representations, respectively, and is determined through:
$$
\mathrm {F D} \left(\mu_ {r}, \Sigma_ {r}, \mu_ {g}, \Sigma_ {g}\right) = \left\| \mu_ {r} - \mu_ {g} \right\| _ {2} ^ {2} + \operatorname {T r} \left(\Sigma_ {r} + \Sigma_ {g} - 2 \left(\Sigma_ {r} \Sigma_ {g}\right) ^ {\frac {1}{2}}\right). \tag {1}
$$
The FD corresponds to the Wasserstein-2 distance between Gaussians with the corresponding means and covariances, and is thus a valid metric between the first two moments of real and generated distributions. If these two distributions happen to be characterized by their first two moments (e.g. they are both Gaussian), then the FD becomes a metric between distributions, not just their first two moments. We use 50,000 images for both the real and generated images. To move away from the standard Inception-V3 network which [103] used as a feature extractor we drop the "I" from this metric and the following, where appropriate.
$\mathbf{FD}_{\infty}$ $\mathrm{FD}_{\infty}$  [22] aims to remove the inherent bias of the FD due to the finite number of samples. It is determined by evaluating the FD at 15 regular intervals over the number of samples  $N$ , from  $N = 5,000$  to  $N = 50,000$ , fitting a linear trend to the 15 data points, and using the trend to infer a FD value at  $N = \infty$ ,  $\mathrm{FD}_{\infty}$ .
sFID The spatial FID (sFID) [76] is the FID computed using a representation from the intermediate mixed 6/conv layer of the Inception-V3 network [103] trained for ImageNet1k [23], rather than the standard (pool3, 2048 dimensional) layer. The sFID relies on the Inception-V3 network, and thus we do not report it for other architectures.
KD The Kernel distance (KD) [8] aims to replace the FD with something that is a proper distance between distributions regardless of whether the distributions are characterized by their first two moments. In order to this, an unbiased estimate of the maximum mean discrepancy [39] is used:
$$
\operatorname {K D} \left(\left\{x _ {i} ^ {g} \right\} _ {i = 1} ^ {n}, \left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right) = \frac {1}{n (n - 1)} \sum_ {i \neq i ^ {\prime}} ^ {n} k \left(x _ {i} ^ {g}, x _ {i ^ {\prime}} ^ {g}\right) + \frac {1}{m (m - 1)} \sum_ {j \neq j ^ {\prime}} ^ {m} k \left(x _ {j} ^ {r}, x _ {j ^ {\prime}} ^ {r}\right) - \frac {2}{n m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} k \left(x _ {i} ^ {g}, x _ {j} ^ {r}\right), \tag {2}
$$
where  $k$  is a positive definite kernel that is chosen as a hyperparameter. We use the standard 3rd degree polynomial kernel.
IS The Inception score (IS) [96] is maximized when the entropy of the distribution of labels predicted by the Inception-V3 model is minimized for every input (i.e. generated image) and when the predictions are evenly distributed across all 1000 possible labels. It is given by:
$$
\operatorname {I S} \left(\left\{x _ {i} ^ {g} \right\} _ {i = 1} ^ {n}\right) = e ^ {\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {K L} \left(p \left(y \mid x _ {i} ^ {g}\right) \| p (y)\right)}, \tag {3}
$$
where  $p(y|x)$  denotes the output of the Inception-V3 network when  $x$  is the input, and  $p(y)$  the observed frequencies of labels in the training data. We use a refactored version from https://github.com/sbarratt/inception-score-pytorch/blob/master/inception_score.py to compute this metric. The IS relies on the Inception-V3 network, and thus we do not report it for other architectures.
FLS The feature likelihood score (FLS) [53] is a recently proposed density estimation method that requires training, generated, and test samples. The goal of FLS is to be as close to likelihood evaluation even when there is no likelihood available. In order to do this, the FLS fits a kernel density estimate (KDE) to the generated samples (after having been transformed by the encoder). The bandwidths of the KDE are chosen so as to maximize the log-likelihood of training data, and the FLS is then given by (an affine transformation of) the test log-likelihood obtained by the KDE. We use the default implementation from https://github.com/marcojira/fls.
B.2 Metrics for fidelity, diversity, and rarity
Precision, recall, density, coverage As proxies for sample fidelity we use precision [95, 64] and density [75]. Both precision and density are based upon nearest neighbours computed in a representation space. Precision counts the binary decision of whether a generated sample is contained in any neighbourhood sphere of training points, while density counts how many real-sample neighbourhood spheres contain the sample. To quantify sample diversity we use recall [95, 64] and coverage [75], which similarly to their counterparts precision and density, are based upon nearest neighbours computed in a representation space. Formally, precision is given by:
$$
\operatorname {p r e c i s i o n} \left(\left\{x _ {i} ^ {g} \right\} _ {i = 1} ^ {n}, \left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} \left(x _ {i} ^ {g} \in S \left(\left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right)\right), \tag {4}
$$
where  $\mathbb{1}(\cdot)$  denotes the indicator function,  $S(\{x_{j}^{r}\}_{j = 1}^{m}) = \cup_{j = 1}^{m}B(x_{j}^{r},\mathrm{NND}_{k}(x_{j}^{r}))$ , where  $B(x,r)$  denotes a Euclidean ball centered at  $x$  with radius  $r$ , and  $\mathrm{NND}_k(x_j^r)$  is the distance between  $x_{j}^{r}$  and its  $k^{th}$  nearest neighbour in  $\{x_j^r\}_{j = 1}^m$ , excluding itself. Similarly, recall is given by:
$$
\text {r e c a l l} \left(\left\{x _ {i} ^ {g} \right\} _ {i = 1} ^ {n}, \left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right) = \frac {1}{m} \sum_ {j = 1} ^ {m} \mathbb {1} \left(x _ {j} ^ {r} \in S \left(\left\{x _ {i} ^ {g} \right\} _ {i = 1} ^ {n}\right)\right), \tag {5}
$$
density is given by:
$$
\text {d e n s i t y} \left(\left\{x _ {i} ^ {g} \right\} _ {i = 1} ^ {n}, \left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right) = \frac {1}{k n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} \mathbb {1} \left(x _ {i} ^ {g} \in B \left(x _ {j} ^ {r}, \mathrm {N N D} _ {k} \left(x _ {j} ^ {r}\right)\right)\right), \tag {6}
$$
and coverage is given by:
$$
\text {c o v e r a g e} \left(\left\{x _ {i} ^ {g} \right\} _ {i = 1} ^ {n}, \left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right) = \frac {1}{m} \sum_ {j = 1} ^ {m} \max  _ {i = 1, \dots , n} \mathbb {1} \left(x _ {i} ^ {g} \in B \left(x _ {j} ^ {r}, \mathrm {N N D} _ {k} \left(x _ {j} ^ {r}\right)\right)\right). \tag {7}
$$
Precision, recall, density, and coverage are determined with the code provided at https://github.com/clovaai/generative-evaluation-prdc associated with the work [75]. We use 5 nearest neighbors  $(k = 5)$  and 10,000 samples throughout, as standard.
Rarity score To determine the "harity" of an individual image  $x$  we use the rarity score [42]. Similar to precision, recall, density, and coverage, the rarity score uses  $k$ -nearest neighbours:
$$
\operatorname {r a r i t y} \left(x, \left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right) = \min  _ {j \in J \left(x, \left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right)} \mathrm {N N D} _ {k} \left(x _ {j} ^ {r}\right), \tag {8}
$$
where  $J(x,\{x_j^r\}_{j = 1}^m) = \{j = 1,\dots ,m\mid x\in B(x_j^r,\mathrm{NND}_k(x_j^r))\}$
Vendi score To study inter- vs intra-class diversity for class-conditional image generation we utilise the Vendi score [33]. This score does not require a reference dataset. Formally, the Vendi score is given by:
$$
\operatorname {V S} \left(\left\{x _ {i} ^ {g} \right\} _ {i = 1} ^ {n}\right) = e ^ {- \sum_ {i = 1} ^ {n} \lambda_ {i} \log \lambda_ {i}}, \tag {9}
$$
where  $\lambda_{i}$  is the  $i^{th}$  eigenvalue of the  $n\times n$  matrix  $K / n$ , where  $K_{ii'} = k(x_i^g, x_{i'}^g)$  for a positive semi-definite kernel  $k$  (chosen as a hyperparameter) with  $k(x,x) = 1$ . The convention that  $0\log 0 = 0$  is used to compute the Vendi score. The Vendi score over the whole dataset reports the overall sample diversity, and when conditioned on class it quantifies the intra-class diversity of samples. We separate the two in order to study mode concentration of class conditional models - we find some models have high inter-class diversity, which is encouraged through the class conditioning, while lacking intra-class diversity. While recall and coverage can also be determined per class, the small number of generated samples available for each class (e.g. 100 for each ImageNet class) results in difficulty constructing robust nearest-neighbour-based estimates. As per the original paper, we use a linear kernel to compute the score. We also tried using the same polynomial kernel that we used for computing KD, but found no clear benefit from doing so.
B.3 Metrics for memorization
We also investigate a form of overfitting that we term memorization, in which models memorize individual images from their training data and emit them at generation time.
AuthPct AuthPct [1] deems each generated sample either as authentic or inauthentic and returns the fraction of authentic samples. Inauthentic samples are those for which the distance to the nearest point in the training set is less than the distance between that training sample and its nearest neighbour in the training set.
$\mathbf{C}_T$  The  $C_T$  score [70] is a global three sample non-parametric test explicitly designed to detect memorization. It uses the training set, the test set, and the generated samples, and summarizes how often training samples are closer to generated samples than they are to samples in the test set through a Mann-Whitney hypothesis test [69]. The data is split into a number of cells using  $k$ -means clustering, the score is determined in each cell, then the per-cell score is averaged into the final one. We use the default implementation from https://github.com/casey-meehan/data-copying, namely  $k = 3$  and preprocessing the representation space using a 64 component PCA.
FLS-POG We denote the percentage of overfit Gaussians as determined in FLS as FLS-POG [53]. For each generated sample we compute the difference between the train and test log-likelihoods, and report percentage of generated samples that had a higher log-likelihood under the training set than the test set. We use the default implementation from https://github.com/marcojira/cls/.
Memorization ratio with calibrated  $l_{2}$  distance The calibrated  $l_{2}$  distance [101] of a generated sample  $x_{i}^{g}$  is given by:
$$
l \left(x _ {i} ^ {g}, \left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right) = \frac {\left\| x _ {i} ^ {g} - \mathrm {N N} _ {1} \left(x _ {i} ^ {g}\right) \right\| _ {2}}{\frac {1}{k} \sum_ {k ^ {\prime} = 1} ^ {k} \left\| \mathrm {N N} _ {1} \left(x _ {i} ^ {g}\right) - \mathrm {N N} _ {k ^ {\prime}} \left(\mathrm {N N} _ {1} \left(x _ {i} ^ {g}\right)\right) \right\| _ {2}}, \tag {10}
$$
where  $\mathrm{NN}_k(x)$  denotes the  $k^{th}$  nearest neighbour (in Euclidean distance) to  $x$  in the training data without  $x$ , i.e.  $\{x_j^r\}_{j=1}^m \setminus \{x\}$ . Note that this metric is computed directly on pixel space, with no encoder being used. Low values of  $l(x_i^g, \{x_j^r\}_{j=1}^m)$  indicate that  $x_i^g$  memorized a training sample. A threshold  $\tau$  is set and the generated sample  $x_i^g$  is then considered as "memorized" if its calibrated  $l_2$  distance is smaller than  $\tau$ . The memorization ratio is given by the percentage of memorized samples:
$$
\text {m e m o r i z a t i o n} \left(\left\{x _ {i} ^ {g} \right\} _ {i = 1} ^ {n}, \left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {1} \left(l \left(x _ {i} ^ {g}, \left\{x _ {j} ^ {r} \right\} _ {j = 1} ^ {m}\right) <   \tau\right). \tag {11}
$$
While the memorization ratio can provide a good assessment of how much a model memorized its training samples (Section 5.2), no default configuration of the  $k$  and  $\tau$  hyperparameters achieves this across all models and datasets: these values need to be hand-tuned in order to meaningfully flag memorization. Thus, despite our recommendation to use this metric to detect memorization, we do not believe that it should be used as a basis to compare models.
B.4 Encoders
With the aim of finding a more generalized perceptual representation space across the span of natural images we employed a number of alternative encoders. Here we outline the specific implementations used, and provide evidence for the ViT-L/14 architecture choice. For self-supervised methods we include short descriptions on the family of training employed, grouping into categories based on those of [4].
ConvNeXt As a more modern supervised benchmark we use a ConvNeXt-large architecture trained on ImageNet22k [67]. We use the timm implementation provided here https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/convnext.py and select the convnext_large_in22k model.
SwAV SwAV [14] featured heavily in a previous work advocating for the use of self-supervised feature extractors for generative model evaluation [72], as at the time it was a state-of-the-art self-supervised image representation model. SwAV is trained on ImageNet1k using a canonical correlation analysis method, and learns by clustering data while enforcing consistency between cluster assignments produced for different views of an image. We use the ResNet50 architecture and weights from [72], with code provided
at https://github.com/stanis-morozov/self-supervised-gan-eval/blob/main/src/ self_supervised_gan_eval/resnet50.py.
SimCLRv2 SimCLRv2 [19] is a contrastive method which learns visual representations by encouraging the similarity between two often heavily augmented views of an image, while discouraging the similarity between distinct training images. SimCLRv2 is a CNN trained on ImageNet1k. We used the PyTorch implementation available at https://github.com/Separius/SimCLRv2-Pytorch.
CLIP CLIP is a multi-modal language-image model [87], which trains an image encoder and text encoder to predict which images were paired with which texts. We use CLIP ViT [87]. CLIP-B/32 was studied for generative evaluation in a controlled experiment in [7] (while they do not state the architecture used in the paper we found it at https://github.com/eyalbetzale1/fcd/blob/main/fcd.py). In the next section we show that this choice is sub-optimal, and that a better choice is the OpenCLIP ViT-L/14 implementation [51] trained on DataComp-1B [35]. Thus, we use OpenCLIP ViT-L/14 trained on DataComp-1B.
DINOv2 DINOv2 [80] is from the self-distillation/discriminative family, which learns representations by passing two different views of an image to two encoders and mapping one to the other by means of a predictor. DINOv2 is a ViT-L/14 architecture trained on a large custom dataset of 142M images built by combining common datasets for classification, segmentation, depth estimation, and retrieval tasks, plus additional images from the internet. We justify the architecture choice alongside CLIP in the following section.
MAE Masked autoencoder (MAE) [44] is from the masked image modelling family, and learns to directly reconstruct masked image patches. We use the ViT-L PyTorch implementation from https://github.com/facebookresearch/mae, which was trained on ImageNet1k.
data2vec data2vec [3] is from the masked image modelling family, and learns by predicting latent representations of the full input data based on a masked view of the input. We use the Hugging Face implementation found here https://huggingface.co/docs/transformers/model_doc/data2vec. We use ViT-L, which was trained on ImageNet1k.
DreamSim The DreamSim method [34] uses a dataset of human similarity judgments over image pairs in order to fine-tune encoders to better align with human perception of image similarity. DreamSim was publicly released after the initial submission of this paper. We also highlight that the focus of our work is different than DreamSim: our human trials focus on image quality rather than similarity, and our subsequent analysis uses these human judgements to evaluate current metrics for generative models, while DreamSim creates a dataset of human-perceived image-to-image similarities and designs an encoder that reflects these human judgements. In other words, DreamSim focuses on obtaining an encoder which maps images that humans assess as similar to nearby points in latent space, whereas we focus on finding an encoder where distances between probability distributions on its latent space, such as FD, correlate with human judgment of fidelity. Our experiments use the full DreamSim ensemble, the concatenation of the representations of three different self-supervised models (DINO [15], CLIP, and OpenCLIP) that have each been fine-tuned on the DreamSim dataset.
We note that while separating our encoders into such self-supervised families allows for a broad understanding of the different methodologies employed during training and hints at potential characteristics at their learned representation spaces, models can in practice employ a mix of approaches. In particular, DINOv2 also employs a masked modeling objective, however it was not masking images but rather the latent-space with a teacher network used to provide targets [80].
B.4.1 Determining ViT model size for generative evaluation
To determine which model size provides a good perceptual space for generative evaluation with ViT-based models we calculated FD and precision of the 11 ImageNet generative models under varying model sizes of CLIP and DINOv2 encoders. For CLIP we also studied models trained using different sets of training data (OpenAI LAION-400M, LIAON-2B, and DataComp-1B), while DINOv2 encoders all used an identical training set. CLIP models not denoted with OpenAI were from OpenCLIP. While larger models achieve better accuracy under a linear evaluation protocol
Figure 8: ImageNet results as a function of ViT model size for CLIP (top) and DINOv2 (bottom).