The Role of ImageNet Classes in Fréchet Inception Distance
Authors: Tuomas Kynkäänniemi
Tero Karras
Miika Aittala
Timo Aila
Jaakko Lehtinen
Venue: ICLR 2023 notable top 25% (2023)
Keywords: generative models
evaluation
Fréchet Inception Distance

Abstract: Fréchet Inception Distance (FID) is the primary metric for ranking models in data-driven generative modeling. While remarkably successful, the metric is known to sometimes disagree with human judgement. We investigate a root cause of these discrepancies, and visualize what FID "looks at" in generated images. We show that the feature space that FID is (typically) computed in is so close to the ImageNet classifications that aligning the histograms of Top-$N$ classifications between sets of generated and real images can reduce FID substantially — without actually improving the quality of results. Thus, we conclude that FID is prone to intentional or accidental distortions. As a practical example of an accidental distortion, we discuss a case where an ImageNet pre-trained FastGAN achieves a FID comparable to StyleGAN2, while being worse in terms of human evaluation.

Published as a conference paper at ICLR 2023
THE ROLE OF IMAGENET CLASSES IN FRÉCHET INCEPTION DISTANCE
Tuomas Kynkäänniemi
Tero Karras
Aalto University
NVIDIA
tuomas. kynkaanni emi @aalto. fi
ckarras@nvidia.com
Miika Aittala
Timo Aila NVIDIA tai taila@nvidia.com
Jaakko Lehtinen
NVIDIA
Aalto University & NVIDIA inehtinen@evidia com
maittalaenvidia.com
ABSTRACT
Fréchet Inception Distance (FID) is the primary metric for ranking models in data- driven generative modeling. While remarkably successful, the metric is known sometimes disagree with human judgement. We investigate a root cause of these discrepancies, and visualize what FID "looks at" in generated images. We show that the feature space that FID is (typically) computed in is so close to the Im- ageNet classifications that aligning the histograms of Top-N classifications be- tween sets of generated and real images can reduce FID substantially - without actually improving the quality of results. Thus, we conclude that FID is prone to intentional or accidental distortions. As a practical example of an accidental distortion, we discuss case where an ImageNet pre-trained FastGAN achieves a FID comparable to StyleGAN2, while being worse in terms of human evaluation.
INTRODUCTION
Generative modeling has been an extremely active research topic in recent years. Many promi- nent model types, such as generative adversarial networks (GAN) (Goodfellow et al., 2014), varia- tional autoencoders (VAE) (Kingma & Welling, 2014), autoregressive models (van den Oord et al., 2016b;a), flow models (Dinh et al., 2017: Kingma & Dhariwal, 2018) and diffusion models (Sohl- Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020) have seen significant improvement. Additionally, these models have been applied to rich set of downstream tasks, such as realistic im- age synthesis (Brock et al., 2019; Razavi et al., 2019; Esser et al., 2021; Karras al., 2019; 2020b;a; 2021), unsupervised domain translation (Zhu et al., 2017: Choi et al., 2020; Kim et al., 2020), image super resolution (Ledig et al., 2017; Kligler (Park et al., 2019; 2020; Huang et al., 2022) and generating images based on a text prompt (Ramesh al., 2021; Nichol et al., 2022; Ramesh al., 2022; Saharia et al., 2022).
al., 2019; Saharia et al., 2021), image editing
Given the large number of applications and rapid development of the models, designing evaluation metrics for benchmarking their performance is an increasingly important topic. It is crucial to reli- ably rank models and pinpoint improvements caused by specific changes in the models or training setups. Ideally, generative model should produce samples that indistinguishable from the train- ing set, while covering all of its variation. To quantitatively measure these aspects, numerous metrics have been proposed, including Inception Score (IS) (Salimans al., 2016), Fréchet Inception Dis- tance (FID) (Heusel et al., 2017). Kernel Inception Distance (KID) (Binkowski et al., 2018), 8), and Precision/Recall (Sajjadi et al., 2018: Kynkäänniemi et al., 2019; Naeem al., 2020). Among these metrics, FID continues to be the primary tool for quantifying progress.
The key idea in FID (Heusel et al., 2017) is to separately embed real and generated images to a vesion-relevant feature space, and compute distance between the two distributions, as illustrated in Figure In practice, the feature space is the penultimate layer (pool3, 2048 features) of an Im- ageNet (Deng et al., 2009) pre-trained Inception-Vy classifier network (Szegedy al., 2016), and the distance is computed as follows. The distributions of real and generated embeddings are sepa- rately approximated by multivariate Gaussians, and their alignment is quantified using the Fréchet

---

Published as a conference paper at ICLR 2023
Inception-V V3
Real Real Jeatures
Affine
Real logis
Softmax
Protabilines
Rimurgation
Resi
Nim M.,)
Wis x2)
FID
distrnce
Logits FID (Sec.
distance
Match Top- (Soc.
Nips Mi)
Ni mng)
Binamaation
Inception-V V33
Teaturso
Affine
Logits
Softmax
Generated
logit
Logits
Class grobabilttion
Figure 1: Overview of the Fréchet Inception Distance (FID) (Heusel et al., 2017). First, the real and generated images are separately passed through pre-trained classifier network, typically the Inception- V3 (Szegedy et al., 6), to produce two sets of feature vectors. Then, both distributions of features approximated with multivariate Gaussians, and FID is defined as the Fréchet distance between the two Gaussians. In Section 3, we will compute alternative FIDs in the feature spaces of logits and class probabilities, instead of the usual pre-logit space.
(equivalently, the 2- 2-Wasserstein earth mover's) distance (Dowson & Landau, 1982)
FID (pr: Ex; Mg: Eg) = p4, - Tr Σ, + Σ 2
(I)
where (Jl, E and (My. Σg) denote the sample mean and covariance of the embeddings of the real and generated data, respectively, and Tr(-) indicates the matrix trace. By measuring the distance between the real and generated emheddings, FID is a clear improvement over IS that ignores the real data altogether. FID has been found to correlate reasonably well with human judgments of the fidelity of generated images (Heusel et al., 2017; Xu et al., 2018; Lucic et al., 2018), while being conceptually simple and fast to compute.
Unfortunately, FID conflates the resemblance to real data and the amount of variation to a single value (Sajjadi al., 2018; Kynkäänniemi et al., 2019), and its numerical value is significantly af- fected by various details, including the sample count (Binkowski et al., 2018; Chong & Forsyth, 2020), the exact instance of the feature network, and even low-level image processing (Parmar al., 2022). Appendix A gives numerical examples of these effects. Furthermore, several authors (Karras et al., 2020b; Morozov et al., 2021; Nash et al., 2021; Borji, 2022; Alfarra et al., 2022) observe that there exists discrepancy in the model ranking between human judgement and FID in non- ImageNet data, and proceed to introduce alternative metrics. Complementary to these works, we focus elucidating why these discrepancies exist and what exactly is the role of ImageNet classes.
The implicit assumption FID is that the feature space embeddings have general perceptual rel- evance. If this were the case, an improvement in FID would indicate corresponding perceptual improvement in the generated images. While feature spaces with approximately this property have been identified (Zhang al., 2018), there are several reasons why we doubt that FID's feature space behaves like this. First, the known perceptual feature spaces have very high dimensionality (~~M), partially because they consider the spatial position of features in addition their presence. Unfortu- nately, there may be contradiction between perceptual relevance and distribution statistics. It is not clear how much perceptual relevance small feature spaces (2048D for FID) can have, but is also hard to see how distribution statistics could be compared in nigh-dimensional feature spaces using a finite amount of data. Second, FID's feature space is specialized to ImageNet classification, and is thus allowed to be blind to any image features that fail to help with this goal. Third, FID's feature space ("pre-logits") is only one affine transformation away from the logits, from which a softmax produces the ImageNet class probabilities. We can thus argue that the features correspond almost directly to ImageNet classes (see Appendix B). Fourth, ImageNet classifiers are known to base their decisions primarily textures instead of shapes (Geirhos et al., 2019; Hermann et al., 2020).
Together, these properties have important practical consequences that we set out to investigate. In Section 2 gradient-based visualization technique, Grad-CAM (Selvaraju al., 2017), to
2

---

Published as a conference paper at ICLR 2023
Real features
Inception-N
(Mg Σ
Spatial average
Fréchet distance
Feature maps x 8 x 2048
New generated image
Feature vector I
(116 Ex
Gradient of Fréchet distance wit.. feature maps
Upsample
(1 Mg
FID sensitivity heatmap
Grad-CAM
Gen. features
Figure 2: Visualizing which regions of an image FID is the most sensitive to. We augment the pre-computed feature statistics with a newly generated image, compute the FID, and use Grad- CAM (Selvaraju et al., 2017) to visualize the spatial importance in low-resolution feature maps that are subsequently upsampled to match the input resolution.
visualize what FID "looks at" in generated images, and observe that its fixation on the most promi- nent ImageNet classes makes it more interested in, for example, seat belts and suits, than the hu- man faces in FFHQ (Karras et al., 2019). It becomes clear that when a significant domain gap exists between dataset of interest and ImageNet, many of the activations related ImageNet class templates are rather coincidental. We call such poorly fitting templates fringe features or fringe classes. As matching the distribution of such fringe classes between real and generated images becomes an obvious way of manipulating FID, we cxamine such "attacks" in detail in Section 3. The unfortunate outcome is that FID can be significantly improved with hardly any improvement in the generated images by selecting subset of images that happen to match some number of fringe features with the real data. We conclude with an example of practical relevance in Sec- tion 4, showing that FID can be unreliable when ImageNet pre-trained discriminators (Sauer et al., 2021; Kumari et al., 2022) are used in GANs. Some of the improvement in FID comes from ac- cidental leaking of ImageNet features, and the consequent better reproduction of ImageNet-lin -ike aspects in the real data. We hope that the new tools we provide open new opportunities to better understand the existing evaluation metrics and develop new ones in the future. Code is available ht / thub. waam//ynkaaato wttpyynnaaaattteeeaaaaeeeeeeeeee role-o f- www.-f--emneeeelleeeeeeeeeeeeeee
in-fidd
WHAT DOES FID LOOK AT IN AN IMAGE?
We will now inspect which parts of an image FID is the most sensitive to. We rely on Grad- CAM (Selvaraju et al., 2017) that has been extensively used for visualizing the parts of an image that contribute the most to the decisions of a classifier. We want to use similarly for visualizing which parts of one generated image contribute the most to FID. A key challenge is that FID is de- fined only between large sets of images (50k), not for a single image. We address this difficulty by pre-computing the required statistics for set of 49,99 999 generated images, and augmenting them then use Grad-CAM to visualize the parts of the image that
with the additional image of interest. have the largest influence FID. We will first explain visualization technique in more detail, followed by observations from individual images, and from aggregates images.
Our visualization technique Figure gives an outline of our visualization technique. Assume we have pre-computed the mean and covariance statistics (H,, E,) and (/1.: Eg) for 50,000 real and 49,999 generated images, respectively. These Gaussians are treated as constants in our visualization. Now, we want to update the statistics of the generated images by adding one new image. Computing the FID from the updated statisties allows us to visualize which parts of the added image influence it the most. Given image, we feed it through the Inception-V3 network to get activations A before the pool3 layer. The spatial resolution here is x 8 and there are 2048 feature maps. The spatial averages of these feature maps correspond to the 2048-dimensional feature space where FID calculated. We then update the pre-computed statistics (#g. Σg) by including the features f of the new sample (Pebay, 2008): (1L N Mg + J. Σ N-1 - + ( ) -4g)). Here, N = 50, 000 is the size of the updated set. To complete the forward pass, we evaluate FID using these modified statistics.
N-
3

---

Published as a conference paper at ICLR 2023
Micrepboac
Moctarboard
Oboc
Bow
Acalemic gown
Bow de
falt
Abasa
Serthescape
Kgrptiio
Wankbaain
kgyptim
Wanden
Tuub
Pembaa
Figure 3: StyleGAN2 generated images along with heatmap visualizations of the image regions that FID considers important in FFHQ (top) and LSUN CAT (bottom). Yellow indicates regions that are more important and blue regions that are less important, i.e., modifying the content of the yellow regions affects FID most strongly. As many of the yellow areas are completely outside the intended subject, we sought an explanation from the ImageNet Top-3 class predictions. FID very strongly focused on the area that corresponds to the predicted Top-1 I class whatever that may be. We discuss the qualitative difference between FFHQ and LSUN CAT in the main text.
In backward pass, we first estimate the importance ag for each of the k feature maps
2
OFID(H, E. u.. Eg)
a 8 ΣΣ
0Ah
Then, an 8 x 8-pixel spatial importance map is computed as linear combination Ee o Ah. Note that Selvaraju et al. (2017) originally suggested using a ReLU to only visualize the regions that have positive effect the probability of given class; do not employ this trick, since interested in visualizing both positive and negative effects on FID. Additionally, we upsample the importance map using Lanczos filtering match the dimensions of the input image. Finally, vert the values of the importance map to a heatmap visualization. See Appendix for comparisons of our FID heatmaps and standard Grad-CAM.
Observations from individual images Figure shows heatmaps of the most important regions for FID FFHQ (Karras et al., 2019) and LSSUN CAT (Yu al., 2015). Given that in FFHQ the goal is to generate realistic human faces, the regions FID considers most important seem rather unproductive. They are typically outside the person's face. To better understand why this might happen, recall that ImageNet does not include "person" or "face" category. Instead, some other fringe features (and thus classes) are necessarily activated for each generated image. Interestingly, observe that the heatmaps seem to correspond very well with the areas that the image's ImageNet Top-1 class prediction would occupy. Note that these ImageNet predictions were not used when computing the heatmap; they are simply tool for understanding what FID focusing on. As low FID) mandates that the distributions of the most prominent fringe classes are matched between real and generated images, one has to wonder how relevant it really is to match the distributions of "seat belt" or "oboe" in this dataset. In any case, managing to perform such matching would seem open possible loophole in FID; will investigate this further in Section
In contrast to human faces, ImageNet does include multiple categories of cats. This helps FID to much better focus on the subject matter in LSUN CAT, although some fringe features from the image background are still getting picked up. and very low FID would require that "washbasin", etc. are similarly detected in real and generated images.
4

---

Published as a conference paper at ICLR 2023
All classes
Bow tie
Scat belt
Mortarboard
Number of images
Real
Mean image
Generated
Mean hestrrap
Mocarvent
Lipitk
Bunsc
MI. unco
tird ad
Fur ooe
Coodoy 3e
Suk
Sworkit Feeter too
Pajars
(a)
(b)
Figure 4: (a) Distribution of the ImageNet Top-1 classes, predicted by Inception- V3, for real and StyleGAN2 generated images in FFHQ. (b) Mean images and Grad-CAM heatmaps among all classes, and images classified as "bow tie", "seat belt" and "mortarboard". These were computed from generated images. Strikingly, when averaging over all classes, FID is the most sensitive to ImageNet objects that are located outside the face area.
Observations from aggregates of images Figure 4a shows the distribution of ImageNet Top-1 classifications for real and generated images in FFHQ Typically, the predicted classes acces- sories, but their presence might be correlated with persons in the images. In a further test, we calculated a heatmap of the average sensitivity over 50k generated images. Figure 4b shows average images and heatmaps for all classes, and for images that are classified to some specific class (e.g., "bow tie"). If average over all classes, FID is the most sensitive to other areas of the image than the human face. If we compute a heatmap of the average sensitivity within class, they highlight the regions where the corresponding ImageNet class is typically located in the images. Appendix C provides additional single image and average heatmaps. It also provides further validation that the image regions FID the most sensitive to are correctly highlighted by our approach.
In related work, van Steenkiste et al. (2020) found that in images with multiple salient objects, FID focuses on only one of them and hypothesized that the metric could be easily fooled by gener- ative model that focuses on some image statistics (e.g., generating the correct number of objects) rather than content. They believe that this is because Inception V3 is trained on a single object classification.
PROBING THE PERCEPTUAL NULL SPACE IN FID
Our findings with Grad-CAM raise an interesting question: to what extent could FID be improved by merely nudging the generator to produce images that get classified to the same ImageNet classes as the real data? As we are interested a potential weakness nudging to happen so that the generated results do not actually improve in any real sense. In our terminology, operations that change FID without changing the generated results in a perceptible way are exploiting the perceptual null space of FID.
FID, we furthermore want this
We will first test simple Top-1 (ImageNet classification) histogram matching between real and generated data. As this has only a modest impact FID, we proceed to develop more general distribution resampling method. This approach manages to reduce FID very significantly, indicating that there exists large perceptual null space in FID. We then explore the role of other likely class labels, in addition to the Top-1 label, by aligning the Top-N histograms. In Section 4 we observe that this theoretical weakness also has practical relevance.
In our experiments, we use StyleGAN2 auto-config trained in 256 x 256 resolution without adaptive discriminator augmentation (ADA). The only exception is AFHQ-v2 DoG, where we enable ADA and train in 512 x 512 resolution.' Following standard practice, we compute FID against the training set, using 50k randomly chosen real and generated images and the official TensorFlow version of Inception-V3.2 A difference between our FIDs for StyleGAN2 and the ones reported by Karras et al. (2020a) is caused by the use of different training configurations.
We use the official code https: htt///gittt con httpb///././/.aba//t/ylggaggaaa -datttoorr----occch Pttpps/ /downl oad, tensorf www.orggmood a/ t/taa/////m//aa/maa/// haaage//mmageeet//nneetoooo 10-20111--0000000000000000000000
5

---

Published as a conference paper at ICLR 2023
Table 1: Results of Top-1 histogram matching. We compare the FID of randomly sampled images (FID) against ones that have been resampled to match the Top-1 histogram of the training data (FID Io The numbers represent averages over five FID evaluations. Additionally, we report the corresponding numbers by replacing the Inception V3 feature space with ResNet-50 (FIDReNNet-s)), SwAV (FIDswav), ) and CLIP features (FIDeLIP). Note that we use these alternative feature spaces only when computing FID; the resampling is still done using Inception- V3. The numerical values between different features spaces are not comparable.
Dataset
FID
FID Fi
FiDeeeeer 50
FID
Tip-1
FID
FIDeLI
FiD
Tup-
FFHQ
5.30
4.70
(-11.3%)
6.11
5.59
(-8,5%)
1.42
1.41 (-0.7%)
2.76
2.74 (-0.7%)
LSUN CAT
8.25
7.37
(-10.75))
12.33
11.29
(-8.1%)
2.99
2.96 (-1.0%
8.94
8.83 (-1.2%)
LSUN CAR
5.65
5.17
(-8.5%)
8.79
8.51
(-3.2%)
2.39
2.38 (-0.4%)
7.75
7.73 (-0.5%)
LSUN PLACES
12.96
11.76
(-9.3%)
15.20
13.61
10.0%)
3.12
3.02 (-3:2%))
16.35
16.17 (-1.1%)
AFHQ-V2 DoG
10.25
9.39 (-8.4%)
13.71
13.19
(-0,8%)
2.78
2.77 (-0.4%
4.25
4.16 (-2.1%)
Top-1 histogram matching Based the Grad-CAM visualizations in Section 2, one might sus- pect that to achieve a low FID, it would be sufficient to match the Top-1 class histograms between the sets of real and generated images. We tested this hypothesis by computing the Top-1 histogram for 50k real images, and then sampling an equal number of unique generated images for each Top-1 class. This was done by looking at the class probabilities at the output of the Inception- V classifier (Figure 1), and discarding the generated images that fall into a bin that is already full. Over multiple datasets, this simple Top-1 histogram matching consistently improves FID, by ~ 10% (Table 1).
Does this mean that the set of generated images actually improved? A genuine improvement should also be clearly visible in FIDs computed using alternative feature spaces. To this end, we calculated FIDs in the feature spaces of ResNet-50 ImageNet classifier (FIDResNet-50) (He al., 2016), self- supervised SWAV classifier (FIDswAv) (Caron et al., 2020; Morozov et al., 2021), and CLIP image encoder (FIDeLI)) (Radford et al., 2021; Sauer et al., 2021). Interestingly. FeDResNet-so drops almost as much as the original FID, even though the resampling was carried out with the Inception V3 fea- tures. This makes sense because both feature spaces are necessarily very sensitive to the ImageNet classes. FWD drops substantially less, probably because it was never trained to classify the Ima- geNet data. The muted decrease in FIDCLIP also in line with expectations because CLIPP never saw ImageNet data; it was trained with a different task of matching images with captions. 4 We can thus conclude that the observed decrease in FID is closely related to the degree of ImageNet pre-training, and that the alternative feature spaces fail to confirm clear increase in the result quality.
As a ten percent reduction in FID may not be significant enough for a human observer to draw reliable conclusions from the sets of generated images, we proceed to generalize the histogram matching to induce a much larger drop in FID.
Matching all fringe features We will now design a general technique for resampling the distri- bution of generated images, with the goal of approximately matching all fringe features. We will subsequently modify this approach to do Top-N histogram matching, extending the simple "draw samples until it falls into the right bin" bin" -approchh for Top-N would be computationally infeasible.
Our idea is to first generate larger set of candidate images (5x oversampling), and then carefully select a subset of these candidates so that FID decreases. We approach this by directly optimiz- ing FID as follows. First, select candidate set of 250k generated images and compute their Inception v3 features. We then assign a non- negatiie scalar weight w; to each generated image, and optimize the weights to minimize FID computed from weighted means and covariances of the gener- ated images. After optimization, we use the weights as sampling probabilities and draw 50k random samples with replacement from the set of 250k candidate images. More precisely, we optimize
min (114 1g (w)) 112 Tr E. Ex(w) - (1)
weighted mean and covariance of generated features fú, respectively. In practice we optimize the where (y(w)
- ()())) T (f: - Hy(w)) are the Σ wifi and Eg(w) Σ. E, wy (f: wv
Σ. my
We use the ViT-B/32 model available in https: /ttg//gttttu/tt com/ www.:pena/ioiiccooooo opena /
'One may argue that FIDCuP is less sensitive and that 1% decrease is significant. However, in Section 4, we show a case where FIDeLIP decreases by 40% and observe qualitative improvement in the results.

---

Published as a conference paper at ICLR 2023
Table 2: Results of matching all fringe features. We compare the FID of randomly sampled im- ages (FID) against ones that have been resampled to approximately match all fringe features with the training data (FID"). The numbers represent averages over ten FID evaluations. FeDResNet-50- FID, CUP match the descriptions in Table 1. The gray column (FID*) shows an additional experiment where the resampling is done using logits instead of the usual pre-logits.
Dataset
FID
FID
FID
FID
FID
FID
FIDGUP
FID
FFHQ
5.30
1,78
(-66.4%) 2.22 (-56.195)
6.11
3.85
(-47.055)
1.42
1.24 (-12.78)
2.76
2.64 (-4.3%)
LSUN CAT
8.25
3.05
(-63.0)
3.88 (-58.075
12:33
7.17
(-41.855)
2.99
2.71 (-0.4%)
8.94
7.98 (-10.7%)
LSUN CAR
5.65
2.11
02.7%)
2.59
( (4.25)
8.79
5.88
88.155)
2.39
2.15 < 10.090
7.75
7.33 6449)
LSUN PLACES
12.96
3.59
(-72.1%) 4.43 (-65-4%)
15.20
9.35
(-16 190)
3.12
2.60 (-16.7%)
16.35
14.54
(-1.15))
AFHQ-V2 DoG
10:25
5.92
(-42.9%)
6:27
(-38.895)
13.71
11:38
(-17.190
2.78
2.62 (-5.895)
4.25
4.04 (-49%%
(a) Images with small weights
(b) Images with large weights
Figure 5: Uncurated random StyleGAN2 samples from images with (a) the smallest 10% of weights and (b) the largest 10% of weights after optimizing the weights to improve FID. Both sets con- tain both realistic images and images with clear visual artifacts in roughly equal proportions. See Appendix D for larger sample.
weights in Equation 3 via gradient descent. We parameterize the weights as log(w;) to avoid negative values and facilitate easy conversion to probabílities. Note that we do not optimize the parameters of the generator network, only the sampling weights that are assigned to each generated image in the candidate set. Optimizing FID directly in this way is form of adversarial attack, but not nearly as strong as modifying the images or the generator to directly attack the Inception- V3 network. In any case, our goal only to elucidate the perceptual null space FID, and we do not advocate this resampling step to improve the quality of models (Issenhuth et al., 2022; Humayun et al., 2022).
Table shows that FIDs can be drastically reduced using this approach. An improvement by as much as 60% would be considered major breakthrough in generative modeling, and it should be completely obvious when looking at the generated images. Yet, the uncurated grids in Appendix indisputable improvement. To confirm the visual result quantitatively,
D fail to demonstrate again compute FIDs of the resampled distributions the alternative feature spaces. We see a sub- stantially smaller improvement in feature spaces that did not use ImageNet classifier pre-training. While it is possible that the generated results actually improved in some minor way, we can nev- ertheless conclude that a vast majority of the improvement in FID occurred in its perceptual null space, and that this null space is therefore quite large. In other words, FID can be manipulated to great extent through the ImageNet classification probabilities, without meaningfully improving the generated results.
Figure 5 further shows images that obtain small and large weights in the optimization; it is not obvious that there is visual difference between the sets, indicating that the huge improvement in FID (-66.4%) cannot simply attributed to discarding images with clear artifacts. Appendix also shows that the resampling fools KID just as thoroughly as FID, even though we do not directly optimize KID.
Finally, it makes only small difference whether the weight optimization is done in the typical pre-logit space (denoted FID"L) in the logit space (FID'), confirming that these spaces encode approximately the same information. Figure illustrates the difference between these spaces.
Top-N histogram matching The drastic FID reduction observed in the previous section provides clues about the upper bound of the size of the perceptual null space. We will now further explore the nature of this null space by extending our resampling method to approximate Top-N histogram
7

---

Published as a conference paper at ICLR 2023
6.0
3.12
5.0
2.60
FID, match Top-N
CLIP
FID
FID, match Middle-N
4.0
2.0%
CLP match Top-N
FIDELI.. match Middle-N
3.0
1.56
2.0 25
50
100
250
500
Number of classes (N)
Figure 6: Softly matching Top-N class distributions in FFHQ through resampling. FID (solid curves, left-hand y scale) decreases sharply with increasing number N of classes included in the Top-N indicator vectors. At the same time, FIDCLIP (dashed curves, right-hand y scale) remains almost constant, indicating that the apparent improvements in FID are superfluous. The orange control curves have been computed from classes in the middle of the sorted probability vectors, indicating that the top classes indeed have much stronger influence. As the numerical values of FID and FIDCLIP are not comparable, the left and right y axes have been normalized such that the relative changes represented accurately.
matching. Then, by sweeping over N, we can gain further insights to how important the top classi- fication results are for FID.
We implement this by carrying out the weight optimization in the space of class probabilities (see Figure 1). We furthermore binarize the class probability vectors by identifying the N classes with the highest probabilities and setting the corresponding entries to and the rest to O. These vectors now indicate, for each image, the Top-N classes, while discarding their estimated probabilities. By matching the statistics of these indicator vectors between real and generated distributions, we optimize the co-occurrence of Top-N classes. The result of the weight optimization is therefore an approximation of Top-N histogram matching. With N = 1, the results approximately align with simple histogram matching (Table 1). Note that the binarization done before the weight optimization begins, and thus we don't 't need its (non-computable) gradients.
Figure 6 shows how FID (computed in the usual pre-logit space) changes as we optimize Top-N histogram matching with increasing N. We observe that even with small values of N, FID improves rapidly, and converges to value slightly higher than was obtained by optimizing the weights in the pre-logit space (FID in Table 2). This demonstrates that FID is, significant degree, de- termined by the co-occurrence of top ImageNet classes. Furthermore, it illustrates that FID is the most interested in a handful of features whose only purpose is help with ImageNet classification, not on some careful analysis of the whole image. As a further validation of this tendency, we also computed a similar optimization using binarized vectors computed using N classes chosen from the middle of the sorted probabilities (orange curves in Figure 6). The results show that the top classes have a significantly higher influence on FID than those ranked lower by the Inception- V3 classifier. Finally, as before, we present control FIDCLIP (dashed curves) that shows that CLIP's feature space is almost indifferent to the apparent FID improvements yielded by the better alignment of Top-NV ImageNet classes. Though only present results for FFHQ here, qualitative behavior is similar for LSUN CAT/PLACES/CARR and AFHQ-v2 DOG (see Appendix D).
4 PRACTICAL EXAMPLE: IMAGENET PRE-TRAINED GANS
Our experiments indicate that it is certainly possible that some models receive unrealistically low FID simply because they happen to reproduce the (inconsequential) ImageNet class distribution detected in the training data. Perhaps the most obvious way this could happen in practice is when ImageNet pre-training is used for a GAN discriminator (Sauer et al., 2021; Kumari et al., 2022). This approach has been observed to lead to much faster convergence and significant improvements in FID, but since the discriminator is readily sensitive to the ImageNet classes, maybe also guides the gen- erator to replicate them? Perhaps part of the improvement is in the perceptual nullspace of FID?
*Our binarization works when S 50% of classes are set 1. After that the roles of 0 and flip and the FID curves repeat themselves symmetrically. That is why we used middle entries instead of the lowest entries.
8

---

Published as a conference paper at ICLR 2023
FID 5.28, Recall 0.45. Fr = 4.67
FID 5.30. Recall
0.46. FID = 2.76
(a) Projected FastGAN
(b) StyleGAN2
Figure 7: Uncurated samples from (a) Projected FastGAN and (b) StyleGAN2. Both models achieve similar FID even though the Projected FastGAN samples contain more artifacts. In contrast, Pro- jected FastGAN has significantly higher FIDCUP, consistent with the observed quality differential.
We study this in the same context as Sauer et al. (2021) by training Projected FastGAN (Liu al., 2021; Sauer et al., 2021) that uses an ImageNet pre-trained EfficientNet (Tan & Le, 2019) as feature extractor of the discriminator, and compare against StyleGAN2 in FFHQ.
The human preference study conducted by Sauer et al. (2021) concludes that despite the dramatic improvements in FID, Projected FastGAN tends to generate lower quality and less diverse FFHQ samples than StyleGAN2. We verify this by comparing samples from Projected FastGAN and Style- GAÑ2 in a setup where FIDs are roughly comparable and the models reproduce a similar degree of variation as measured by Recall (Kynkäänniemi et al., 2019). Visual inspection of uncurated sam- ples (Figure 7) indeed reveals that Projected FastGAN produces much more distortions in the human faces than StyleGAN2 (see Appendix E for larger image grids).
In this iso-FID comparison, FIDCLIP agrees with human assessment StyleGAN2 is rated signifi- cantly better than Projected FastGAN. It therefore seems clear that Projected FastGAN has lower FID than it should have, confirming that at least some of its apparent improvements are in the per- ceptual null space. We believe that the reason for this is the accidental leak of information from the pre-trained network, causing the model to replicate the ImageNet- like aspects in the training data more keenly. This observation does not mean that ImageNet pre-training mean that such pre- teeiiraining can make FID unreliable in practice.
bad idea, but does
We suspect similar interference can happen when the ImageNet pre-trained classifiers are used cu- rate the training data (De Vries et al., 2020) or as part of the sampling process (Watson et al., 2022).
CONCLUSIONS
The numerical values of FID have a number of important uses. Large values indicate training failures quite reliably, and FID appears highly dependable when monitoring the convergence of training run. FID improvements obtained through hyperparameter sweeps other trivial changes generally seem to translate to better (subjective) results, even when the distributions are well aligned.7 The caveats arise when two sufficiently different architectures and/or training setups are compared. If one of them is, for some reason, inclined to better reproduce the fringe features, it can lead to a much lower FIDs without a corresponding improvement in the human- hubaanobservable quality.
Particular care should be exercised when introducing ImageNet pre-training to generative models (Sauer al., 2021; 2022; Kumari al., 2022), as it may compromise the validity of FID quality metric. This effect is difficult to quantify because the current widespread metrics (KII and Precision/Recall) also rely on the feature spaces of ImageNet classifiers. As partial solution, the FID improvements should at least be verified using a non- Imagene trained Fréchet distance. Viable alternative feature spaces include CLIP (Radford et al., 2021: Sauer et al., 2021), self-supervised SwAV (Caron et al., 2020; Morozov et al., 2021), and an uninitialized network (Naeem et al., 2020; Sauer et al., 2022). We hope that our methods help examine the properties of these feature spaces in future work.
"We use ht https: (iggtuu........ com/ aut onomousvi ss on/ /oo/pro///a ptppprojeee projecctó_gaaa with default parameters. Bassed on personal communication with individuals who have trained over 10,000 generative models.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Samuli Laine for helpful comments. This work was partially supported by the European Research Council (ERC Consolidator Grant 866435), and made use of computational resources provided by the Aalto Science-IT project and the Finnish IT Center for Science (CSC).
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-Scale Machine Learning Heterogeneous Systems, 2015. URL https: //www wwnsorflow..ooooo.ooooo www.e//wwwooo.ooo.oooooooooooooo
Motasem Alfarra, Juan Pérez, Anna Frühstücck, Philip H. S. Torr, Peter Wonka, and Bernard Ghanem. On the Robustness of Quality Measures for GANs. In Proc. ECCV, 2022.
Sefi Bell-Kligler, Assaf Shocher, and Michal Trani. Blind Super-Rssolution Kernel Estimation using an Internal-GAN. In Proc. NeurlPS, 2019.
Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, and A. Gretton. Demystifying MMD GANs. In Proc. ICIR, 2018.
Ali Borji. Pros and cons of GAN evaluation measures: New developments. Comput. Vis. Image Underst., 2022.
Andrew Brock, Jeff Donahue, and K. Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In Proc. ICLR, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In Proc. NeurIPS, 2020.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Woo Ha. StarGAN v2: Diverse Image Synthesis for Multiple Domains. In Proc. CVPR, 2020.
Min Jin Chong and D. Forsyth. Effectively Unbiased FID and Inception Score and Where to Find Them. Proc. CVPR, 2020.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. Proc. CVPR, 2009.
Terrance DeVries, Michal Drozdzal, and Graham Taylor. Instance Selection for GANs. In Pruc. NeurIPS, 2020.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density Estimation using Real NVP. In Proc. ICLR, 2017.
D. Dowson and B. Landau. The Fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis, 12:450-455, 1982.
Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming Transformers for High-Resolution Image Synthesis. In Proc. CVPR, 2021.
Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel Measures of Con- ditional Dependence. In Proc. NIPS, 2007.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In Proc. ICLR, 2019.
10

---

Published as a conference paper at ICLR 2023
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. In Proc. NIPS, 2014.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research, 2012.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recogni- tion. In Proc. CVPR, 2016.
Katherine Hermann, Ting Chen, and Simon Kornblith. The Origins and Prevalence of Texture Bias in Convolutional Neural Networks. In Proc. NeurIPS, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Proc. NIPS, 2017.
Jonathan Ho, Ajay Jain, and P. Abbeel. Denoising Diffusion Probabilistic Models. In Proc. NeurIPS, 2020.
Xun Huang, Arun Mallya, Ting-Chun Wang, and Ming- Yu Liu. Multimodal Conditional Image Synthesis with Product-of-Expeet Experts GANs. In Proc. ECCV, 2022.
Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining. In Proc. ICLR, 2022.
Thibaut Issenhuth, Ugo Tanielian, David Picard, and Jeremie Mary. Latent reweighting, an almost free improvement for GANs. Proc. WACV, 2022.
Tero Karras, Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adver- sarial Networks. In Proc. CVPR, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training Generative Adversarial Networks with Limited Data. In Proc. NeurlPS, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and Improving the Image Quality of StyleGAN. In Proc. CVPR, 2020b.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-Free Generative Adversarial Networks. Proc. NeurIPS, 2021.
Junho Kim, Minjae Kim, Hyeonwoo Kang, and KwangHee Lee. U-GAT-IT: Unsupervised Gen- erative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation. In Proc. ICLR, 2020.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible Ixl Convolu- tions. In Proc. NeurIPS, 2018
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. In Proc. ICLR, 2014.
Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Ensembling Off-the-shelf Models for GAN Training. In Proc. CVPR, 2022.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved Precision and Recall Metric for Assessing Generative Models. In Proc. NewrIPS, 2019.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew P. Aitken, Alykhan Tejani, J. Totz, Zehan Wang. and W. Shi. Pealo-Realistic Single Image Super- Resolution Using Gen- erative Adversarial Network. In Proc. CVPR, 2017.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis. Proc. ICLR, 2021.
Mario Lucic, Karol Kurach, Marcin Michalski, S. Gelly, and O. Bousquet. Are GANs Created Equal? A Large-Scale Study. In Proc. NeurIPS, 2018.

---

Published as a conference paper at ICLR 2023
Stanislav Morozov, Andrey Voynov, and Artem Babenko. On Stuervisee Image Representa- tions for GAN Evaluation. In Proc. ICLR, 2021.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh Yunjey Choi, and Jaejun Yoo. Reliable Fidelity and Diversity Metrics for Generative Models. In Proc. ICML, 2020.
Charlie Nash, Jacob Menick. S. Dieleman, and P. Battaglia. Generating Images with Sparse Repre- sentations. In Proc. ICML, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In Proc. JCML, 2022.
Taesung Park, Ming- Yu Liu, T. Wang, and Jun-Yan Zhu. Semantic Image Synthesis With Spatially- Adaptive Normalization. In Proc. CVPR, 2019.
Taesung Park. Jun- Yan Zhu, O. Wang, Jingwan Lu, E. Shechtman, Alexei A. Efros, and Richard Zhang. Swapping Autoencoder for Deep Image Manipulation. In Proc. NeurlPS, 2020.
Gaurav Parmar, Richard Zhang, and Jun-Yan Yan Zhu. On Aliased Resizing and Surprising Subtleties in GAN Evaluation. In Proc. CVPR, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, Pigg-PPerfomance Deep Learning Library. In Proc. NeurlPS, 2019.
Philippe Pierre Pebay. Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments. Technical Report SAND2008-66 6212, Sandia National Lab- oratories, 2008.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proc. ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-Image Generation. In Proc. ICML, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. CoRR, abs/2204.06125, 2022.
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating Diverse High-Fidelity Images with VQB2- In Proc. NeurIPS, 2019.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sali- mans, Jonathan Ho. David J Fleet, and Mohammad Norouzi. Photorealistic Text-to-Image Diffu- sion Models with Deep Language Understanding. CoRR, abs/2205. 1487, 2022.
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucie, Olivier Bousquet, and Sylvain Gelly. Assessing Generative Models via Precision and Recall. In Proc. NewrIPS, 2018.
Tim Salimans, I. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Im- proved Techniques for Training GANs. In Proc. NIPS, 2016.
Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected GANs Converge Faster. In In Proc. NeurlPS, 2021.
Axel Sauer, Katja Schwarz, and Andreas Geiger. StyleGAN-XL: Scaling StyleGAN to Large Di- verse Datasets. In Proc. TOG, 2022.
12

---

Published as a conference paper at ICLR 2023
Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient- -assedd Localization. In Proc. ICCV, 2017.
Jascha Sohl-Dickstein, Bric A. Weiss, Niru Maheswaranathan, and S. Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In Proc. ICML, 2015.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribu- tion. In Proc. NeurIPS, 2019.
Christian Szegedy, V. Vanhoucke, Ioffe, Jonathon Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. In Proc. CVPR, 2016.
Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Proc. ICML, 2019.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel Recurrent Neural Networks. In Proc. ICML, 2016a.
Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional Image Generation with PixeICNN Decoders. In Proc. NIPS, 2016b.
Sjoerd van Steenkiste, Karol Kurach, Jürgen Schmidhuber, and Sylvain Gelly. Investigating object compositionality in Generative Adversarial Networks. Neural Networks, 2020.
Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality. Proc. ICLR, 2022.
Qiantong Xu, Gao Huang, Yang Yuan, Chuan Guo, Yu Sun, Felix Wu, and Kilian Q. WWei-- berger. An Empirical Study Evaluation Metries of Generative Adversarial Networks. ArXiv, abs/1806.07755, 2018
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop. CoRR, abs/ 550.003353,, 2015.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as Perceptual Metric. Proc. CVPR, 2018.
Jun-Yan Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired Image-to-Image Translation using Cycle-Consisie Adversarial Networks. In Proc. ICCV, 2017.
13

---

Published as a conference paper at ICLR 2023
Sample size
FFHQ
LSUN Cat
FFHQ
LSUN Cat
5k + 5k
11.65 = 0.04
17.38 0.13
TensorFlow Py PyTorhh
5.30 0.04
8.25 = 0.04
10k + 10k
8.31 = 0.06
12.42 1: 0.06
3.69 0.05
6.64 = 0.03
50k + 50k
5.30 = 0.04
8.25 + 0.04
All 50k
5.14 0.04
7.83 0.03
(a) Number of samples
(b) Inception V3 instance
Figure 8: FID is very sensitive to (a) the number samples (real + generated) and (b) the exact instance of the Inception- V3 network. The tables report the mean + standard deviation of FID for given StyleGAN2 generator over ten evaluations with different random seeds.
A NUMERICAL SENSITIVITY OF FID
Previous work has uncovered various details that have surprisingly large effect on the exact value of FID (Binkowski et al., 2018; Lucic et al., 2018; Parmar et al., 2022; Chong & Forsyth, 2020). These observations important to acknowledge because reproducing results from comparison methods not always possible and one might be forced to resort to copying the reported FID results. In that case, small differences in the FID evaluation protocol might cause erroneous rankings between models.
Number of samples and bias. FID depends strongly the number of samples used in evalua- tion (Figure 8a). Therefore is crucial to standardize to a specific number of real and generated samples (Binkowski al., 2018).
Network architecture. FID is also very sensitive to the chosen feature network instance or type. Many deep learning frameworks (e.g. PyTorch (Paszke al., 2019), Tensorflow (Abadi al., 2015)) provide their own versions of the Inception-Ve V3 network with distinct weights. Figure 8b shows that FID surprisingly sensitive the exact instance of the Inception-V3 network. The discrepancies are certainly large enough to confuse with state-of-the-art performance. In practice, the official Tensorflow network must be used for comparable results.
Lucic et al. (2018) reported that the ranking of models with FID is not sensitive to the selected network architecture it only has an effect on the absolute value range of FIDs, but the relative ordering of the models remains approximately the same. However, our tests with FIDeLI indicate that this observation likely holds only between different ImageNet classifier networks.
Image processing flaws. Before feeding images to the Inception V3, they need to be resized to 299 299 resolution. Parmar et al. (2022) noted that the image resize functions the most com- monly used deep learning libraries (Paszke et al., 2019; Abadi et al., 2015) introduce aliasing arti- facts due to poor pre- filtffiteiin.. They demonstrate that this aliasing has noticeable effect on FID.
"nttp://dowu /dwwml oad.oat.eee.eoo tensorr www.ww/oooooooo...eo /ode haaaa///mmmannnt///gggeeeo 1--2-05.t99
14

---

Published as a conference paper at ICLR 2023
500
FFHQ, Pearson's p= 1.00
LSUN Cat, Pearson's p 1.00
200
LSUN Places, Pearson's p 0.99
Logits FID
100
50
25
10g
25
100
FID
Figure 9: We observe nearly perfect correlation between FIDs computed from pre-logit features and classification logits since these two are separated by only one affine transformation, Each point corresponds to single StyleGAN2 training snapshot in 256 x 256 resolution.
All regions
35
Important regions
30
Unimportant regions
Sensitivity heatmap
Important regions
20
15
59
8.0
0.025
0,075
All regions
Unimp, regions
Noise scale
(a) Sensitivity to noise
(b) Example images
Figure 10: (a) Adding Gaussian noise to regions that important for FID (blue curve) leads the larger increase compared to adding noise in unimportant regions. (b) Image showing regions FID considers as the most important and noise added different regions at scale 0.05. We recommend zooming in (b) to better assess the noise.
B CORRELATION BETWEEN PRE-LOGITS AND LOGITS FID
Figure demonstrates that FIDs calculated from the pre-logits and logits are highly correlated. The high correlation is explained by the fact that these two spaces are separated by only one affine transformation and without any non-linearities. Note that this test is only a guiding experiment to help find out to what FID is sensitive and it is not guaranteed hold for different GAN architectures or training setups.
C WHAT DOES FID LOOK AT IN AN IMAGE?
Validation of FID sensitivity heatmaps. To validate how reliably sensitivity heatmaps highlight the most important regions for FID, we perform an additional experiment where we add Gaussian noise either important unimportant areas, while keeping the other clean without noise. To
15

---

Published as a conference paper at ICLR 2023
cancel out effects that may arise from adding different amounts of noise, measured in pixel area, divide the pixels of the images equally between the important and unimportant regions.
Figure 10a shows FID when add an increasing amount of Gaussian noise different regions of the images and Figure 10b demonstrates the appearance of the noisy images. Adding noise everywhere in the image is upper bound how greatly FID increase in this test setup. Adding noise to the important regions leads to larger increase in FID, compared to adding noise to the unimportant regions.
Additional FID sensitivity heatmaps. Figure 11 presents more FID sensitivity heatmaps for in- dividual StyleGAN2 generated images using FFHQ and LSUN CAT and their corresponding Ima- geNet Top-1 classifications in the top left corner. For both datasets the regions for which FID is the most sensitive are highly localized and correlate strongly with the Top-1 class.
Figure 12 shows additional mean images and heatmaps for StyleGAN2 generated images for FFHQ that get classified a certain class. On average FID is the most sensitive the pixel locations where the Top-1 1 class is intuitively located and relatively insensitive to the human faces.
Comparison Grad-CAM. Figures 13 and 14 compare our FID sensitivity heatmaps standard Grad-CAM heatmaps (Selvaraju et al., 2017) for FFHQ and LSUN CAT, respectively. Grad-CAM heatmaps, computed using classification probabilities, highlight similar regions in the images as our FID sensitivity heatmaps, showing that the important regions for ImageNet classification overlap heavily with regions that are important for FID. Additionally, Figure 15 shows mean images and FID heatmaps, well mean Top-1 Grad-CAM heatmaps for StyleGAN2 generated FFHQ images.
16

---

Published as a conference paper at ICLR 2023
Blow lue
Bow lie
Sunglan
Stethoscpp
Bow
cab cout
Mertizborrd
Milltay madooia
Pipsheed
Bib
lee
Puntae
Stenbosoope
Bakung
Miepoponn
Stole
Bow Ne
Cowbby bat
Pondão
Gryom
Wig
Sumsereen
Scal Teli
Basson
Boebarsee
Sunstten
Fealher bea
Groecn
FFHQ
Tiger
Fnglish springer
Fuse
Reymlian
Tablly
Tabing
Fgyntian
Quil
Ratyytiom
Fayptien
Tably
Tiger cal
Tabby
Tubly
Stames cal
Hen-er Woedwood
Amgona
Ciib
Siomese
Stumuekk
Tiger bat
Silky fonnirr
Egyysian cal
Persion eat
Persion cat
Tabby
Pesiat
Fgyptian
Tobby
Ghant Schnarrer
Warthhaaah
Egyptian
Bucket
(b) LSUN CAT
Figure 11: Heatmaps of the most important regions for FID for StyleGAN2 images in (a) FFHQ and (b) LSUN CAT, along with their Top-1 classification annotated in the top left corner of each image.
17

---

Published as a conference paper at ICLR 2023
Wig
Microphone
Lipstick
Cowboy hat
Lab coat
Mean image
Mean heatmap
Cloak
Bib
Greenhouse
Hair slide
Shower curtain
Mean image
Mean heatmap
Figure 12: Mean images and heatmaps of regions that are the most important for FID with Style- GAN2 images in FFHQ that get classified to some class, e.g., "lipstick". The heatmaps highlight the regions of Top-1 classes that are typically located outside the face area.
18

---

Published as a conference paper at ICLR 2023
sensirivity
Top-1 Grad-CAM
Top-2 Grad-CAM
Top-3 Grad-CAM
Average
Microphos
Bow
Marturborrd
Acalemic
Abays
Oliv
Bow
Stethesooge
Lab ceat
Minibu
New de
Croom
Kinseo
Figure 13: Comparison of our FID sensitivity heatmaps with standard Grad-CAM in FFHQ The Grad-CAM heatmaps highlight the most important areas for Top-1, 1, Top-2, and Top-3 classification.
We also show an average Grad-CAn heatmap weighted according the classification probabilities.
19

---

Published as a conference paper at ICLR 2023
sensitivity
Top-1 Grad-CAM
Top-2
Top-3 Grad-CAM
Average Grad-CAM
Tably
Tiper
Egypitanccm
Bathrub
Lynx
Tub
Tabby
Expption
Tabby
Portian cat
Figure 14: Comparison of FID sensitivity heatmaps with standard Grad-CAM in LSUN CAT. The Grad-CAM heatmaps highlight the most important areas for Top-1, Top-2, and Top-3 classi- fication. We also show an average Grad-CAM heatmap weighted according to the classification probabilities.
All classes
Bow
Seat bel:
Morarboard
Wig
Miczophone
Lipstick
Cowboy hat
Moan image
senstovity
Mean top-1 ORa--CM
Figure 15: Top: Average of StyleGAN2-gennen FFHQ images whose Top-1 classification matches the given class, e.g., "bow tie". Middle: Average heatmaps of regions that are the most important for FID. Bottom: Corresponding average Grad-CAM heatmaps computed for the Top-1 class.
20

---

Published as a conference paper at ICLR 2023
D PROBING THE PERCEPTUAL NULL SPACE IN FID
Pseudocode and implementation details. Algorithm shows the pseudocode for our resampling method. Function TETTMIIZ-RESAMMLING--E
optimizes the per-image sampling weights such that FID between real and weighted generated features is minimized. The imputs to the func- tion are sets of features for real and generated images F, and Fg. respectively, learning rate a and maximum number of iterations T. Note that the features do not have to be the typical pre-logits features where standard FID is calculated; they can be, e.g., logits or binarized class probabilities. First, we calculate the statisties of real features (lines 3-4) and then initialize the per-image log- parameterized weights w; to zeros (line 7). Then, for T iterations we calculate the weighted mean and covariance of generated features (lines 11-12) and update the weights via gradient descent to minimize FID (line 15). After optimization the log- parameterized weights can be transformed into sampling probabilities with
e where p; is the probability of sampling ith feature. We sample with replacement according to these probabilities to calculate our resampled FIDs (FID"L, FID", FID with 50k real and generated features.
In practice, we use features of 50k real and 250k generated images for all datasets, except for AFHQ-v2 DOG where we use 4678 real and 25k generated images. We use learning rate a = 10.0 when optimizing pre-logits features and a = 5.0 when optimizing logits or binarized class probabilities. We optimize the weights until convergence, which typically requires ~ 100k iterations. We select the weights that lead to the smallest FID with 50k real and 50k generated features that are sampled according the optimized weights. In the optimization, we do not ap- ply exponential moving average to the weights or learning rate decay. We use 32GB NVIDIA Tesla V100 GPU to run our resampling experiments. One weight optimization run with 50k real and 250k generated features takes approximately 48h where most the execution time goes into calculating the matrix square root in FID with eigenvalue decomposition. Code is available at https: / thub.
Image grids for Top-1 matching and pre-logits resampling. Figure 16 shows uncurated image grids when we sample StyleGAN2 generated images randomly, after Top- histogram matching, and after matching all fringe features. Even though FID drops very significantly, the visual appearance of the generated images remains largely unchanged. FIDCLIP also fails to confirm the improvement indicated by FID.
In Figure 17, we show larger set of images that obtain small or large weight after optimizing FID in the pre-logits feature space. low FID after resampling cannot be attributed to simply removing images with clear visual artifacts.
Effect of pre-logits resampling on KID, Table 3 shows that resampling in the pre-logits feature space also strongly decreases Kernel Inception Distance (KID) (Binkowski et al., 2018), and a Ker- nel Inception Distance that is calculated using the radial basis function (RBF) kernel (RBF-KID). While the standard KII) compares the first three moments (Binkowski et al., 2018), RBF-KID con- siders all moments, because the RBF kernel is a characteristic kernel (Gretton et al., 2012; Fukumizu al., 2007). The metrics are computed in the same feature space as FID and therefore we hypothe- size that they share approximately the same perceptual null space. To calculate RBF-KID, we used RBF scatter parameter y = d' where d 2048 the dimensionality of Inception V3 pre-logits. We experimented with different scatter parameter values (7 E { 8d 4dl 2d that they all lead to similar qualitative behavior.
2 8 } and observed d d d
Top-N histogram matching. We show further results from approximate Top-N histogram match- ing in LSUN CAT/CAR/PLACES and AFHQ-v2 DoG in Figure 18. FID can be consistently im- proved by aligning the Top-N histograms of real and generated images. Furthermore, the largest decrease in FID can be obtained by including information of the most probable classes.
E PRACTICAL EXAMPLE: IMAGENET PRE-TRAINED GANS
Figure 19 shows larger image grids for StyleGAN2 and Projected FastGAN in FFHQ
21

---

Published as a conference paper at ICLR 2023
Algorithm Resampling algorithm pseudocode.
1: function
RESAMPLING-
Fg, a, T)
2: Calculate feature statistics of reals.
3: H4, + F1 Σ. f;
4:
E F Ef
(fi 14)
5:
6:
Initialize log- a---parameterized per-image weights w to zeros.
7:
w; Vi
8:
9:
for T iterations do
10:
Compute weighted mean and covariance of generated features.
(2(ww) -- E. e. giie
11:
Eg(w) + E,e E e (1gw) (fz 1gg (w))
13:
14:
Update the weights.
15:
w t- w - FID (mp Er, My (w), Eg(w))
17:
return
Table 3: Optimizing FID also decreases KID and RBF-KID significantly. We compare the KIDs of randomly sampled images (KID, RBF-KID) against KIDs computed by resampling according the weights obtained from optimizing FID the pre-logits features (KID Pi. RBF-KID The numbers represent averages over ten evaluations.
Dataset
PI
KIDx10"
KID x10!
RBF-KID
RBF-KID
FFIIQ
5.30
1.78
(-66.4%)
1.52
0.19 (-67
0.67
0.08
(-88.1%)
LSUN CAT
8.25
3.05
(-63.0%)
3.00
0.37 (-87.7%)
1.32
0.16
(-87.9%)
LSUN CAR
5.65
2.11
(-62.1%)
2.49
0.32 (-87.19)
1.19
0.16
(-86.8%)
LSUN PLACES
12.96
3.59
(-7233%
7.43
0.31 (-95
3.04
0.14
(-96.4%)
AFHQ-V2 DOG
10.25
6.92
(-42.29%
2.05
0.12
(-94.19))
0.06
(-94.2%)
22

---

Published as a conference paper at ICLR 2023
(a) Random sample (FID 5.30, Recall 0.46. FIDau 2.76)
(b) Top-1 matching (FID 4.70, Recall = 0.45, FIDCLIP = 2.74)
(c) Pre-logits resampling (FID 1.78, Recall 0.40, FIDCLP 2.64)
Figure 16: FID can be drastically reduced by using our resampling approach without improving the visual fidelity of the generated images in any obvious way. (a) Randomly sampled StyleGAN2 images (b) Randomly sampled StyleGAN2 images after Top-1 histogram matching (c) StyleGAN2 images sampled according to the weights obtained by matching all fringe features.
23

---

Published as a conference paper at ICLR 2023
a Images with small weights
(b) Images with large weights
Figure 17: Random StyleGAN2 images sampled among (a) the smallest 10% of weights and (b) the largest 10% of weights. Both sets contain realistic looking images and images with visual artifacts.
24

---

Published as a conference paper at ICLR 2023
150
FIDeLIP
40
170
M.Bdh duace
100
Number
38
Number
(a) LSUN CAT
(b) AFHQ-v2 DOG
12:
F1D
RDDrup
carciag Took
casling
mudbin Topil
maaciin daso
100
200 Number of classes (N)
s00
Numbar classes (N)
(c) LSUN PLACES
(d) LSUN CAR
Figure 18: Additional results from aligning Top-N class histograms. For all datasets adding infor- mation from the Top-N ImageNet classes consistently leads to the largest decrease in FID while FID remains almost unchanged.
25

---

Published as a conference paper at ICLR 2023
(a) Projected FastGAN (FID 5.28, Recall 0.45, FIDCLIP 4.67)
(b) StyleGAN2 (FID = 5.30, Recall 0.46, FID = 2.76)
Figure 19: Uncurated samples of (a) Projected FastGAN and (b) StyleGAN2 generated images.
While Projected FastGAN achieves better FID, the samples contain more distortions and artifacts.
26


Reviews:
[{"review_id": "r0gXmWB8u_", "reviewer": "Reviewer_iTAh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducts a deep, detailed study using the FID score for evaluating the quality of generative models. This paper reveals that imagenet pretrained model feature-based metric can be biased, leading to low generation quality with high FID score if the generator aligns the histograms of Top-N classifications be- tween sets of generated and real images. Numerical results and visualizations support the claim. ", "summary_of_the_review": "Good paper, I recommend for acceptance.", "strength_and_weaknesses": "Strength:\n1. This paper provides an insightful study of the popular FID score used in GAN, and shows that using ImageNet pre-trained features can be biased. This is new knowledge.\n\n2. Paper is clearly written. The reader can quickly understand the major lesson.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "hZgInegK3YP", "reviewer": "Reviewer_Xd21", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work uses visualization techniques to investigate the area of concern of FID in the generated images. Through the study and analysis, the authors have an interesting finding that the feature space computed by FID usually is very close to ImageNet classification. The paper also gives apractical example of an accidental distortion and provides suggestions for viable alternative feature spaces.", "summary_of_the_review": "This paper explores possible reasons why FID sometimes deviates from human judgment and provides guidance for the subsequent use of FID.", "strength_and_weaknesses": "This paper is well organized, with reasonable structure and clear logic. My concerns are as follows.\n\n1. In page 6 (bottom), the paper mention the first step adopt ''5×oversampling''. Why you choose ''5×oversampling''? How does this parameter affect the match process?\n2. In Table 1, FID_{SwAV} and FID_{CLIP} have ∼ 1% decrease after matching. I'm very curious about the reasons for this decline.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "gYcSj7uZvz", "reviewer": "Reviewer_9T7x", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors carry out an extensive empirical assessment of a popular evaluation metric for generative models of natural images: the Fréchet Inception Distance. In particular, experiments are designed to indicate properties of the distance when used on data that diverges somehow from ImageNet-1k, used to pre-train the Inception architecture used as an embedding mapping to compress images prior to computing the metric. Results indicate that, under such distribution shifts, the Fréchet Inception Distance can be misleading as it is affected by features that are not as relevant in the downstream data as they were when supervisedly training the Inception encoder. Importantly, results revealed that generated samples tend to reduce the metric when they match the top classes appearing in the subset of ImageNet used to compute the metric, which can be exploited using simple resampling procedures, as demonstrated in the paper.", "summary_of_the_review": "In summary, I would describe the paper as a solid empirical assessment of an important problem, but lacking the proposal of solutions, which, in my opinion, limit too much the scope of the paper. I would be more than happy to bump up my scores if the authors are able to introduce results that are more indicative of improving directions and leave less for future work.", "strength_and_weaknesses": "Pros:\n\n+Authors focused on a highly relevant problem: evaluating evaluation metrics themselves.\n\n+A somewhat large-scale empirical evaluation is carried out to reveal issues in the so commonly used Fréchet Inception Distance: It is noisy/high variance when used in data that somehow differs from ImageNet-1k.\n\n+More interestingly, authors show evidence for what they called the perceptual null space. I.e, FID is exploitable and one can modify it to a great extent without imposing perceptual improvement in generated samples.\n\nCons:\n\n-My main concern with the paper is the size of its scope, which is a subjective concern rather than a technical one admittedly. While the work defines clear hypothesis involving the properties of FID and provides clear evidence supporting those, it lacks in providing solutions or the means to address the observed issues. Authors did observe that using self-supervised backbones, for instance, would alleviate the issue, but that isn't given enough focus and it is not clear whether we should move towards this kind of solution.\n\n-Another important concern which is inline with the above is the scope of the empirical assessment. While it clearly supported the main claims and gives a clear demonstration of the issues of FID, it's limited to that. Some lacking components I would mention would be:\n\n1-Studying the effect of using backbones trained with approaches other than supervision on ImageNet-1K. I would expect that simply moving from training the encoder on ImageNet-1K to a dataset with a larger label set, e.g., ImageNet-21K, would alleviate observed issues quite significantly, and render \"attacks\" much less efficient to execute. A similar effect could be obtained with training paradigms other than supervision, as indicated by some of the reported results. So, it would be very informative to repeat some of the experiments with a self-supervised backbone trained on a highly diverse dataset (i.e., with a large label set).\n\n2-Verifying whether features relevant for FID indeed correspond those relevant for supervision. I would suggest the authors somehow compare the FID heatmaps they proposed with standard GradCAM. Do those match?\n\n3-Studying whether the same behaviour in FID is observed with KID under varying kernels. FID only accounts for the first two moments while KID, under the right kernel, should account for high order moments as well. Would accounting for those extra moments help overcome the issue with FID or would they make it for an even more vulnerable metric? The results in Appendix D indicate the latter, but only an analysis under varying kernels would be conclusive.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VuFm_Q_poyx", "reviewer": "Reviewer_ncn9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a careful evaluation of on of the widely used metrics in the generative models literature; Fréchet Inception Distance (FID). The experiments demonstrates and equivalence between the feature of Inception V3 module space in which FID is computed and the logits. That is, FID is mainly sensitive towards features that correlates with categories in the ImageNet dataset. Further, several experiments were carried out to match the Top-1 and Top-N output distribution of generated images with the ImageNet one. The results show that one can significantly improve FID without actual improvement in the image quality. At last, experiments in Figure 7 show promising results of computing FID in the CLIP embedding space that might be more generic than ImageNet pre training,.", "summary_of_the_review": "This paper has several merits including the motivation and experimental setup. However, I have two main concerns regarding the novelty and the a fix to the spotted issue in FID.\n", "strength_and_weaknesses": "Strengths: \n\n1- The problem tackled in this paper is important: Metrics used to assess the quality of generative models need to be reliable.\nThe results of this paper suggests that one could improve such metrics without actual improvements in the perceptual quality of the generative model.\n\n2- Constructing an adversarial attack to FID through matching the Top-1/Top-N histograms of the output space between the generated distribution of images and ImageNet is both novel and insightful. \n\n3- The paper well motivates the experimental setup through showing the sensitivity of FID to each pixel in a given image. Experiments in Figure 3 show that FID could pay more attention to non-semantic parts of the generated images.\n\n\nWeaknesses: \n\n1- While this paper proposes very interesting analysis of FID, most of the conclusions of this paper are not surprising. For example, [A] discussed the potential of constructing adversarial attacks to the inception score. Moreover, [B] showed that one could generate images of noise that has good FID scores. Further, they show that  FID favors distribution of images with more artifacts by adjusting the truncation level in StyleGAN2.\n\nIt would be ideal to distinguish the new insights that this paper brings over the previous analysis of GAN metrics.\n\n2- Despite that this paper is mainly analyzing FID in the lens of ImageNet pre training, I believe that proposing a step towards fixing the spotted issue in FID is necessary.\nThe results of FID_{CLIP} presented towards the end of the paper are promising.\nCan you conduct some of the following experiments to validate the usefulness of such metric:\n\n2.a- Compute FID_{CLIP} for several GANs in the literature such as StyleGANV1, StyleGANV2, StyleGANV2, Big GAN. Does FID_{CLIP} order these GANs in the same way their perceptual quality order?\n\n2.b- Does FID_{CLIP} separate , for instance, a distribution of images from a distorted one (e.g. blurring and Gaussian noise).\n\n2c- Is FID_{CLIP} better due to a better architecture, Better pre training, or both? For instance, if we assume we have a subset of the training set of CLIP available, can we fool FID_{CLIP} with a similar approach to the one presented in this paper?\n\nGenerally, I appreciate the efforts put in this work. I see several merits despite the aforementioned weaknesses which I hope to be addressed during the discussion.\n\n[A]: “Improved Techniques for Training GANs”, 2016\n\n[B]: “On the robustness of Quality Measures for GANs”, 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%