Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work uses visualization techniques to investigate the area of concern of FID in the generated images. Through the study and analysis, the authors have an interesting finding that the feature space computed by FID usually is very close to ImageNet classification. The paper also gives apractical example of an accidental distortion and provides suggestions for viable alternative feature spaces.
strength_and_weaknesses: This paper is well organized, with reasonable structure and clear logic. My concerns are as follows.

1. In page 6 (bottom), the paper mention the first step adopt ''5×oversampling''. Why you choose ''5×oversampling''? How does this parameter affect the match process?
2. In Table 1, FID_{SwAV} and FID_{CLIP} have ∼ 1% decrease after matching. I'm very curious about the reasons for this decline.


clarity,_quality,_novelty_and_reproducibility: This paper is very valuable and overall clearly written. The finding and discussion about the important metric FID are very meaningful. The visualizations are convincing and insightful. 
summary_of_the_review: This paper explores possible reasons why FID sometimes deviates from human judgment and provides guidance for the subsequent use of FID.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a careful evaluation of on of the widely used metrics in the generative models literature; Fréchet Inception Distance (FID). The experiments demonstrates and equivalence between the feature of Inception V3 module space in which FID is computed and the logits. That is, FID is mainly sensitive towards features that correlates with categories in the ImageNet dataset. Further, several experiments were carried out to match the Top-1 and Top-N output distribution of generated images with the ImageNet one. The results show that one can significantly improve FID without actual improvement in the image quality. At last, experiments in Figure 7 show promising results of computing FID in the CLIP embedding space that might be more generic than ImageNet pre training,.
strength_and_weaknesses: Strengths: 

1- The problem tackled in this paper is important: Metrics used to assess the quality of generative models need to be reliable.
The results of this paper suggests that one could improve such metrics without actual improvements in the perceptual quality of the generative model.

2- Constructing an adversarial attack to FID through matching the Top-1/Top-N histograms of the output space between the generated distribution of images and ImageNet is both novel and insightful. 

3- The paper well motivates the experimental setup through showing the sensitivity of FID to each pixel in a given image. Experiments in Figure 3 show that FID could pay more attention to non-semantic parts of the generated images.


Weaknesses: 

1- While this paper proposes very interesting analysis of FID, most of the conclusions of this paper are not surprising. For example, [A] discussed the potential of constructing adversarial attacks to the inception score. Moreover, [B] showed that one could generate images of noise that has good FID scores. Further, they show that  FID favors distribution of images with more artifacts by adjusting the truncation level in StyleGAN2.

It would be ideal to distinguish the new insights that this paper brings over the previous analysis of GAN metrics.

2- Despite that this paper is mainly analyzing FID in the lens of ImageNet pre training, I believe that proposing a step towards fixing the spotted issue in FID is necessary.
The results of FID_{CLIP} presented towards the end of the paper are promising.
Can you conduct some of the following experiments to validate the usefulness of such metric:

2.a- Compute FID_{CLIP} for several GANs in the literature such as StyleGANV1, StyleGANV2, StyleGANV2, Big GAN. Does FID_{CLIP} order these GANs in the same way their perceptual quality order?

2.b- Does FID_{CLIP} separate , for instance, a distribution of images from a distorted one (e.g. blurring and Gaussian noise).

2c- Is FID_{CLIP} better due to a better architecture, Better pre training, or both? For instance, if we assume we have a subset of the training set of CLIP available, can we fool FID_{CLIP} with a similar approach to the one presented in this paper?

Generally, I appreciate the efforts put in this work. I see several merits despite the aforementioned weaknesses which I hope to be addressed during the discussion.

[A]: “Improved Techniques for Training GANs”, 2016

[B]: “On the robustness of Quality Measures for GANs”, 2022
clarity,_quality,_novelty_and_reproducibility: Generally, the paper has clear claims and a nice experimental setup supporting them. I have a concern regarding the novelty of the main contribution of the work conveyed in the first weakness. I hope the authors provide a clear response for that regard.
summary_of_the_review: This paper has several merits including the motivation and experimental setup. However, I have two main concerns regarding the novelty and the a fix to the spotted issue in FID.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors carry out an extensive empirical assessment of a popular evaluation metric for generative models of natural images: the Fréchet Inception Distance. In particular, experiments are designed to indicate properties of the distance when used on data that diverges somehow from ImageNet-1k, used to pre-train the Inception architecture used as an embedding mapping to compress images prior to computing the metric. Results indicate that, under such distribution shifts, the Fréchet Inception Distance can be misleading as it is affected by features that are not as relevant in the downstream data as they were when supervisedly training the Inception encoder. Importantly, results revealed that generated samples tend to reduce the metric when they match the top classes appearing in the subset of ImageNet used to compute the metric, which can be exploited using simple resampling procedures, as demonstrated in the paper.
strength_and_weaknesses: Pros:

+Authors focused on a highly relevant problem: evaluating evaluation metrics themselves.

+A somewhat large-scale empirical evaluation is carried out to reveal issues in the so commonly used Fréchet Inception Distance: It is noisy/high variance when used in data that somehow differs from ImageNet-1k.

+More interestingly, authors show evidence for what they called the perceptual null space. I.e, FID is exploitable and one can modify it to a great extent without imposing perceptual improvement in generated samples.

Cons:

-My main concern with the paper is the size of its scope, which is a subjective concern rather than a technical one admittedly. While the work defines clear hypothesis involving the properties of FID and provides clear evidence supporting those, it lacks in providing solutions or the means to address the observed issues. Authors did observe that using self-supervised backbones, for instance, would alleviate the issue, but that isn't given enough focus and it is not clear whether we should move towards this kind of solution.

-Another important concern which is inline with the above is the scope of the empirical assessment. While it clearly supported the main claims and gives a clear demonstration of the issues of FID, it's limited to that. Some lacking components I would mention would be:

1-Studying the effect of using backbones trained with approaches other than supervision on ImageNet-1K. I would expect that simply moving from training the encoder on ImageNet-1K to a dataset with a larger label set, e.g., ImageNet-21K, would alleviate observed issues quite significantly, and render "attacks" much less efficient to execute. A similar effect could be obtained with training paradigms other than supervision, as indicated by some of the reported results. So, it would be very informative to repeat some of the experiments with a self-supervised backbone trained on a highly diverse dataset (i.e., with a large label set).

2-Verifying whether features relevant for FID indeed correspond those relevant for supervision. I would suggest the authors somehow compare the FID heatmaps they proposed with standard GradCAM. Do those match?

3-Studying whether the same behaviour in FID is observed with KID under varying kernels. FID only accounts for the first two moments while KID, under the right kernel, should account for high order moments as well. Would accounting for those extra moments help overcome the issue with FID or would they make it for an even more vulnerable metric? The results in Appendix D indicate the latter, but only an analysis under varying kernels would be conclusive.



clarity,_quality,_novelty_and_reproducibility: Clarity: The manuscript is clearly written, pleasant to read, and easy to follow.

Quality: Hypothesis are clearly stated, the experimental design is clearly described, and results are discussed in the light of the posed hypothesis, which I think makes it for a high quality empirical work.

Novelty: The results are novel to my knowledge although to me it seems the conclusions pretty much align with the common sense in the community in that FID can be useful but not fully relied upon. Authors clearly showed that to be indeed the case.

Reproducibility: The experimental design is clearly discussed and it covers openly available data and models.
summary_of_the_review: In summary, I would describe the paper as a solid empirical assessment of an important problem, but lacking the proposal of solutions, which, in my opinion, limit too much the scope of the paper. I would be more than happy to bump up my scores if the authors are able to introduce results that are more indicative of improving directions and leave less for future work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper conducts a deep, detailed study using the FID score for evaluating the quality of generative models. This paper reveals that imagenet pretrained model feature-based metric can be biased, leading to low generation quality with high FID score if the generator aligns the histograms of Top-N classifications be- tween sets of generated and real images. Numerical results and visualizations support the claim. 
strength_and_weaknesses: Strength:
1. This paper provides an insightful study of the popular FID score used in GAN, and shows that using ImageNet pre-trained features can be biased. This is new knowledge.

2. Paper is clearly written. The reader can quickly understand the major lesson.

clarity,_quality,_novelty_and_reproducibility: This is a high-quality paper, novel, and can be reproduced.
summary_of_the_review: Good paper, I recommend for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors conduct an extensive study of the role of ImageNet classes on the resulting FID score. This work is important as FID is the most metric most widely used to evaluate generative models. Through careful analysis, the authors show that one can decrease FID by as much as 60% without improving sample quality.

The strength in the paper is extensive empirical evaluation. The one minor weakness, as pointed out by some reviewers, is the lack of a "fix" of the metric. Still, given the pervasiveness of FID in generative model evaluation, this work is important to the community. 
justification_for_why_not_higher_score: While this paper was a strong technical paper, what would push it over the edge for me is if there were clear fixes to the metric.
justification_for_why_not_lower_score: Given how important FID is to the generative modeling community, and given how extensive the empirical evaluation is, I cannot justify a lower score.
Decision: Accept: notable-top-25%