Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper generalizes the widely-used FID metric for image generation evaluation by fitting a mixture of Gaussians instead of a single Gaussian on the extracted features. The advantage of the proposed approach is it removes the unrealistic assumption of FID that the extracted features from some encoder networks (such as Inception-v3) are approximately Gaussian. The consequence of changing to GMM is that calculating the 2-wasserstein distance now requires a relaxation/approximation, which may result in approximation error and more computation. Empirically, the paper demonstrate that the proposed metric WaM may be less sensitive to noise perturbations.
main_review: Strengths: The paper has a simple yet interesting observation that the extracted features from widely-used encoder networks on widely-used image datasets are not actually distributed as Gaussian, which violates the assumption of FID. The paper then leverage recent advances on calculating 2-wasserstein distance between GMMs to obtain a generalization of FID. From this perspective, the proposed metric has the potential of addressing some limitations of FID and contributing to be a better evaluation metric for image generation.

That being said, I have a number of concerns/questions as discussed below:
- Because of the intractability of computing wasserstein distance between GMMs, the paper has to use an approximation by restricting the family of joint distribution to GMMs. In practice, how can we guarantee the quality of such approximation (which might be large)/quantify the suboptimality gap (since wasserstein between Gaussians are exact and the noise brought by approximation error may be too large wrt the signal in theory)? Moreover, as the paper discussed, this issue can be mitigated by using a large enough K (number of components). However, this may incur much more computation? For example, in equation 2, we need to compute K^2 wasserstein distance as well as a discrete optimal transport problem. When K=50 as used in the paper, we need to compute 2500x 2-wasserstein? Also a too large K will cause over-fitting, so we need to carefully tune this parameter, which is not good for a standard/widely-used evaluation metric since we need to make sure all researchers in this domain are using the same hyperparameter for fair comparisons while the optimal one is hard to know.
- At the beginning of sec 3.1, we know that preactivations of the convolution layers are Gaussian and Fig.1 indeed looks like a truncated gaussian? The truncation is mainly because the ReLU layer to force all features to be positive. Thus I may not agree with the argument in sec 3.2: "The first option is to use features before the average pooling layer and ReLU operation because these features may actually
be Gaussian. However, these features are extremely high dimensional (64  2048 = 131;072) and thus very hard to estimate accurately."
Can we only remove the ReLU operation and keep the pooling layer to reduce the dimension while the features without passing ReLU are still approximately Gaussian? It seems this option is not studied in the experiments.
- Roughly speaking, there is no new techniques proposed in this paper but a straightforward application of recent papers for calculating 2-wasserstein distance between GMMs. Thus I consider it an empirical study, where thorough and convincing experimental studies are needed. Then an important question is whether the evaluation protocols (R_FID, R_WaM, R) are good enough. From my perspective, I think there are issues with them: comparing metrics with different scales are hard and using these ratios may not make sense either. Suppose metric A range from [0, 100] and metric B range from [0, 10], also assume the sample quality linearly change within their ranges. Now if metric A change from 1 to 3 (hence R_A = 3) while metric B change from 5 to 10 (hence R_B=2), then we cannot say metric A is more sensitive than metric B because within A's range [1,3] corresponds to 2% sample quality change while [5,10] corresponds to 50% sample quality change. In short, although the authors found this issue during comparing them, I do not think the solution is good enough, which corresponds to the central empirical findings of this paper.
- The way that adversarial perturbations are constructed may lead to results that are not fair for FID. We may similarly construct adversarial examples to fool WaM by focusing the change more on third and higher moments without changing first two moments too much? Also neither adversarial constructions correspond to practical settings for evaluating deep generative models. So results are not convincing enough for demonstrating the practical value and the practical benefit is not clear. It seems to me that a better evaluation metric for generative model is one that correlates better with human perception? For example, we often observe that a set of samples look quite realistic but have a bad FID score and vice versa. So experiments about the alignment of WaM score and human perception may be needed to verify if this leads to a better metric. Robustness to some noisy perturbations may not be a central or even desirable property for a good evaluation metric, since detecting such perturbation may also be desirable in some scenatio.

---------------After rebuttal----------------------- Thank you for the response on my feedback. Some of my major concerns still remain (e.g. the justification of using ratios when comparing different metrics with different or unbounded scales, the fairness of the empirical evaluation still seems not convincing enough to me). Thus I tend to keep my original rating.
summary_of_the_review: This paper propose a simple yet interesting generalization to FID by leveraging recent advances on approximating wasserstein distance between GMMs. However, as a purely empirical paper, I think there are some major flaws/issues in the evaluation procedure and thus the empirical study is not convincing enough to demonstrate the pratical advantage brought by the proposed approach.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper targets to a new evaluation metric for the performance of generative models given a set of real images and a set of fake images. The authors show that Inception-v3 features of the imageNet dataset are not Gaussian and remedy this issue by modeling image features using Gaussian mixture models (GMMs) and formulating the distribution distance between two GMMs by MW2. They demonstrate their metric is less sensitive than FID against image perturbations.
main_review: Strengths:
+ Clear writing and demonstrations.
+ Meaningful research topic.

Weaknesses:
- The technical novelty is marginal. It is a plain combination between GMM  for distribution estimation and MW2 for distance formulation.
- The technical correctness is doubtful. In Figure 1 we see the feature distributions are non-negative, single-sided, and long-tail. If a single Gaussian is not suitable enough to estimate this distribution, a mixture of Gaussians would not be much better in theory. Instead, KID [1] fundamentally handles this issue: KID does not assume a parametric form for the distribution of ReLU activation; its cubic kernel fits the long-tail distribution better; it has an unbiased estimator. Please cite [1] and discuss why the proposed WaM would be more advantageous than KID.
- The technical details are unclear. How to calculate FID when perturbing a set of images? FID is calculated by two large sets of images, say, 50k images per set. Did the authors perturb the 50k images at the same time to increase FID? If so, how if some images are perturbed much more than the others? In this case, please show the image examples with the maximal amount of perturbations. Maybe the differences are visually perceptible.
- The experiments are not convincing.
  - For the claim that FID is more sensitive than the proposed WaM, in Section 5.1 images are perturbed by backpropagating through FID. There is no wonder FID looks more sensitive given this kind of perturbations. How about the other kind of perturbations: backpropagating through WaM?
  - For Figure 5, all the FID and WaM values are in the range of 100+. The metric sensitivity in that range is meaningless because our research community cares about small FID/WaM values. BigGAN already achieves <10 FID on ImageNet. We therefore need a metric to be sensitive enough in this range to distinguish cutting-edge generative model techniques. 
  - Figure 6 is meaningless neither because the results are the sensitivity on real images. Research community cares about the quality on generated images.
  - The baseline comparisons are insufficient. As mentioned above, please compare to KID [1], in terms of the sensitivity in the small value range, especially with the perturbations backpropagated through KID.

[1] Bińkowski, Mikołaj, et al. "Demystifying mmd gans." ICLR 2018.
summary_of_the_review: See the main review.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new metric (WaM) for the evaluation of images generated by a network, as an alternative to the commonly used FID metric. The authors show that this alternative metric is more robust to perturbations of the images (both adversarially chosen and random), when compared to FID.
main_review: This paper proposes an alternative metric (WaM) for the evaluation of generative networks, as an alternative to FID. The motivation behind this metric is the fact that the mathematical formulation for the FID metric requires the distributions to be Gaussian, which the authors demonstrate is not true for the Inception-v3 model. As such, the authors propose modeling both the distributions of Inception-v3 and that of the examined model as GMMs, in order to capture more information about both distributions. The authors perform two experiments to demonstrate that their proposed metric is a more robust alternative to FID, namely calculating both metrics between the ImageNet validation set and perturbed (adversarial and not) versions of either part of the ImageNet training set or a set of BigGAN generated images.

Strengths:
- The paper has a very clear structure: the authors identify a problem with the commonly used FID metric (violation of the gaussianity assumption), propose an alternative metric which is mathematically sound, and demonstrate its benefits compared to FID. This makes the paper well motivated and easy to follow.
- The motivation behind the paper is also cleanly presented. In particular, Figures 1 and 2 highlight the main problem behind the FID metric, the fact that the features of the Inception-v3 model do not follow a Gaussian distribution. This solidifies the goal behind the proposed WaM metric.
- The experiments proposed demonstrate that the WaM metric is more robust to noise in the distributions, when compared to FID. This is a useful property, since for our metric to be reliable, we expect it to not be easily affected by noise.

Negatives:
- I believe that it would be beneficial if the authors made more comparisons between WaM and other metrics used for the evaluation of GANs, other than FID. In particular, the WInD metric (Dimitrakopoulos et al. 2020), is very closely related to WaM, as the authors note (WInD also has a similar goal of circumventing the gaussianity assumption for FID). I suggest the authors further compare the theoretical aspects of WaM and WInD, as weil as calculating the WInD metric in their experiments.
- The experiments performed demonstrate that, when using BigGAN generated images, the benefit of WaM over FID is not that clear (although it is still affected less). This is especially true in the case of non-adversarial perturbations. I believe that the authors should further discuss this phenomenon, given that the overarching goal of this metric is the evaluation of generative models (currently there is some discussion at the end of section 5.2, but I believe it should be extended nonetheless).

Minor comments / Questions:
- In order to model the distributions, the number K of GMM components is identified. Is it assumed that it is the same for both distributions?
- There is a small typo in the caption of Figure 2: “isotrophic” -> “isotropic”.

References:
P Dimitrakopoulos, G Sfikas, and Christophoros Nikou. WInD: Wasserstein inception distance for evaluating generative adversarial network performance. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3182–3186. IEEE, 2020.
summary_of_the_review: I believe that this is a very clearly motivated paper, with a solid structure and experiments which support the main claims of the authors. Some comparisons with previous work on the subject are still needed in my opinion, but overall this is a solid work, and I believe it should be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper propose a new method to evaluate GANs. Current prevailing  evaluation of GANs is FID, which has one assumption that the evaluated data (or feature) has Gaussian distribution. However, this assumption is false when practically applying it.  Inspired by  recent work MW$_2$,  authors propose GMMs to evaluate two sets of distribution. 
main_review: Strengths
1. Authors clearly indicate the shortcoming of FID, which requires the Gaussian distribution of the evaluated feature. In practical, it is hard for the extracted feature to meet this requirement. The proposed method based GMMs could overcome this issue. Directly using GMMs still faces challenge, thus authors consider the relaxed problems of only considering joint distribution over GMMs.

2. Authors express simple and convincing examples (Figures 1~2) to support the motivation. For example, I like Figure 2 to show the failure case when using FID. 

3.The paper is easy to follow and well-written.

Weaknesses

1.  It seems the proposed method benefits from MW$_2$. I could not find more theoretical contribution for GAN evaluation, although them theoretical contribution is not necessary.  

2. The experiment look weird except for Figures 1 and 2. The motivation of this paper is address the shortcoming of FID, which is about Gaussian assumption.  However, the experiment is to show the sensitiveness, which is not corresponding to the motivation.  Another question is about GAN evaluation,  is less sensitiveness important? as what is argued in this paper, it is true.

3. For different K,  do WaM always win compared to FID about the sensitiveness? I know it is time-consuming.

4. Do authors try to use the proposed method for more  GAN methods?  for  example,  comparing StyleGAN and StyleGANv2 methods, what is the results of the proposed method? I would like to see the application of the proposed method on GANs-based methods. Hoverer, this paper only focuses on comparison  with FID. 

5. More baselines should be compared to support the proposed method, such as KID and WInD.

---------------After rebuttal-----------------------
Thanks for authors' response. I think authors did not address my concerns about why authors performance the evaluation of the sensitiveness, and more comparison with the variants of StyleGAN. I agree with $\mathbf{reviewers 3jkq}$: KID also fixes the similar issue. I change myself into negative.
summary_of_the_review: Authors propose to use GMMs to evaluate GANs. I think it benefits from MW$_2$. I think the contribution is not enough. Furthermore, the experiment should be improved.
review_title: null

Meta-review: This paper proposes a new evaluating metric for assessing the quality of model-generated images, that aims to correct some of the problems with the popular FID metric. The reviewers acknowledge the importance of this problem, but do not find the empirical evaluation convincing. In particular, they highlight the following issues
* Comparing FID and the new metric on examples that are adversarially selected against FID does not provide a fair comparison.
* The methods are compared on images of bad quality (FID > 25) that are therefore not informative.
* The comparison against existing techniques is incomplete
* The reviewers raise concerns about how the comparison is done quantitatively

The reviewers are not sufficiently convinced by the author response regarding these issues. I therefore recommend not accepting the paper.
Decision: Reject