Evaluating generative networks using Gaussian mixtures of image features
Authors: Lorenzo Luzi
Carlos Ortiz Marrero
Nile N Wynar
Richard Baraniuk
Michael J. Henry
Venue: ICLR 2022 Submitted (2022)

Abstract: We develop a measure for evaluating the performance of generative networks given two sets of images. A popular performance measure currently used to do this is the Fréchet Inception Distance (FID). However, FID assumes that images featurized using the penultimate layer of Inception-v3 follow a Gaussian distribution. This assumption allows FID to be easily computed, since FID uses the 2-Wasserstein distance of two Gaussian distributions fitted to the featurized images. However, we show that Inception-v3 features of the ImageNet dataset are not Gaussian; in particular, each marginal is not Gaussian. To remedy this problem, we model the featurized images using Gaussian mixture models (GMMs) and compute the $2$-Wasserstein distance restricted to GMMs. We define a performance measure, which we call WaM, on two sets of images by using Inception-v3 (or another classifier) to featurize the images, estimate two GMMs, and use the restricted 2-Wasserstein distance to compare the GMMs. We experimentally show the advantages of WaM over FID, including how FID is more sensitive than WaM to image perturbations. By modelling the non-Gaussian features obtained from Inception-v3 as GMMs and using a GMM metric, we can more accurately evaluate generative network performance.

Under review as a conference paper at ICLR 2022
EVALUATING GENERATIVE NETWORKS USING GAUSSIAN MIXTURES OF IMAGE FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
We develop measure for evaluating the performance of generative networks given two sets of images. A popular performance measure currently used to do this is the Fréchet Inception Distance (FID). However, FID assumes that images featurized using the penultimate layer of Inception-v3 follow a Gaussian distri- bution. This assumption allows FID be easily computed, since FID uses the 2- 2-Wasserstein distance of two Gaussian distributions fitted to the featurized im- ages. However, not Gaussian: in particular, each marginal is not Gaussian. To remedy this prob- lem, we model the featurized images using Gaussian mixture models (GMMs) and compute the 2- 2-Wasserstein distance restricted to GMMs. We define perfor- mance measure, which we call WaM, on two sets of images by using Inception-v3 (or another classifier) to featurize the images, estimate two GMMs, and use the re- stricted 2- 2-Wasserstein distance to compare the GMMs. We experimentally show the advantages of WaM over FID, including how FID is more sensitive than WaM to image perturbations. By modelling the non-Gaussian features obtained from Inception-v3 GMMs and using GMM metric, more accurately evalu- ate generative network performance.
show that Inception-v3 features of the ImageNet dataset are
INTRODUCTION
Generative networks, such as generative adversarial networks (GANs) (Goodfellow et al., 2014a) and variational autoencoders (Kingma & Welling, 2013), model distributions implicitly by trying to learn a map from a simple distribution, such as a Gaussian, to the desired target distribution. Using generative networks, one can generate new images (Brock al., 2018; Karras al., 201 2019a;b; 2017; Kingma & Welling, 2013), superresolve images (Ledig et al., 2017; Wang et al., 2018), solve inverse problems (Bora et al., 2017), and perform host of image-to-image translation tasks (Isola et al., 2017; Zhu et al., 2017; 2016). However, the high dimensionality of an image distribution makes it difficult to model explicitly, that is, to estimate the moments of the distribution via some parameterization. Just estimating the covariance of a distribution requires p(++1) 2 parameters, where p is the feature dimension. For this reason, modelling distributions implicitly, using transformations of simple distributions, can be useful for high dimensional data. Since the generator network is typically nonlinear, the explicit form of the generated distribution is not known. Nonetheless, these generative models allow one to sample from the learned distribution.
Because we only have access to samples from these generative networks, instead of explicit prob- ability density functions, evaluating their performance can be difficult. As such, several ways of evaluating the quality of the samples drawn from generative networks (Borji, 2019) have been pro- posed, the most popular of which is the Fréchet Inception distance (FID) (Heusel et al., 2017). FII fits Gaussian distributions to features extracted from set of a real images and set of GAN- generated images. The features are typically extracted using the Inception-v3 classifier (Szegedy et al., 6a). These two distributions are then compared using the 2-Wasserstein (Villani, 2009; 2003) metric. While FID has demonstrated its utility in providing computationally efficient metric for assessing the quality of GAN-genertted images, closer examination reveals that the fundamental assumption of the FID method -ammely, that the underlying feature distributions Gaussian is invalid. more accurate model of the underlying features will capture a more comprehensive and informative assessment of GAN quality.

---

Under review as a conference paper at ICLR 2022
In this paper, first show that the features used to calculate FID not Gaussian, violating the main assumption in FID (Section 3). The 2- Wasserstten metric, which FID uses, cannot be extended past Gaussians easily because it is typically computationally intractable and does not have closed formed solutions for many families of distributions. Moreover, FID is only capturing the first two moments of the feature distribution and completely ignores all information present in the higher order moments. Missing this information biases FID toward artificially low values, an undesirable property for performance metric.
Thus, we propose using a Gaussian mixture model (GMM) (McLachlan & Peel, 2000) for the fea- tures instead, because GMMs can model more complex distributions and capture higher order mo- ments. GMMs are estimated efficiently and there exists a Wasserstein-type metric for GMMs (Delon & Desolneux, 2020) (Section 4) which allows us to generalize FID. We use this to develop our gen- erative model evaluation metric, WaM. We provide code for the community to use WaM at (link will be added after acceptance).
Finally, we show that WaM is not sensitive to visually imperceptible noise FID (Section 5). Since GMMs can capture more information than Gaussians, WaM more accurately identifies dif- ferences between sets of images and avoids the low score bias of FID. We therefore reduce the issue of FID being overly sensitive to various noise perturbations (Borji, 2019) by modelling more information in the feature distributions. We test perturbation sensitivity using additive isotropic Gaussian noise and perturbed images which specifically attempt to increase FID using backpropa- gation (Mathiasen & Hvilshoj, 2020b). The ability of WaM to model more information in the feature distribution makes it better evaluation metric for generative networks.
2
RELATED WORK
2.1 WASSERSTEIN DISTANCE
There are several ways define a distance metric between probability distributions. A popular metric from optimal transport (Villani, 2003; 2009) is the p-Wasserstein metric. We first are given Polish metric space X with a metric d. Given p E (0, oo) and two distributions P and Q on X with finite moments of order the p- D-Wasserstein metric is given by
(PP inf xxxx d(z, v) 1
where the infimum taken over all joint distributions of and Q. Different values of p yield different metric properties; in image processing, the 1-Wasserstein distance on discrete spaces is used and called the earth mover distance (Rubner al., 2000). The 2-Wasserstein metric (Dowson & Landau, 1982; Olkin & Pukelsheim, 1982) is often used when comparing Gaussians since there exists closed form solution for
Wi (N(m): E1).N (22,E2)) 11 p2
Tr Σ + E2 - - ((
(1)
as is used to calculate the Fréchet Inception distance.
2.2 FID AND VARIANTS
The Fréchet Inception distance (FID) (Heusel et al., 2017) is a performance measure typically used evaluate generative networks. order compare two sets of images, X, and X2, they are featurized using the penultimate layer of the Inception-v3 network to get sets of features F1 and Pz. For ImageNet data, this reduces the dimension of the data from 224 224 150,528 to 2048. At this point, Heusel et al. assume that these features are Gaussian and use Equation (1) to obtain distance between them.
There are several ways that FID has been improved. One work has shown that FID is biased (Chong & Forsyth, 2020), especially when is computed using a small number of samples. They show that FID is unbiased asymptotically and show how to estimate the asymptotic value of FID to obtain an unbiased estimate. Others have used a network different from Inception-y3 to evaluate data that is not from ImageNet; for example, LeNet-like (LeCun et al., 1989) feature extractor can be used
2

---

Under review as a conference paper at ICLR 2022
for MNIST. In this work we focus on several different ImageNet feature extractors because of their widespread use. Modelling ImageNet features has been improved due to a conditional version of FID (Soloveitchik et al., 2021) which extends FID to conditional distributions, and class-aware Fréchet distance (Liu et al., 2018) which models the classes with GMMs. In this work, we do not consider conditional versions of FID,, but work be extended to fit such formulation in straightforward manner. Moreover, we use GMMs over the feature space rather than one component per class as is done in the class-aware Fréchet distance.
Another metric related to our proposed metric is called WInD (Dimitrakopoulos et al., 2020). WInD uses combination of the 1- I-Wasserstein metric on discrete spaces with the 2- Wasserstein metric on R.. For this reason, it is not W-ssersteein metric in I' or between GMMs. For example, if P and Q are mixture of Dirac delta functions then the WInD distance hetween them becomes the 1-Wasserstein distance. However, if P and Q are Gaussians, then the WInD distance between them becomes the 2- Wasserstein distance. Moreover, if and Q are arbitrary GMMs, the relationship between WInD and the p- p-Wasserstein metrics is not clear. This means that WInD can alternate between the 1- I-Wasserstein and 2- 2-Wasserstein distance depending on the input distributions. this paper, focus using metric which closely follows the 2- 2-Wasserttein distance is currently done with FID.
2.3
2W2
A closed form solution for the 2-Wasserstein distance between GMMs is not known. This is be- cause the joint distribution between two GMMs is not necessarily GMM. However, if we restrict ourselves to the relaxed problem of only considering joint distributions over GMMs, then the re- sulting 2- 2-Wasserstein distance of this new space is known. The restricted space of GMMs is quite large since GMMs can approximate any distribution arbitrary precision, given enough mixture components. So given two GMMs, P and Q, we can calculate
=inf inf / d(z, (1, 7 Jxxx
M =
where the infimum is over all joint distributions which are also GMMs. Constraining the class of joint distributions is a relaxation that has been done before (Bion-Nadal et al., 2019) due to the difficulty of considering arbitrary joint distributions. This metric, MW2, appears in a few different sources in the literature (Chen et al., 2016; 2018; 2019) and has been studied theoretically (Delon & Desolneux, 2020); recently, implementations of this quantity have emerged."
The practical formulation of this problem is done as follows. Let P = EEe mas and Q = Σx= Oglli be two GMMs with Gaussians M4 for i E (1,..., Kol,j E [I,...,K,]] Then, we have that
MW3 Q) min Σ Yi (u,,)))
where "y is taken to be the joint distribution over the two categorical distributions [T] and [ax O:,]; hence, y in this case is actually matrix. Thus, MW2 can be implemented as a discrete optimal transport plan and efficient software exists to compute this (Flamary et al., 2021).
MW2 is a great candidate for modelling the distance between GMMs for several reasons; most importantly, it is an actual distance metric. Since we are restricting the joint distribution to be a GMM. we see that MW2 must be greater than or equal the 2- 2-Wasserstein distance between two GMMs. Moreover, MW, clearly approximates the 2- 2-Wasserstein metric; Delon & Desolneux derive bounds showing how close MW2 is to Wz. It is also computationally efficient to compute because can be formulated as a discrete optimal transport problem, making it practical. The strong theoretical properties and computational efficiency MW, make it a prime candidate to calculate the distance between GMMs.
hhttps://gi thub. com/ judel gmmet
3

---

Under review as a conference paper at ICLR 2022
ResNet- feature #172
ResNet-50 feature #559
ResNeXt-101 (32x8d)
Inception-v3 feature # #1216
feature #1653
2
2
2
2
0
3
Feature value
Feature value
Feature value
Feature value
Histograms showing non-Gaussianity of randomly chosen features from the ImageNet validation dataset featurized by ResNet-18, 18, ResNet-50, ResNeXt-101 (32x8d) and Inception-v3. They are non-negative because these features are passed through a ReLU layer and then average pooled; for this reason, we have spike around O. These histograms empirical distributions and thus have of 1.
Figure 1:
-15
15
15
15
15
15
15 Feature value
Feature value
Feature value
Feature value
Feature value
(a)
(b)
(c)
(d)
(e)
/ =
P1, -8V5, 2,5
100/3
M1, 12 = -4V5,
==
0 = 10
01, 02 2v10. v15 b= 10v/
01 02 2/5,2V5
b= 5v2
1i. T2 = 0.2, 0.8
#1. T2 0.5,
Figure 2: The FID score hetween each pair of the distributions shown above is zero although they are clearly different distributions. This is because Equation (1) is only defined for Gaussians, and FID treats any input distribution Gaussian, even is not. We plot one dimensional distributions here visualization purposes, but the FID score will remain zero even we extend these distributions to their high dimensional isotrophic counterparts, All that is required for the FID score between two distributions to be zero is that their first two moments match. Figure 2a is the only Gaussian distribution. Figures 2b and 2d are Gaussian mixtures with two components, Figure a uniform distribution, and 2c Laplace distribution.
3 INCEPTION-V3 HAS NON-GAUSSIAN FEATURES ON IMAGENET
3.1 NON-GAUSSIAN FEATURES CAN DIFFER AND HAVE ZERO FID
The calculation of FID assumes that features from the penultimate layer of Inception-v3 (Szegedy al., 2016a) are Gaussian. This layer average pools the outputs several convolutional layers which are rectified via the ReLU activation. Though an argument can be made for why the preac- tivations of the convolutional layers are Gaussian (using the central limit theorem), the rectified and averaged outputs are clearly not. In fact, they are likely to be averages of rectified Gaus- sians (Beauchamp, 2018). Although these features are high dimensional and cannot be visual- ized, we plot the histograms of randomly selected feature extracted with Inception-v3, ResNet- 18, ResNet-50, and ResNeXt-101 (d2 in Figure 1. construct these histograms using the 50,000 images in the ImageNet validation dataset. We see that none these randomly selected features look Gaussian.
If the Gaussian assumption of FID is false, one can achieve low FID values while having drastically different distributions, as shown on Figure 2. This is true in part because FID only considers the first two moments of the distributions being compared: differences in skew and higher order moments are not taken into account in the FID calculation. This can cause FID to be extremely low when the distributions being compared quite different.
4

---

Under review as a conference paper at ICLR 2022
3.2 IMAGENET FEATURES ARE NOT GAUSSIAN
Testing if Inception-v3 features are Gaussian is not trivial because they are 2048-dimensional. Even if each marginal distribution appears Gaussian, we cannot be sure that the joint distribution is Gaus- sian. However, if the marginals are not Gaussian, this implies that original distribution is not Gaus- sian. Therefore, we conducted a series of Kolmogoroy- statistical nonparametric goodness-of-fn test that verifies whether an underlying probability distri- bution, in our case the marginals, differs from a hypothesized distribution, a Gaussian distribution.
hypothesis tests (Dodge, 2008), a
We calculated features from the entire ImageNet validation dataset using ResNet-18, ResNet-50, ResNeXt-101 (d)x and Inception-v3. For each set of features, we then tested each marginal using the Kolmogono-Smiimon
tests with the hypothesis that the features come from normal distribution. Using a p-value of 0.01, the test found that 100% of the marginals fail to pass the hypothesis. This confirms, with high certainty, that neither the marginals nor the whole feature distribution is Gaussian.
Since the features of Inception-v3 not Gaussian, have few options. The first option is to features before the average pooling layer and ReLU operation because these features may actually be Gaussian. However, these features are extremely high dimensional (64 2048 131, 002) and thus very hard to estimate accurately. Another option we have is to use a different network for feature extraction; however, most networks which perform very well on ImageNet have high dimensionality convolutional features followed by ReLU and average pooling, c.g., ResNeXt-101 (32x 8d). Moreover, trying to obtain Gaussian features is not a general solution because even if the training data has Gaussian features, new data may not. Therefore, we decided to model these non Can-Gausiaan features using Gaussian mixture models which can capture information past the first two moments of a distribution.
4 WAM MODEL DETAILS
4.1 WAM A GAUSSIAN MIXTURE MODEL CAN LEARN MORE COMPLEX DISTRIBUTIONS
In this work we use the Gaussian mixture model (GMM) to model non-Gaussian features. GMMs are a generalization of Gaussian distributions (i.e., when the number of components equal 1) and hence we can generalize FID) using the formulas discussed in Section 2.3. Moreover, any distribution can be approximated to arbitrary precision using a GMM (Delon & Desolneux, 2020). Estimation of GMM parameters are also computationally efficient and have been studied thoroughly (Bishop, 2006; McLachlan & Peel, 2000). Most importantly. we can calculate the distance between GMMs using equation 2.
We calculate our performance metric for generative models by using the MW2 (Delon & Desolneux, 2020) metric for GMMs GMMs estimated from extracted features of images. The procedure is summarized as follows: We first pick a network, such as Inception-v3, to calculate the features. These features are then used estimate a GMM with K components. We do this for real data and for generated data, We then calculate the FID of each combination of components, one from the real data GMM and one from the generated data GMM. Then, we solve a discrete optimal transport problem using the 2-Wasserstein distance squared as the ground distances to obtain WaM. We use = 50,000 samples because this was shown be an approximately unbiased (Chong & Forsyth, 2020) estimate of FID. We call our metric WaM since it is Wasserstein-tyee metric on GMMs of image features.
We fit the GMM to the data using the expectation maximization algorithm implemented in the scikit- learn (Pedregosa et al., 2011) package in Python. We model the features with full covariance ma- trices so that we are truly generalizing FID. One can fit diagonal or spherical covariance matrices speed is required (as supported in our code), but this will yield simpler GMMs. We consid- ered several GPU implementations of GMM fitting instead of the scikit-learn CPU implementation. However, the sequential nature of the expectation maximization algorithm caused the run times to be similar for GPU and CPU algorithms.
5

---

Under review as a conference paper at ICLR 2022
18
ResNet-50
ResNeXt-101 (32x8d)
Inception v3
1 10 30
50
1
10
30
50
1
10
30
50
1
10
30
50
K
K
K
Figure 3: AIC curves for ResNet-18. 18, R0sNet-50. ResNeXt-101 (32x8d), and Inception-v3 features used for picking the number of mixture components K. We choose K 10 for ResNet-18, K 50 for ResNet-500, and K = 30 for both ResNeXt-100 (32 8d) and Inception-v3.
4.2 USING DIFFERENT NETWORKS
While Inception-v3 v is excellent network to use for feature extraction because of its high accuracy on the ImageNet classification task, we also use ResNet-18, ResNet-50, and ResNeXt-101 (32x8d) For each network, Inception-v3. We use ResNet-18 because features are only 512-dimensional and hence can be calculated faster than Inception-v3. ResNet-50 performs better than ResNet-18 and in some of our experiments. Finally, ResNeXt-100 (32x8d) achieves the highest accuracy in the ImageNet classification task of all the pretrained classifiers on Pytorch (Paszke et al., 2019).
the penultimate layer for feature extraction, was done originally for
included
4.3 PICKING AND FITTING THE GMM
When modelling features, we must pick the number of components we choose have in our GMM. If we pick K = 1 (and use Inception-v3 as our feature extractor), then we just calculate FID. The more components we pick, the better our fit will be. However, we pick K to be too large, such as K 2 N, then we may overfit in the sense that we can have each component centered around single data points. This is clearly not desirable, so we fit all of our GMMs with a maximum of K = 50 components.
We use the Akaike information criterion (AIC) to choose K since likelihood criteria are well suited for density estimation (MeLachlan & Peel, 2000). However, calculating AIC for multiple compo- nents will take significant computation time and power if done every time one wants to calculate WaM. For this reason, pick specific K based on the ImageNet validation set. A value for K which models the ImageNet validation dataset well should be a good K for modelling similar image datasets. As shown in Figure 3, the AIC curves have varying shapes. We the kneed method (Satopaa et al., 201 1) for our choice of K (using $ = 1 in the official implementation ?) for the ResNet-18 features since have textbook power-law like curve. For ResNet-50, ResNeXt- 101 (32 x 8d), and Inception-v3 we use our best judgement since the curves do not have a knee for reasonable values of K.
EXPERIMENTS
5.1 TARGETED PERTURBATIONS - A REAL EXAMPLE OF WHEN FID FAILS AND WAM DOES NOT
Although features extracted from classifiers are not Gaussian, we do not have a perfect model for them. In fact, it is difficult to come up with distributions of features without images, because we typ- ically have to calculate them. Hence, if we want see when FID fails and WaM does not, must construct data that will give us a distribution of features which cannot be modelled well with FID. This can be done if we start with set of images, perturb them in order to increase FID, then calcu- late WaM on the perturbed images. Since WaM is a generalization of FID, the perturbed images will
Phttps://aa thub. com/arvked /need

---

Under review as a conference paper at ICLR 2022
Original (BigGAN)
Perturbed (BigGAN)
Original (ImageNet)
Perturbed (ImageNet)
FID 25.13
FID 98.69
FID = 2.51
FID = 54.12
WaM = 142.12
WaM 297.49
WaM = 52.60
WaM = 174.76
RAD 3.93
RADD 21.5
RwaM = 2.09
RwaM = 3.32
R = 1.88
R 6.48
Figure 4: Samples of images showing targeted perturbations which artificially inflate FID but not WaM. The two original images above are randomly selected from a set of 50,000 images generated by BigGAN and a set of 50,000 images of the ImageNet training dataset. We cannot visually perceive the difference hetween the original and perturhed images, despite the datasets from which they were selected elearly demonstrating drastic change in FID. Note that FID of the original ImageNet training data is approximately times lower than for the BigGAN generated images. The FID, WaM, and It values were calculated using ResNet-18.
affect WaM as well, particularly the first and second moments of the feature distribution. However, since WaM can capture more information than FID on the feature distributions, we hypothesize that WaM will not be as affected as FID.
We construct these perturbed sets of images by backpropagating through FID using the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014b). Backpropagation through FID was recently shown to be useful in finding adversarial examples for FID and for improving image quality of trained generators (Mathiasen & Hvilshøj, 2020b;a)².
To calculate FID or WaM, we must compare two sets of images; thus, we always compare to the ImageNet validation set. This allows us calculate the FID and WaM of the ImageNet validation set against real images from the ImageNet training set, generated images from BigGAN Brock et al. (2018), and perturbed images from each. We used 50,000 images for each set when doing all the comparisons. To produce the adversarial images, we extracted the features from all the 50,000 ImageNet validation images, then ran FGSM with an e = 0.01 and batch size of 16 until we perturbed all 50,000 of our target images (e.g., ImageNet training set). It is worth noting that during training we calculated the gradients that maximize FID between the batch of 16 images and the extracted features the ImageNet validation set.
Comparing FID and WaM is difficult because they are different metrics with different seales. For this reason, we must normalize them when comparing. Thus, we define RED to be the ratio of the FID of the perturbed images over the FID of the original images. Hence, REnD shows how much FID has increased due to the perturbation. Similarly, we define RwaM to be the ratio of WaM squared of the perturbed images over WaM squared of the original images. FID is typically reported as the 2- 2-Wasserstein squared distance, hence we square WaM so that it is also a squared distance. Then define R. Rym to be the ratio for these two increases. Hence, for R 1 we have that FID increased faster than WaM due perturbation.
When we perturb images generated from BigGAN (Brock et al., 2018) or the ImageNet training data cannot visually perceive a difference, shown in Figure 4. However, for the BigGAN images, FID increases by a factor of RADD 3.93 while WaM only increases by a factor of RwaM = 2.09. This difference is even
evident with real images drawn from the ImageNet training data set. We see that the FID score after perturbation increases by REDD 21.5 times! Since WaM only increases by RwaM 3.32 times, we see that FID increased R 6.48 times more than WaM for
'Athhough the authors of the paper introduced Fast FID, we hackpropagated through FID our work.

---

Under review as a conference paper at ICLR 2022
original
0.01
0.05
0.1
0.2
0 0.5
0
0.01
0.05
0.1
0.2
0.5
FID(orig)
183.59
183.59
183.59
183.59
183.59
(pD(t
190.57
241.62
324.32
569.18
731.74
(ori)
285.04
285.04
285.04
285.04
285.04
WaM2 (pert)
292.63
339.39
418.59
614.83
741.13
RFID
1.04
1.32
1.77
3.1
3.99
RwaM
1.03
1.19
1.47
2.16
2.6
R
1.01
1.11
1.2
1.44
1.53
Figure 5: values for -ggnerttee images using additive isotropic Gaussian noise showing that FID is slightly more sensitive than WaM to noise perturbations of generated images. The noise perturbations in this experiment all greater magnitude than the targeted perturbations in Section 5.1. The original image above randomly selected from of 50,000 images generated by BigGAN. The FID, WaM, and R values calculated using ResNet-18.
imperceptible, but targeted, perturbation. A metric which reflects perceptual quality perfectly would not be affected whatsoever by these perturbations. Neither FID nor WaM are perfect, but WaM's lower sensitivity visually imperceptible perturbation is better aligned with the objective of assessing perceptual quality in images.
Even though these perturbations are targeted specifically increase FID (and not WaM), this is still a fair comparison of these two metrics. WaM can learn a Gaussian distribution (c.g.. if all the com- ponents are the same), yet FID and WaM yield different results in this experiment, implying that the features are not modeled well by FID and benefit from the additional information captured by WaM. Moreover, since the AIC decreases after K 1, we know that WaM using more information from the feature distributions. WaM uses more information than FID and is less sensitive than FID to these imperceptible perturbations.
Since Kernel Inception distance (KID) Binkowski et al. (2018) may also be able capture the higher order moments of the features, we ran our experiments using KID as well. Our results show that KID is significantly more affected than WaM by the perturbations targeted to fool FID. The details discussed in Section of the supplementary material. Thus, WaM is less sensitive to imperceptible FIID-targetd noise than both FID and KID.
5.2 RANDOM PERTURBATIONS
In this section we show that WaM is also less sensitive than FID to additive isotropic Gaussian noise. We do this by corrupting images generated from BigGAN and the ImageNet training dataset by adding isotropic Gaussian noise with standard deviation E {0.01, 0.05, 0.1, 0.2, 0.5} and then calculating their features. Samples of how these noisy images compare the original are shown in Figures 5 and 6. In these experiments, we use ResNet- 18 to extract the features. The e = 0.01 used in Section 5.1 corresponds approximately
0.0014, meaning that the additive random noise in Figures 5 and 6 perturbs the images much more than the targeted noise in Figure 4.
We see that FID and WaM do not increase much when calculated using noisy BigGAN generated images, but FID skyrockets when calculated using ImageNet training data. This is likely due to FID not being able to capture the differences between the ImageNet training and validation set. One can justly assume that both data sets are sampled from the same distribution; however, comparing the distributions from which they are sampled. We are comparing the two sets of images from the training and validation set, which not the same. Therefore, FID' inability model the correct distribution of features causes it to become extremely sensitive to this noise, even when is barely visually perceptible. This sensitivity of FID noise has been noted before (Heusel et al.,
not
8

---

Under review as a conference paper at ICLR 2022
original
0.01
0.05
0.1
0 0.2
0.5
0.01
0.05
0.1
0.2
0.5
FID(orig)
4.08
4.08
4.08
4.08
4.08
FID(pert)
4.52
19.83
51.14
120.57
323.05
(org))
48.75
48.75
48.75
48.75
48.75
WaM (pert)
50.18
99.95
120.09
184.36
400.65
RFID
1.11
4.87
12.55
29.59
79.27
RwaM
1.03
2.05
2.46
3.78
8.22
R
1.08
2.37
5.09
7.82
9.65
Figure R values for real images (ImageNet training data) using additive isotropic Gaussian noise showing that FID is significantly more sensitive than WaM to noise perturbations of real images. The noise perturbations in this experiment randomly selected from of 50,000 images of the ImageNet training dataset. In contrast
all greater in magnitude than the targeted perturbations in Section 5.1. The original
image above Figure 5, we see that FID is more sensitive to these perturbations when the images look more realistic. The FID and WaM values were calculated using ResNet-18.
2017; Borji, 2019). FID is affected R = 5.09 times as much as WaM when the noise is barely visible (a 0.1), making WaM must desirable to use in noisy contexts.
We also use KID to evaluate BigGAAN-generttdd and ImageNet images and discuss the details in Section of the supplementary material. Our findings show that KID has similar sensitivity as WaM on BigGAN-generated images but that KID skyrockets on ImageNet images as compared to WaM. This means that for imperceptible additive noise on realistic images, WaM is less sensitive than KID and can offer a better means to evaluate generative models which produce realistic images.
good metric for evaluating generative network performance should be able to capture the quality of generated images at all stages. FID does not do this well. FID is sensitive to noise perturbations, especially when the images look realistic; hence, R is much larger for the ImageNet training data than it is for the BigGAN generated images. As generative networks get better and better, must use more information (not just the first and second moment) from the feature distribution in order to accurately evaluate generated samples.
CONCLUSIONS
We generalize the notion of FID by modeling image features with GMMs and computing relaxed Wasserstein distance on the distributions. Our proposed metric, WaM, allows us to accurately model more complex distributions than FID, which relies on the invalid assumption that image features follow a Gaussian distribution. Moreover, we show that WaM is less sensitive to both imperceptible targeted perturbations that modify the first two moments of the feature distribution and imperceptible additive Gaussian noise. This is important because we want a performance metric which is truly reflective of the perceptual quality of images and will not vary much when visually imperceptible noise is added. We can use WaM to evaluate networks which generate new images, superresolve images, solve inverse problems, perform image-to-image translation tasks, and more. As networks continue to evolve and generate more realistic images, WaM can provide a superior model of the feature distributions, thus enabling more accurate evaluation of extremely-realistie generated images.
REFERENCES
Maxime Beauchamp. On numerical computation for the distribution of the convolution n in- dependent rectified gaussian variables. Journal de la Société Française de Statistique, 159(1):

---

Under review as a conference paper at ICLR 2022
88-111, 2018.
Mikolaj Binikowski. Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. arXiv preprint arXiv:1801.01444 2018.
Jocelyne Bion-Nadal, Denis Talay, et On a wasserstein-type distance between solutions to stochastic differential equations. The Annals of Applied Probability, (9(3):1609-1660 2019.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using genera- tive models. In International Conference on Machine Learning, pp. 537-546. PMLR, 2017.
Ali Borji. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding, 179:41 65, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint 10Xiv:8809.11000000 2018.
Yongxin Chen, Tryphon T Georgiou, and Allen Tannenbaum. Optimal transport for gaussian mixture models. IEEE Access,
2018.
Yukun Chen, Jianbo Ye, and Jia Li. A distance for hmmss based on aggregated wasserstein metric and state registration. European Conference on Computer Vision, pp. 451 451-466. Springer, 2016.
Yukun Chen, Jianbo Ye, and Jia Li. Aggregated wasserstein distance and state registration for hidden markov models. IEEE transactions on pattern analysis and machine intelligence, 42(9): 2133-2147, 2019.
Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6070-6079, 2020.
Julie Delon and Agnès Desolneux.
posserstein-type distance in the space of gaussian mixture models. SIAM Journal on Imaging Sciences, 13(2):936-970, 2020.
P Dimitrakopoulos, G Sfikas, and Christophoros Nikou. Wind: Wasserstein inception distance for evaluating generative adversarial network performance. ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3182-3186. IEEE, 2020.
Yadolah Dodge. Kolmogoro--Smirov Test, pp. 283-287, Springer New York, New York, NY, 2008 ISBN 978-0-387-32833-1 doi: 10.10077978-0-33--22833 1211, https: //do::///o/ .org/ 10. 100.100//00 978 ---------77--------- 3-----3-2333--2 214.
DC Dowson and BV Landau, The fréchet distance between multivariate normal distributions. Jour- nal of multivariate analysis, 12(3):450-455 1982.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanis- las Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy. Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica Sutherland, Romain Tavenard, Alexander Tong. and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78): 1-8, 2021. URL http: jmlr....... orq/ papers/ v22 202 20 441.011111..................
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems (NIPS), pp. 2672-2680, 201
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 1:Xiv:4412.60 /412.5522. 2014b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NIPS), pp. 6626-6633 6637, 2017.
10

---

Under review as a conference paper at ICLR 2022
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-ine -omag translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and patter recognition, pp. 1125-1132 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2019a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. arXiv preprint arXiv:1912.04958, 2019b.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding
variational bayes. arXiv preprint
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hub- bard, and Lawrence Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 46814661189 1661-4 2017.
Shaohui Liu, Yi Wei, Jiwen Lu, and Jie Zhou. An improved evaluation framework for generative adversarial networks. arXiv preprint arXiv:1803.07474,
2018. 2018.
Alexander Mathiasen and Frederik Hvilshøj. Backpropagating through fr' 'echet inception distance. arXiv preprint 2XXx:2009..4075.5 2020a.
Alexander Mathiasen and Frederik Hvilshøj. Fast fr), 'echet inception distance. arXiv preprint arXiv:2009. 14075, 2020b.
Geoffrey McLachlan and David Peel. Finite Mixture Models. John Wiley & Sons, Inc., edition, 10 2000. ISBN 0-471-00626-2
Ingram Olkin and Friedrich Pukelsheim. The distance between two random vectors with given dispersion matrices. Linear Algebra and its Applications, 48:257-263, 1982.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. Curran Associates, Inc., 2019. URL http: /papers.papppeeaaaae new.ennerniii cc cc/paper////eeeeecceecceeeeeeeee http:p//c..........a.....e.eaaaa pdf.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance metric for image retrieval. International journal of computer vision, 40(2):99-121,
2000. 2000.
Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. Finding a" kneedle" in haystack: Detecting knee points in system behavior. In 201 31st international conference on distributed computing systems workshops, pp. 166-171. IEFF, 2011.
11

---

Under review as a conference paper at ICLR 2022
Michael Soloveitchik, Tzvi Diskin, Efrat Morin, and Ami Wiesel. Conditional frechet inception distance. arXiv preprint 1rXi:22003.1521, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 1--2826, 2016a.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016b.
Cédric Villani. Topics in optimal transportation. American Mathematical Soc., 2003.
Cédric Villani. Optimal Iransport: old and new, volume 338. Springer, 2009.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong. Yu Qiao, and Chen Change Loy. ESRGAN: Enhanced super-resolution generative adversarial networks. In Pro- ceedings of the European conference on computer vision (ECCV) workshops, 2018.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipu- lation the natural image manifold. In European conference computer vision, pp. 597-613. Springer, 2016.
Jun-Yan Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
12

---

Under review as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
A KERNEL INCEPTION DISTANCE EXPERIMENTS
Kernel Inception distance (KID) Binikowski al. (2018) is popular method evaluate the perfor- mance of a GAN which uses embeddings from powerful classifiers, such as Inception-v3 Szegedy al. (2016b). We use the cubic polynomial kerne,, i.., k(z, y) () y 1) for æ, y R to compute similarities between featurized samples, as is typically done. We use this method to evaluate WaM's sensitivity to imperceptible noise perturbations. To this, we define RKID to be the ratio of the KID of the perturbed images over the KID of the original images. We further define RyI
R'
Rwws
The targeted perturbation experiments in Section 5.1 of the original paper had values of R = 1.88 and R 6.48 for BigGAN generated and real images, respectively. Hence, Figure 7 shows that KID is is still significantly affected by these perturbations even though they are constructed to fool FID, not KID. WaM less sensitive than FID and KID in this setting, implying that does not depend as heavily on the first two moments and can capture more higher order information than both metrics.
Original (BigGAN)
Perturbed (BigGAN)
Original (ImageNet)
Perturbed (ImageNet)
KID 0.071
KID 0.248
KID = 0.006
KID 0.387
2 142.12
2 297.49
WaM 52.60
WaM 174.76
RXID 3.492
RgID = 61.191
Rwam 2.09
Rwam 3.32
R' = 1.67
R' = 18.43
Figure Samples images showing targeted perturbations which artificially inflate FID but not WaM; how- ever, we show KID values being indirectly inflated more than WaM. The two original images above are ran- domly selected from set of 5,0000 images generated by BigGAN and set of 50,001 images of the ImageNet training dataset. We cannot visually perceive the difference between the original and perturbed images, despite the datasets from which they were selected clearly demonstrating drastic change KID. The KID, WaM, and R' values were calculated using ResNet-18.
We now consider the random perturbations in Section 5.2 of the original paper and evaluate R' on them, as shown in Figures 8 and 9. We see that KID has similar sensitivity to WaM on BigGAN generated images but much higher sensitivity real images. In fact, KID has higher sensitivity real images than FID. We stress that the ability to evaluate realistic images important because that is what want generate. Therefore, WaM provides means evaluate realistic images better than FID and KID under imperceptible noise perturbations.
13

---

Under review as a conference paper at ICLR 2022
original
0.01
0.05
0.1
0.2
0.5
0.01
0.05
0.1
0.2
0.5
KID(on
2.11
2.11
2.11
2.11
2.11
KID(pert)
2.22
2.74
3.47
4.65
5.23
(ori)
285.04
285.04
285.04
285.04
285.04
(per(
292.63
339.39
418.59
614.83
741.13
RKWD
1.05
1.29
1.64
2.2
2.47
Rwam
1.03
1.19
1.47
2.16
2.6
R'
1.02
1.09
1.12
1.02
0.95
Figure I'' values BigeAN--eeneratd images using additive isotropic Gaussian noise showing that KID has similar sensitivity as WaM noise perturbations generated images. The original image above was randomly selected from a set of 50,000 images generated by BigGAN. The KID, WaM, and R' values were calculated using R8sNet-18.
original
0.01
0.05
0 0.1
σ 0.2
0.5
0.01
0.05
0.1
0.2
0.5
KID(orig)
0.025
0.025
0.025
0.025
0.025
KID(pert)
0.033
0.187
0.496
1.146
2.745
(oig)
48.75
48.75
48.75
48.75
48.75
WaM2 (pert)
50.18
99.95
120.09
184.36
400.65
RKID
1.283
7.363
19.528
45.143
108.118
Rwam
1.03
2.05
2.46
3.78
8.22
R'
1.247
3.591
7.928
11.939
13.157
Figure 9: R' values for real images (ImageNet training data) using additive isotropic Gaussian noise showing
that KID is significantly more sensitive than WaM to noise perturbations of real images. The original image above randomly selected from set of 50,000 images of the ImageNet training dataset. In contrast Figure 5, we see that KID more sensitive to these perturbations when the images look more realistic. The FID and WaM values were calculated using ResNet-18.
14


Reviews:
[{"review_id": "oyDYCKPrcm2", "reviewer": "Reviewer_xevY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes an alternative metric (WaM) for the evaluation of generative networks, as an alternative to FID. The motivation behind this metric is the fact that the mathematical formulation for the FID metric requires the distributions to be Gaussian, which the authors demonstrate is not true for the Inception-v3 model. As such, the authors propose modeling both the distributions of Inception-v3 and that of the examined model as GMMs, in order to capture more information about both distributions. The authors perform two experiments to demonstrate that their proposed metric is a more robust alternative to FID, namely calculating both metrics between the ImageNet validation set and perturbed (adversarial and not) versions of either part of the ImageNet training set or a set of BigGAN generated images.\n\nStrengths:\n- The paper has a very clear structure: the authors identify a problem with the commonly used FID metric (violation of the gaussianity assumption), propose an alternative metric which is mathematically sound, and demonstrate its benefits compared to FID. This makes the paper well motivated and easy to follow.\n- The motivation behind the paper is also cleanly presented. In particular, Figures 1 and 2 highlight the main problem behind the FID metric, the fact that the features of the Inception-v3 model do not follow a Gaussian distribution. This solidifies the goal behind the proposed WaM metric.\n- The experiments proposed demonstrate that the WaM metric is more robust to noise in the distributions, when compared to FID. This is a useful property, since for our metric to be reliable, we expect it to not be easily affected by noise.\n\nNegatives:\n- I believe that it would be beneficial if the authors made more comparisons between WaM and other metrics used for the evaluation of GANs, other than FID. In particular, the WInD metric (Dimitrakopoulos et al. 2020), is very closely related to WaM, as the authors note (WInD also has a similar goal of circumventing the gaussianity assumption for FID). I suggest the authors further compare the theoretical aspects of WaM and WInD, as weil as calculating the WInD metric in their experiments.\n- The experiments performed demonstrate that, when using BigGAN generated images, the benefit of WaM over FID is not that clear (although it is still affected less). This is especially true in the case of non-adversarial perturbations. I believe that the authors should further discuss this phenomenon, given that the overarching goal of this metric is the evaluation of generative models (currently there is some discussion at the end of section 5.2, but I believe it should be extended nonetheless).\n\nMinor comments / Questions:\n- In order to model the distributions, the number K of GMM components is identified. Is it assumed that it is the same for both distributions?\n- There is a small typo in the caption of Figure 2: “isotrophic” -> “isotropic”.\n\nReferences:\nP Dimitrakopoulos, G Sfikas, and Christophoros Nikou. WInD: Wasserstein inception distance for evaluating generative adversarial network performance. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3182–3186. IEEE, 2020.", "summary_of_the_paper": "This paper proposes a new metric (WaM) for the evaluation of images generated by a network, as an alternative to the commonly used FID metric. The authors show that this alternative metric is more robust to perturbations of the images (both adversarially chosen and random), when compared to FID.", "summary_of_the_review": "I believe that this is a very clearly motivated paper, with a solid structure and experiments which support the main claims of the authors. Some comparisons with previous work on the subject are still needed in my opinion, but overall this is a solid work, and I believe it should be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DK4ZDTLcZTC", "reviewer": "Reviewer_3jkq", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Clear writing and demonstrations.\n+ Meaningful research topic.\n\nWeaknesses:\n- The technical novelty is marginal. It is a plain combination between GMM  for distribution estimation and MW2 for distance formulation.\n- The technical correctness is doubtful. In Figure 1 we see the feature distributions are non-negative, single-sided, and long-tail. If a single Gaussian is not suitable enough to estimate this distribution, a mixture of Gaussians would not be much better in theory. Instead, KID [1] fundamentally handles this issue: KID does not assume a parametric form for the distribution of ReLU activation; its cubic kernel fits the long-tail distribution better; it has an unbiased estimator. Please cite [1] and discuss why the proposed WaM would be more advantageous than KID.\n- The technical details are unclear. How to calculate FID when perturbing a set of images? FID is calculated by two large sets of images, say, 50k images per set. Did the authors perturb the 50k images at the same time to increase FID? If so, how if some images are perturbed much more than the others? In this case, please show the image examples with the maximal amount of perturbations. Maybe the differences are visually perceptible.\n- The experiments are not convincing.\n  - For the claim that FID is more sensitive than the proposed WaM, in Section 5.1 images are perturbed by backpropagating through FID. There is no wonder FID looks more sensitive given this kind of perturbations. How about the other kind of perturbations: backpropagating through WaM?\n  - For Figure 5, all the FID and WaM values are in the range of 100+. The metric sensitivity in that range is meaningless because our research community cares about small FID/WaM values. BigGAN already achieves <10 FID on ImageNet. We therefore need a metric to be sensitive enough in this range to distinguish cutting-edge generative model techniques. \n  - Figure 6 is meaningless neither because the results are the sensitivity on real images. Research community cares about the quality on generated images.\n  - The baseline comparisons are insufficient. As mentioned above, please compare to KID [1], in terms of the sensitivity in the small value range, especially with the perturbations backpropagated through KID.\n\n[1] Bińkowski, Mikołaj, et al. \"Demystifying mmd gans.\" ICLR 2018.", "summary_of_the_paper": "This paper targets to a new evaluation metric for the performance of generative models given a set of real images and a set of fake images. The authors show that Inception-v3 features of the imageNet dataset are not Gaussian and remedy this issue by modeling image features using Gaussian mixture models (GMMs) and formulating the distribution distance between two GMMs by MW2. They demonstrate their metric is less sensitive than FID against image perturbations.", "summary_of_the_review": "See the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "53tNUgJAbKB", "reviewer": "Reviewer_Dqh4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. Authors clearly indicate the shortcoming of FID, which requires the Gaussian distribution of the evaluated feature. In practical, it is hard for the extracted feature to meet this requirement. The proposed method based GMMs could overcome this issue. Directly using GMMs still faces challenge, thus authors consider the relaxed problems of only considering joint distribution over GMMs.\n\n2. Authors express simple and convincing examples (Figures 1~2) to support the motivation. For example, I like Figure 2 to show the failure case when using FID. \n\n3.The paper is easy to follow and well-written.\n\nWeaknesses\n\n1.  It seems the proposed method benefits from MW$_2$. I could not find more theoretical contribution for GAN evaluation, although them theoretical contribution is not necessary.  \n\n2. The experiment look weird except for Figures 1 and 2. The motivation of this paper is address the shortcoming of FID, which is about Gaussian assumption.  However, the experiment is to show the sensitiveness, which is not corresponding to the motivation.  Another question is about GAN evaluation,  is less sensitiveness important? as what is argued in this paper, it is true.\n\n3. For different K,  do WaM always win compared to FID about the sensitiveness? I know it is time-consuming.\n\n4. Do authors try to use the proposed method for more  GAN methods?  for  example,  comparing StyleGAN and StyleGANv2 methods, what is the results of the proposed method? I would like to see the application of the proposed method on GANs-based methods. Hoverer, this paper only focuses on comparison  with FID. \n\n5. More baselines should be compared to support the proposed method, such as KID and WInD.\n\n---------------After rebuttal-----------------------\nThanks for authors' response. I think authors did not address my concerns about why authors performance the evaluation of the sensitiveness, and more comparison with the variants of StyleGAN. I agree with $\\mathbf{reviewers 3jkq}$: KID also fixes the similar issue. I change myself into negative.", "summary_of_the_paper": "This paper propose a new method to evaluate GANs. Current prevailing  evaluation of GANs is FID, which has one assumption that the evaluated data (or feature) has Gaussian distribution. However, this assumption is false when practically applying it.  Inspired by  recent work MW$_2$,  authors propose GMMs to evaluate two sets of distribution. ", "summary_of_the_review": "Authors propose to use GMMs to evaluate GANs. I think it benefits from MW$_2$. I think the contribution is not enough. Furthermore, the experiment should be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0S1pM9W6XM", "reviewer": "Reviewer_KkXg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The paper has a simple yet interesting observation that the extracted features from widely-used encoder networks on widely-used image datasets are not actually distributed as Gaussian, which violates the assumption of FID. The paper then leverage recent advances on calculating 2-wasserstein distance between GMMs to obtain a generalization of FID. From this perspective, the proposed metric has the potential of addressing some limitations of FID and contributing to be a better evaluation metric for image generation.\n\nThat being said, I have a number of concerns/questions as discussed below:\n- Because of the intractability of computing wasserstein distance between GMMs, the paper has to use an approximation by restricting the family of joint distribution to GMMs. In practice, how can we guarantee the quality of such approximation (which might be large)/quantify the suboptimality gap (since wasserstein between Gaussians are exact and the noise brought by approximation error may be too large wrt the signal in theory)? Moreover, as the paper discussed, this issue can be mitigated by using a large enough K (number of components). However, this may incur much more computation? For example, in equation 2, we need to compute K^2 wasserstein distance as well as a discrete optimal transport problem. When K=50 as used in the paper, we need to compute 2500x 2-wasserstein? Also a too large K will cause over-fitting, so we need to carefully tune this parameter, which is not good for a standard/widely-used evaluation metric since we need to make sure all researchers in this domain are using the same hyperparameter for fair comparisons while the optimal one is hard to know.\n- At the beginning of sec 3.1, we know that preactivations of the convolution layers are Gaussian and Fig.1 indeed looks like a truncated gaussian? The truncation is mainly because the ReLU layer to force all features to be positive. Thus I may not agree with the argument in sec 3.2: \"The first option is to use features before the average pooling layer and ReLU operation because these features may actually\nbe Gaussian. However, these features are extremely high dimensional (64 \u0002 2048 = 131;072) and thus very hard to estimate accurately.\"\nCan we only remove the ReLU operation and keep the pooling layer to reduce the dimension while the features without passing ReLU are still approximately Gaussian? It seems this option is not studied in the experiments.\n- Roughly speaking, there is no new techniques proposed in this paper but a straightforward application of recent papers for calculating 2-wasserstein distance between GMMs. Thus I consider it an empirical study, where thorough and convincing experimental studies are needed. Then an important question is whether the evaluation protocols (R_FID, R_WaM, R) are good enough. From my perspective, I think there are issues with them: comparing metrics with different scales are hard and using these ratios may not make sense either. Suppose metric A range from [0, 100] and metric B range from [0, 10], also assume the sample quality linearly change within their ranges. Now if metric A change from 1 to 3 (hence R_A = 3) while metric B change from 5 to 10 (hence R_B=2), then we cannot say metric A is more sensitive than metric B because within A's range [1,3] corresponds to 2% sample quality change while [5,10] corresponds to 50% sample quality change. In short, although the authors found this issue during comparing them, I do not think the solution is good enough, which corresponds to the central empirical findings of this paper.\n- The way that adversarial perturbations are constructed may lead to results that are not fair for FID. We may similarly construct adversarial examples to fool WaM by focusing the change more on third and higher moments without changing first two moments too much? Also neither adversarial constructions correspond to practical settings for evaluating deep generative models. So results are not convincing enough for demonstrating the practical value and the practical benefit is not clear. It seems to me that a better evaluation metric for generative model is one that correlates better with human perception? For example, we often observe that a set of samples look quite realistic but have a bad FID score and vice versa. So experiments about the alignment of WaM score and human perception may be needed to verify if this leads to a better metric. Robustness to some noisy perturbations may not be a central or even desirable property for a good evaluation metric, since detecting such perturbation may also be desirable in some scenatio.\n\n---------------After rebuttal----------------------- Thank you for the response on my feedback. Some of my major concerns still remain (e.g. the justification of using ratios when comparing different metrics with different or unbounded scales, the fairness of the empirical evaluation still seems not convincing enough to me). Thus I tend to keep my original rating.", "summary_of_the_paper": "This paper generalizes the widely-used FID metric for image generation evaluation by fitting a mixture of Gaussians instead of a single Gaussian on the extracted features. The advantage of the proposed approach is it removes the unrealistic assumption of FID that the extracted features from some encoder networks (such as Inception-v3) are approximately Gaussian. The consequence of changing to GMM is that calculating the 2-wasserstein distance now requires a relaxation/approximation, which may result in approximation error and more computation. Empirically, the paper demonstrate that the proposed metric WaM may be less sensitive to noise perturbations.", "summary_of_the_review": "This paper propose a simple yet interesting generalization to FID by leveraging recent advances on approximating wasserstein distance between GMMs. However, as a purely empirical paper, I think there are some major flaws/issues in the evaluation procedure and thus the empirical study is not convincing enough to demonstrate the pratical advantage brought by the proposed approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new evaluating metric for assessing the quality of model-generated images, that aims to correct some of the problems with the popular FID metric. The reviewers acknowledge the importance of this problem, but do not find the empirical evaluation convincing. In particular, they highlight the following issues
* Comparing FID and the new metric on examples that are adversarially selected against FID does not provide a fair comparison.
* The methods are compared on images of bad quality (FID > 25) that are therefore not informative.
* The comparison against existing techniques is incomplete
* The reviewers raise concerns about how the comparison is done quantitatively

The reviewers are not sufficiently convinced by the author response regarding these issues. I therefore recommend not accepting the paper.
Decision: Reject