Evaluating generative networks using Gaussian mixtures of image features

1 INTRODUCTION . Generative networks , such as generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014a ) and variational autoencoders ( Kingma & Welling , 2013 ) , model distributions implicitly by trying to learn a map from a simple distribution , such as a Gaussian , to the desired target distribution . Using generative networks , one can generate new images ( Brock et al. , 2018 ; Karras et al. , 2019a ; b ; 2017 ; Kingma & Welling , 2013 ) , superresolve images ( Ledig et al. , 2017 ; Wang et al. , 2018 ) , solve inverse problems ( Bora et al. , 2017 ) , and perform a host of image-to-image translation tasks ( Isola et al. , 2017 ; Zhu et al. , 2017 ; 2016 ) . However , the high dimensionality of an image distribution makes it difficult to model explicitly , that is , to estimate the moments of the distribution via some parameterization . Just estimating the covariance of a distribution requires p ( p+1 ) 2 parameters , where p is the feature dimension . For this reason , modelling distributions implicitly , using transformations of simple distributions , can be useful for high dimensional data . Since the generator network is typically nonlinear , the explicit form of the generated distribution is not known . Nonetheless , these generative models allow one to sample from the learned distribution . Because we only have access to samples from these generative networks , instead of explicit probability density functions , evaluating their performance can be difficult . As such , several ways of evaluating the quality of the samples drawn from generative networks ( Borji , 2019 ) have been proposed , the most popular of which is the Fréchet Inception distance ( FID ) ( Heusel et al. , 2017 ) . FID fits Gaussian distributions to features extracted from a set of a real images and a set of GANgenerated images . The features are typically extracted using the Inception-v3 classifier ( Szegedy et al. , 2016a ) . These two distributions are then compared using the 2-Wasserstein ( Villani , 2009 ; 2003 ) metric . While FID has demonstrated its utility in providing a computationally efficient metric for assessing the quality of GAN-generated images , closer examination reveals that the fundamental assumption of the FID method—namely , that the underlying feature distributions are Gaussian—is invalid . A more accurate model of the underlying features will capture a more comprehensive and informative assessment of GAN quality . In this paper , we first show that the features used to calculate FID are not Gaussian , violating the main assumption in FID ( Section 3 ) . The 2-Wasserstein metric , which FID uses , can not be extended past Gaussians easily because it is typically computationally intractable and does not have closed formed solutions for many families of distributions . Moreover , FID is only capturing the first two moments of the feature distribution and completely ignores all information present in the higher order moments . Missing this information biases FID toward artificially low values , an undesirable property for a performance metric . Thus , we propose using a Gaussian mixture model ( GMM ) ( McLachlan & Peel , 2000 ) for the features instead , because GMMs can model more complex distributions and capture higher order moments . GMMs are estimated efficiently and there exists a Wasserstein-type metric for GMMs ( Delon & Desolneux , 2020 ) ( Section 4 ) which allows us to generalize FID . We use this to develop our generative model evaluation metric , WaM . We provide code for the community to use WaM at ( link will be added after acceptance ) . Finally , we show that WaM is not as sensitive to visually imperceptible noise as FID ( Section 5 ) . Since GMMs can capture more information than Gaussians , WaM more accurately identifies differences between sets of images and avoids the low score bias of FID . We therefore reduce the issue of FID being overly sensitive to various noise perturbations ( Borji , 2019 ) by modelling more information in the feature distributions . We test perturbation sensitivity using additive isotropic Gaussian noise and perturbed images which specifically attempt to increase FID using backpropagation ( Mathiasen & Hvilshøj , 2020b ) . The ability of WaM to model more information in the feature distribution makes it a better evaluation metric for generative networks . 2 RELATED WORK . 2.1 WASSERSTEIN DISTANCE . There are several ways to define a distance metric between probability distributions . A popular metric from optimal transport ( Villani , 2003 ; 2009 ) is the p-Wasserstein metric . We first are given a Polish metric space X with a metric d. Given p ∈ ( 0 , ∞ ) and two distributions P and Q on X with finite moments of order p , the p-Wasserstein metric is given by Wp ( P , Q ) = ( inf γ ∫ X×X d ( x , y ) pdγ ( x , y ) ) 1 p where the infimum is taken over all joint distributions γ of P and Q . Different values of p yield different metric properties ; in image processing , the 1-Wasserstein distance on discrete spaces is used and called the earth mover distance ( Rubner et al. , 2000 ) . The 2-Wasserstein metric ( Dowson & Landau , 1982 ; Olkin & Pukelsheim , 1982 ) is often used when comparing Gaussians since there exists a closed form solution for W2 ( N ( µ1 , Σ1 ) , N ( µ2 , Σ2 ) ) = ∥µ1 − µ2∥22 + Tr ( Σ1 +Σ2 − 2 ( Σ 1 2 1 Σ2Σ 1 2 1 ) 1 2 ) , ( 1 ) as is used to calculate the Fréchet Inception distance . 2.2 FID AND VARIANTS . The Fréchet Inception distance ( FID ) ( Heusel et al. , 2017 ) is a performance measure typically used to evaluate generative networks . In order to compare two sets of images , X1 and X2 , they are featurized using the penultimate layer of the Inception-v3 network to get sets of features F1 and F2 . For ImageNet data , this reduces the dimension of the data from 3× 224× 224 = 150,528 to 2048 . At this point , Heusel et al . assume that these features are Gaussian and use Equation ( 1 ) to obtain a distance between them . There are several ways that FID has been improved . One work has shown that FID is biased ( Chong & Forsyth , 2020 ) , especially when it is computed using a small number of samples . They show that FID is unbiased asymptotically and show how to estimate the asymptotic value of FID to obtain an unbiased estimate . Others have used a network different from Inception-v3 to evaluate data that is not from ImageNet ; for example , a LeNet-like ( LeCun et al. , 1989 ) feature extractor can be used for MNIST . In this work we focus on several different ImageNet feature extractors because of their widespread use . Modelling ImageNet features has been improved due to a conditional version of FID ( Soloveitchik et al. , 2021 ) which extends FID to conditional distributions , and a class-aware Fréchet distance ( Liu et al. , 2018 ) which models the classes with GMMs . In this work , we do not consider conditional versions of FID , but our work can be extended to fit such a formulation in a straightforward manner . Moreover , we use GMMs over the feature space rather than one component per class as is done in the class-aware Fréchet distance . Another metric related to our proposed metric is called WInD ( Dimitrakopoulos et al. , 2020 ) . WInD uses a combination of the 1-Wasserstein metric on discrete spaces with the 2-Wasserstein metric on Rp . For this reason , it is not a p-Wasserstein metric in Rp or between GMMs . For example , if P and Q are a mixture of Dirac delta functions then the WInD distance between them becomes the 1-Wasserstein distance . However , if P and Q are Gaussians , then the WInD distance between them becomes the 2-Wasserstein distance . Moreover , if P and Q are arbitrary GMMs , the relationship between WInD and the p-Wasserstein metrics is not clear . This means that WInD can alternate between the 1-Wasserstein and 2-Wasserstein distance depending on the input distributions . In this paper , we focus on using a metric which closely follows the 2-Wasserstein distance as is currently done with FID . 2.3 MW2 A closed form solution for the 2-Wasserstein distance between GMMs is not known . This is because the joint distribution between two GMMs is not necessarily a GMM . However , if we restrict ourselves to the relaxed problem of only considering joint distributions over GMMs , then the resulting 2-Wasserstein distance of this new space is known . The restricted space of GMMs is quite large since GMMs can approximate any distribution to arbitrary precision , given enough mixture components . So given two GMMs , P and Q , we can calculate MW22 ( P , Q ) = inf γ ∫ X×X d ( x , y ) 2dγ ( x , y ) where the infimum is over all joint distributions γ which are also GMMs . Constraining the class of joint distributions is a relaxation that has been done before ( Bion-Nadal et al. , 2019 ) due to the difficulty of considering arbitrary joint distributions . This metric , MW2 , appears in a few different sources in the literature ( Chen et al. , 2016 ; 2018 ; 2019 ) and has been studied theoretically ( Delon & Desolneux , 2020 ) ; recently , implementations of this quantity have emerged.1 The practical formulation of this problem is done as follows . Let P = ∑K0 i=1 πiνi and Q =∑K1 j=1 αjµj be two GMMs with Gaussians νi , µj for i ∈ { 1 , . . . , K0 } , j ∈ { 1 , . . . , K1 } . Then , we have that MW22 ( P , Q ) = min γ ∑ ij γijW22 ( νi , µj ) ( 2 ) where γ is taken to be the joint distribution over the two categorical distributions [ π1 . . . πK0 ] and [ α1 . . . αK1 ] ; hence , γ in this case is actually a matrix . Thus , MW2 can be implemented as a discrete optimal transport plan and efficient software exists to compute this ( Flamary et al. , 2021 ) . MW2 is a great candidate for modelling the distance between GMMs for several reasons ; most importantly , it is an actual distance metric . Since we are restricting the joint distribution to be a GMM , we see that MW2 must be greater than or equal to the 2-Wasserstein distance between two GMMs . Moreover , MW2 clearly approximates the 2-Wasserstein metric ; Delon & Desolneux derive bounds showing how close MW2 is to W2 . It is also computationally efficient to compute because it can be formulated as a discrete optimal transport problem , making it practical . The strong theoretical properties and computational efficiency of MW2 make it a prime candidate to calculate the distance between GMMs . 1https : //github.com/judelo/gmmot ResNet-18 feature # 172 ResNet-50 feature # 559 ResNeXt-101 ( 32×8d ) feature # 1653 Inception-v3 feature # 1216 3 INCEPTION-V3 HAS NON-GAUSSIAN FEATURES ON IMAGENET .

Summary: This paper targets to a new evaluation metric for the performance of generative models given a set of real images and a set of fake images. The authors show that Inception-v3 features of the imageNet dataset are not Gaussian and remedy this issue by modeling image features using Gaussian mixture models (GMMs) and formulating the distribution distance between two GMMs by MW2. They demonstrate their metric is less sensitive than FID against image perturbations.