Using Skew to Assess the Quality of GAN-generated Image Features
Authors: Lorenzo Luzi
Helen Jenne
Carlos Ortiz Marrero
Ryan Murray
Venue: Accepted by TMLR ()

Abstract: The rapid advancement of Generative Adversarial Networks (GANs) necessitates the need to robustly evaluate these models. Among the established evaluation criteria, the Fréchet Inception Distance (FID) has been widely adopted due to its conceptual simplicity, fast computation time, and strong correlation with human perception. However, FID has inherent limitations, mainly stemming from its assumption that feature embeddings follow a Gaussian distribution, and therefore can be defined by their first two moments. As this does not hold in practice, in this paper we explore the importance of third-moments in image feature data and use this information to define a new measure, which we call the Skew Inception Distance (SID). We prove that SID is a pseudometric on probability distributions, show how it extends FID, and present a practical method for its computation. Our numerical experiments support that SID either tracks with FID or, in some cases, aligns more closely with human perception when evaluating image features of ImageNet data.  Our work also shows that principal component analysis can be used to speed up the computation time of both FID and SID. Although we focus on using SID on image features for GAN evaluation, SID is applicable much more generally, including for the evaluation of other generative models.

Published in Transactions on Machine Learning Research (04/2024)
Using Skew to Assess the Quality of GAN-generated Image Features
Lorenzo Luzi"
enzo rice.
Rice University
Pacific Nortwest National Laboratory
Helen Jenne' Pacific Northwest National Laboratory
helen.jenne @ pmml.
Carlos Ortiz Marrero
prtbos.orizmorreemepm.....g
Pacific Northwest National Laboratory North Carolina State University
Ryan Murray North Carolina State University
wwmurray@ncsu.cc
Reviewed on OpenReview:
Abstract
The rapid advancement of Generative Adversarial Networks (GANs) necessitates the need to ro- bustly evaluate these models. Among the established evaluation criteria, the Fréchet Inception Dis- tance (FID) has been widely adopted due to its conceptual simplicity, fast computation time, and strong correlation with human perception. However, FID has inherent limitations, mainly stemming from its assumption that feature embeddings follow a Gaussian distribution, and therefore can be defined by their first two moments. As this does not hold in practice, in this paper we explore the importance of third-moments in image feature data and use this information define new measure, which we call the Skew Inception Distance (SID). We prove that SID is a pseudometric on prob- ability distributions, show how extends FID, and present practical method for its computation. Our numerical experiments support that SID either tracks with FID in some cases, aligns more closely with human perception when evaluating image features of ImageNet data. Our work also shows that principal component analysis can be used to speed the computation time of both FID and SID. Although we focus on using SID on image features for GAN evaluation, SID is applicable much more generally, including for the evaluation of other generative models.
Introduction
The goal of generative modeling is to learn a data distribution that generates novel samples which are indistinguishable from training data. Generative Adversarial Networks (GANs) (Goodfellow et al., 2014)) for image generation are widely known generative models that have seen enormous progress in recent years. As these models continue to evolve, the need for robust and informative evaluation metrics is increasingly important benchmark different models.
Evaluating the quality of artificially generated samples is multifaceted challenge. Ideally, a generative model should produce samples that not only appear to be from the training dataset, but also incorporate its full range of variations, Quantifying this is far from straightforward, leading to an active research domain focused on GAN evalution. However, progress in this area is noticeably slower than the rapid advancements the development of GAN models. In fact, one of the pioneering metrics in GAN evaluation, the Fréchet Inception Distance (FID) (Heusel et al., 2017), continues to well-regardes tool for GAN evaluation.
Thhese authors contributed equally work.

---

Published in Transactions on Machine Learning Research (04/2024)
FID separately embeds the generated and training samples into a feature space via convolutional neural network (CNN), with Inception-v3 being a common choice. These feature representations are then modeled as multivariate Gaussian distributions. The actual FID score is determined by measuring the distance between the two Gaussian distributions using the Fréchet distance (also known as the 2- 2-Wasserstein distance). has been well established that FID correlates well with human judgement, is sensitive to small changes in the distribution (e.g. blurring or adding small artifacts to images), can detect intraclass mode collapse, and is fast to compute (Heusel et al., 2017; Lucie et al., 2018).
That being said, FID is not without shortcomings. Common critiques of FID include its high bias (typically requiring at least 50k samples for its computation), its sensitivity to the choice of CNN for feature extraction, and its presentation as a single numerical value without distinguishing between model fidelity and diversity. Over the years, other metrics have been proposed with their own trade-offs, such as Kernel Inception Distance (Sajjadi et al., 2018). Intrinsic Multi- Scale Distance (IMD) (Tsitsulin et al., 2019), Class-Awar Aware Latent Distance (Swisher, 2022), and precision/recall-ye type metrics (Sajjadi et al., 2018; Kynkäänniemi al., 2019; Naeem et al., 2020). For a more detailed discussion of related work, see Appendix A.
Our investigation is primarily concerned with the assumption of Gaussian characteristics in the embedded features and FID's s failure to account for higher moments of these distributions. It is known that Inception-v3 features, among other classifier features, are skewed (Luzi et al., 2023). In this work, we explore the importance of third moment data for evaluating GAN performance. We introduce a novel measure, Skew Inception Distance (SID), which builds upon the foundation of FID but extends its scope to incorporate third-moment data, effectively accounting for skewness.
1.1 Overview and summary of contributions
We prove theoretical properties of SID in Section 2, in particular describing its properties as a metric. We review the relationship between probability distributions and their moments in Section 2.1, in order to define a metric on the space of moments and obtain a mesric on the space of distributions. Section 3, we provide way to compute skewness by reducing the dimension of the Inception-v3 embeddings using principal component analysis (PCA) and provide two experiments showing that reducing the dimension of the embeddings does not lose important information. This result may be of independent interest in other problems that often rely CNN embeddings, such few-shot learning and out-of-distribuiin detection (Hu et al., 2021; Lee et al., 2018). Finally, in Section 4 we present empirical evaluations of SID, showing that it either behaves similarly to FID or more closely aligns with human perception.
2 Extending FID to include skewness
To compute FID, the set of real images and the set of generated images are featurized using the penultimate layer of Inception-v3, and the means μl and μl and covariance matrices Σ₁ and Eg are estimated assuming that these features are Gaussian. Then FID given by
|| 12112 + Tr (>) + DD - 2/
In this paper, we seek to add a third term to FID to account for skewness. The challenges of this are first that we want to add skewness term to Equation (2) in such a way that the resulting measure is a metric and second that the scale of the skewness term makes sense relative to the other terms.
In order to address these challenges we compute the coskewness tensor, B defined by
$ [[X;
'Although Equation (2) is the typical definition for FID (see c.g. Heusel et al. (2017)), note that Fréchet distance between two multivariate Gaussian distributions [i,,, Σn) and MiH2, M2) is defined as
|| r2l 3 Tr Σ1 E1+ - T ///
The equivalence of Equations (2) and (1) obvious when F. and F2g commute. The equations are equivalent even when E2 and 522 do not commute, due the fact that the eigenvalues of 2 details).
1/2 and E, Flg are identical. Wa & Koelzer (2022, Section 2.1.2) for further
2

---

Published in Transactions on Machine Learning Research (04/2024)
where X" = (Xf, X;)) is defined by x'=x=* = (i) and we compare the coskewness tensors of the sets of featurized real and generated images using the Frobenius norm of their element-wise cube root.
In order to prove that the resulting equation defines a pseudometric, we start by defining metric on the space of moments of a distribution, which as we will see, allows us to obtain pseudometrie on the space of distributions. First, we need formalize the relationship between probability distributions and their moments. The mathematical analysis of this relationship has been classically called the moment problem (Shohat & Tamarkin, 1950).
2.1 Theoretical set-up
The relationship between probability distributions and their moments is often utilized because metrics in families of probability distributions are more complex compared to metrics on subsets of R". Even popular metrics, such as the Wasserstein distance, can be challenging to compute for many families of distributions. Consequently, tractable metrics on probability distributions start with a metric on the original probability space and reduce it to a metric on the moments or parameters of the distributions, which typically have closed-form solutions (e.g.. Hellinger (Gibbs & Su, 2002), Bhattacharyya (Bhattacharyya, 1946), and Wasserstein (Villani, 2021) distances).
We provide several examples of distributions that can be identified with their moments, i.c., given a family of dis- tributions P. indexed by a, we can construct a bijection f that maps Pa to its moments M,, Furthermore, this mapping is often a homeomorphism (meaning that both f and its inverse are continuous) with the appropriate choice of metrics. When f a homeomorphism, we can inherit the topological properties of M, (Munkres, 2000). This means that in many ways, we can work in the space of the moments M.,, which is typically a subset of IR", instead of directly in the distribution space Pa- Not only that, but if the relationship is homeomorphism, then we reduce the infnnteediimeniioall distribution space into finite-dimensional vector space which has tractable methods for comput- ing metrics. Below are some examples. The proofs of the claims made, including the explicit constructions of the homeomorphisms, are left to Appendix B.
Example (The univariate exponential distribution). The univariate exponential distribution is characterized by the following definition (Casella & Berger, 2021):
(x) = Aexp(- -x)) for A E (0, oo).
(3)
There exists a bijection from the distributions {px (0, ز) to the parameter space (0, 00), making exponential distributions completely characterized by their mean 1. Moreover, this bijection homeomorphism between ((0, I -l) and (P,, || (1(
Example 2 (The multivariate Gaussian). The multivariate Gaussian distribution is characterized by the following definition (Mardia et al., 1979):
(ax(=)= 12 exx ( (-- -111 11) 11)
(4)
for mean vector μ IP and covariance matrix Σ in the space of p x p real, positive-definite matrices S The function ʃ that maps P to (j,, Σ) is an isometry between the space of multivariate Gaussian distributions equipped with the 2- Wassersstei metric and the space (n,, Σ) equipped with the metric defined by
d( d(mi E1), (12 EM), En)) -1 12l1 + Tr [1 + 2e 2 2V
It is important to note that when comparing the moments of two distributions, they typically come from the same family of distributions. Let us consider two homeomorphisms: a homeomorphism f from the set of all normal distributions Pn in IR to its moments IR x (0, oo) and a homeomorphism f' from the set of all uniform distributions P., in R to its moments x (0, oo). If we have E P,, and & E Pu. we must have that p ل q because they are different distributions: however, we may have that their moments match exactly. This is because f and f' are different homeomorphisms operating on different spaces, and thus we must take care consider not only the matching of moments but also the underlying distributions they represent.
3

---

Published in Transactions on Machine Learning Research (04/2024)
2.2
Defining SFD: A metric on the first three moments of probability distributions
Let IP denote set of probability measures P that homeomorphic to space M equipped with metric des. which is the space of moments (or parameters) of the distribution. Denoting the homeomorphism as f -> M, we can define metric by
dr(q.p) p) _ (
(5)
Essentially, we use f to push the points p. y to the moment space and then apply our metric there. Most importantly, calculating (4) and (p) is very easy and tractable: we simply calculate the moments in the usual statistical way, for example, using maximum likelihood estimation (Casella Berger, 2021).
Proposition 1. Equation (5) defines a metric dg.
Proof. It is immediate that dp is non-negative, symmetric, and satisfies the triangle incquality. The fact that de is definite follows from the injectivity of f.
As clear from the proof, rather than requiring homeomorphism one could consider an injective f instead at the cost of losing the topologiaal equivalence of P and M. However, this is not necessary in many cases desired homeomorphism exists.
In the case of numerous distributions, a complete description requires multiple moments, so we view our space MI as a product of several spaces: M = M4, ... x Mm. Moreover, each space M, may have its own metric dx, Therefore, we will need consider which metric product space we will use. For any p E [1, oc), we can define product metric (Mendelson, 1990) as
m da (zi, y.).
drg ((II...., Rm); (y1...... Un))
Since all norms on finite-dimensional vector spaces are equivalent (Royden & Fitzpatrick, 1988), we will just pick p 2. Therefore, given each moment space M4, we construct metric on that space dxs, and together construct metric da on the whole moment space M which we use to define a metric (Equation (5)) on the distributions themselves.
We are now ready to define our metric, which we call SFD (Skew Fréchet Distance), using the first three moments of probability distribution:
(1,, St), S1), (12, Ez, S2)) || 2211 Tr E1 + Σ2 2 2VEE + 1//5/
V/52
S2
(6)
Here, S is the element-wise cube root? of the skew 3-tensor S which ensures that each term of SFD has the same units. (The coskewness 3-tensor S has cubed units, while the first and second terms in SFD have linear units before they are squared).
Note that SFD is a generalization of the 2- 2-sssserstein distance for Gaussians: since in the Gaussian case, SFD between two Gaussian the same as the 2- Wasserstein distance.
The first and second terms of SFD define metrics on R" and S. (Givens & Shortt, 1984). So it remains show that the third term defines metric. We actually prove more general result.
Theorem 1. Let X(n1...., ny; C) be the set of all complex-valued, r-tensors with shape (Ri...., ny). Let C be any invertible function. Then for A. B X(nI,...... ny; C) have that
d A. B)
Σ
la(A)) - (B.)
1E (1....
defines metric d.
Too knowledge, there is computationally efficient 3-tensor cube
4

---

Published in Transactions on Machine Learning Research (04/2024)
We postpone the proof to Appendix C.
The metric d looks very similar to Frobenius norm, but it does not induce norm. If we choose a to be the identity map, then we do indeed recover the Frobenius norm. However, there are many choices of a which make it so that d does not induce a norm; for example, if ((x) = we see that the homogeneity property norms violated.
For SFD, we choose a -> R defined by a(x) f/2 and see that SFD a proper metric with the same units for each term. Additionally, we transform the skew term || S - S/ by s' = (+)) m 2 using sigmoid 0. SFD is still a metric since these are invertible transformations which preserve non-negativity. We do this to ensure that the skew term has comparable scaling to the covariance and mean terms. We saw that values of a = 10, 000 and m = 150 seemed to work well in the examples we tested.
If distribution is homeomorphic to its first three moments, then SFD defines metric on that family of distributions. However, if this is false and distribution is homeomorphic more than three moments, then SFD instead defines pseudometric. More formally,
Proposition 2. Suppose defines a family of distributions that are defined by their first three moments, i.e., for all P1. P1.P2 P2 P. we have that (ml Σ,, (122 Σ²,S2) S2) if and only p1 P2- Then, SFD defines proper metric. If there are additional moments which are needed for the equality to hold, then SFD defines pseudometric.
Proof. In the case where IP defines a family of distributions which are defined by their first three moments, SFD defines metric because it is product metric (Royden & Fitzpatrick, 1988), and by Theorem 1, each term in the product metric.
Now suppose P1 and p2 are defined by their moments (n, Σ, S, X1) and (M, Σ, E, X2X2): all their moments are the same except for X and X2 (which can be any order), and X1 * X2. Since SFD is zero in this case, SFD is not strictly definite and, therefore, SFD cannot be metric. However, SFD defines a product over pseudometries and therefore is a pseudometric itself (Freiwald, 2014).
The following example gives instance where SFD is pseudometric.
Example 3. Let P1 and p2 be univariate Gaussian mixtures with two components. The parameters P1 are Pl /22 0, 01 02 1, and ι x2 0.5; that is the first GMM standard normal Gaussian. The parameters of p2 are μ1 #22 = 0, 01 72' 02 v2, ग 2. and TT2 - Both of these distributions have mean O, variance 1, and skew meaning that SFD between these two distributions is zero even though they have non-matching higher-order moments.
While this limitation of SFD, it can still be used to compare distributions even when it is a pseudometric. For example, the Fréchet distance is commonly used distributions that Gaussian.
Although our pscudometric SFD is quite general, for the remainder of the paper, will apply SFD to Inception-v3 embeddings and refer to this specific case as SID.
3 Using dimensionality reduction to compute SID
compute the coskewness tensor in practice, we need to reduce the dimension of the Inception-v3 embeddings, as these embeddings are 2048-dimensional. This requires the coskewness tensor to be 2048 x 2048 2048, which is 64 GB using 32-bit precision. That is just to store one of these tensors, let alone store and manipulate of them! Storing tensors of this size the GPU is out of the question.
In addition, computing the coskewness tensor is the slowest of the three terms; the operation scales as O(d)), where d is the dimensionality of the reduced feature space. In comparison, FID can be computed in O + m²² time, where m << d (Mathiasen & Hvilshoj. 2020). (Although this requires observing that it suffices to compute the eigenvalues of E, Eg; many FID implementations compute Tr(vE,E))) using the s scipy lilallll..... sqrtm function, which requires computing the Schur decomposition of Σg and takes ((d)) time). Because of the memory and computational obstacles of SID, we reduce the dimension of the Inception-v3 embeddings.
5

---

Published in Transactions on Machine Learning Research (04/2024)
Table 1: PCA reduces dimension the features while still maintaining important information. This table demonstrates the classification performance of Inception-v3 on ImageNet using features that have been reduced in dimension with PCA and then linearly projected back into their 2018-dimensional space. We see that the dimensionality reduction does not significantly affect classification performance, even for large reductions, implying that important information in the features is preserved.
Dimensionality
Top-1 Train Acc
Top-5 Train Acc
Top-1 Val Ace
Top-5 Val Acc
2048 (baseline)
91.78%
99.16%
77.21%
93.53%
1024
(50%)
90.77%
98.95%
76.24%
92.83%
512
(25%)
88.48%
98.21%
74.15%
90.82%
256
(12.5%)
81.59%
95.12%
68.40%
85.55%
In order to reduce the dimensionality of the Inception-v3 embeddings, we apply principal component analysis (PCA). PCA is a widely used dimensionality reduction method that projects the data onto a lower-dimensional subspace in such way that maximizes the variance preserved. Because PCA maximizes the preserved variance, is a natural choice for dimension reduction in our case (the intuition here is that, because FID is a function of the variances, if we assume that our variances will decay rapidly, truncating tail variances will not change FID by very much). After applying PCA, we need to normalize the covariance and skew terms. Because we center data matrix, we do not modify the mean with the dimensionality reduction, just the covariance and skew.
Letting X E R=xp denote our centered data matrix (where n > p), we will use the following transformation of X to normalize the FID values for different values of k:
Σ D i XV.. Σ Di]kk
Y
Here, V1 is the matrix consisting of the first ls right singular vectors of X, D is the matrix of singular values, and D is the k x & truncated matrix of singular values. We discuss this normalization further in Appendix D.
We emphasize that for a fixed k, we are transforming both sets of features in the same way so that they are comparable.
To summarize, to compute SID we first compute the Inception-v3 embeddings X, then compute the singular value decomposition of X in order to apply the transformation in Equation (7). After the dimensionality of the embeddings has been reduced, we compute the FID terms and the coskewness tensor.
3.1 The effect of PCA on FID
Before using the reduced-dimensiee features to compute SID, we needed to ensure that PCA reduces the dimension of the features while still maintaining important information. We investigate this question in two ways. First, since Inception-v3 is a classifier, natural question is how the classifier performance deteriorates we reduce the feature dimension. To implement this experiment, first calculate the PCA transform for à {2048, 1024, 512, 256} on the training data. Then, we extracted Inception-v3 features from the training and validation datasets. Next, we reduced the dimension with the precalculated transform and then projected back to the 2048-dimensional space; since the PCA transform is a semi-orthogonal matrix Vx. we simply multiplied by the transpose V. Finally, we sent these features to the final layer of Inception-v3 to calculate training and validation accuracies. The results are summarized in Table We see that even with significant dimensionality reduction (12.5%), there is at most a 10% difference in accuracy, which implies that using PCA reduce the dimension to 256 does not significantly affect the classification power of the embeddings.
Furthermore, we investigated how dimension reduction affected the behavior of FID on common experiments used to compare GAN metrics. These experiments involve corrupting an image and studying how the GAN metric behaves with respect to a parameter controlling the level of corruption. Specifically, we used the following corruptions: Gaus- sian noise, salt and pepper noise, Gaussian blur, and adding rectangles randomly chosen locations (Heusel et al., 2017). We describe these experiments more detail in Section 4.2. In all experiments, the dimension reduction affected the value of FID slightly but not its behavior; examples are shown in Figure 1.

---

Published in Transactions on Machine Learning Research (04/2024)
Salt and Pepper Noise
Added Gaussian Noise
d 5512
d 5512
120
d =256
=256
edde: d =128
600
=128
100
-64
64
=32
-32
80
-66
166
FID
FID
400
60
40
200
20
0
0
0.02
0.04
0.06
0.08
0.1
10
15
20
Gaussian standard deviation
Salt and Pepper noise p (%)
Rectangular Occlusions
Gaussian Blur
d 552
d 5512
-- d =256
80
d 255
25
rde d =128
d.. d 112
- d -64
d 644
20
=*- 322
60
d -32
d 16
d 166
FID
15
40
10
0.2
0.4
0.6
0.8
5
10
15
20
Gausian kernel
Erased percentage (%)
Figure 1: FID behaves similarly even when we project features down to lower dimensions via PCA. We show four types of distortions are applied to ImageNet images: added Gaussian noise, salt and pepper noise, Gaussian blur, and rectangular occlusions. For the Gaussian blur, we use kernel size of S. In each plot, FID is shown as function of the parameter controlling distortion, after reducing the embeddings to dimension E {512, 256, 128, 64, 32, 16). All experiments are done using Inception-v3 feature extractor the entire (50K) ImageNet validation set.
3.2 Time improvement for SID
Table 2 demonstrates that dimensionality reduction leads to significant time and memory savings when computing skewness. These savings are so substantial that the skew calculation can be computed completely on GPU.
Having established that using PCA reduce the dimensionality of Inception-v3 features is reasonable and has com- putational and memory benefits, we turn attention to the behavior of SID performance measure.

---

Published in Transactions on Machine Learning Research (04/2024)
Table Calculating skewness on the GPU is computationally efficient, especially when reducing the feature dimensionality. There is an order of magnitude difference in CPU and GPU times. (+) would not fit completely on the GPU but can be calculated on GPU using mini-batches; however, batch and mini-batch algorithms are not comparable due to intrinsic differences.
Dimensionality
CPU time GPU time
Memory
2048
(baseline)
886.9s
64 GB
1024
(50%)
161.1s
9.1s
8 GB
512
(25%)
35.5s
0,8s
GB
256
(12.5%)
4.66s
0.02s
128 MB
4 SID Experiments
4.1 Features are skewed even after PCA
It is known that Inception-v3 features, among other classifier features, are skewed (Luzi et al., 2023). However, it is not immediately clear whether features are skewed after dimensionality reduction using PCA. In order to test this, we applied standard skewness tests to the camensioo-reducad data. In particular, we performed the Mardia skewness test (Mardia, 1970) on Inception-v3 multivariate 2048- -ineediiensional features with a significance level of 0. 0.1% and the data failed the test at dimensionality reductions of 1024, 512, 256, 128, 64, 32, and 16. We also performed Bolmogorov-Sminon hypothesis tests' (Dodge, 2008) at each marginal with significance values of 0.1% and found that 100% of the marginals failed the tests. This demonstrates that even with extreme dimensionality reduction with PCA, skewness persists in the data. We repeated the experiment with ResNet- 18 features and obtained the same result: all tests failed with significance values of 0.1%.
4.2 Comparing SID and FID
In order to compare SID and FID we studied how these measures are affected by the corruptions introduced in Sec- tion 3.1:
1. Gaussian noise: We added Gaussian noise with E {0, 0.001, 0.005, 0.01, 0.1).
2. Salt and pepper noise: We changed proportion of the pixels in the image to black or white (with equal probability) for E {0, 0.5%, 1.0%, 1.5%.... 20%)..
Gaussian blur: We convolved the image with a Gaussian kernel with standard deviation o E
(0.1, 0.2, ...))....,..)).
4. Adding black rectangles occlusions: We added five rectangles at randomly chosen locations, where the scale increases with scale parameter E (1%, 2%, .7. 20%)..
In these experiments, we found that the skew term (and consequently SID) often behaved similarly to FID.
In other work, larger values were chosen for the parameters controlling these distortions (see Heusel et al. (2017)). But looking at examples of images with these levels of corruption shows that for most values of these parameters, the difference between the corrupted image and the original image is easily perceptible by a human. This is why in order to investigate what is happening at the transition point between noise that is human perceptible, we focused on smaller values of these parameters. Here we see a difference between the skew term and the FID. For example, in Figure 2, which shows the FID and skew term as Gaussian noise is added with E {0.01, 0.015, 0.02, 0.03, 0.04, 0.05, 0.06). we see that the skew does not pick up any difference until the noise becomes perceivable. Once the noise becomes perceivable, the skew increases faster than the covariance and mean terms. The Gaussian blur experiment with e {0.0, 0.1, 0.2,.... 1.0} showed similar results and is shown in Appendix E. Other experiments showed that SID and FID behaved similarly, as shown in Figure 4 in Appendix E.
While in this section, we focused on results for SID using Inception-v3 v3 features on ImageNet data, using different models for feature extraction (namely, ResNet-18, ResNet-50, and ResNeXt-101 (32x8d)) and differ-
data, we also studied SID
'The Kolmogorov-Sminng Smiroov hypothesis probability distribution (the marginal distributions in case) differs from hypothesized distribution, Gaussian distribution.
nonparametric method for evaluating the gondness-of-fit. assesses whether an underlying
8

---

Published in Transactions on Machine Learning Research (04/2024)
FID and the skew term with the added Gaussian noise experiment
6
FID Skew
4
Skew term
FID
0.5
0
0
0
0.01
0.015
0.02
0.03
0.04
0.05
0.06
Added Gaussian noise a
Ground Truth
== 0.0
= 0.01
a 0.015
0.02
0.06
Data + Noise
Noise
Figure 2: Skew tracks with human perception than FID. The noise levels on the images quite low and undetectable until 0.015; yet the FID increases linearly throughout. On the other hand, skew stays low for these undetectable noise levels. These were typical images taken from the 50,000 samples used calculate FID.
ent datasets (CelebA (Liu et al., 2015), FFHQ (Karras et al., 2019), and SVHN (Netzer et al., 2011)). Additionally, to explore SID's applicability different types of generative approaches, studied SID using data from diffusion models, using pretrained diffusion models that were trained on Stanford Cars, CelebA, AFHQ, and the Flowers dataset to generate images. These experiments, presented in Appendices and G, show that SID is robust to the model used for feature extraction and that in all settings SID increases as the noise distortions increase.
5 Conclusion
This paper presents novel metric on distributions of image features that can successfully incorporate third-monne moment data and can be used to evaluate the performance of GANs. Our findings indicate that, particularly when images are subtly distorted, SID can sometimes better match human perception. A notable limitation of SID is its computational complexity, which is Odd)), where d is the dimensionality the features, making dimensionality reduction essen- tial for its calculation. Consequently, the dimensionality reduction we perform estimating SID may not include information that not captured in the largest principal components. Another limitation of our work can be observed in many related works: experiments often struggle faithfully replicate the nuanced distinctions between real and

---

Published in Transactions on Machine Learning Research (04/2024)
GAN-generated images; for instance, GAN images may exhibit multiple types of distortion in the same image. Fur- thermore, assertions regarding alignment with human perception should be substantiated through human studies. The field would greatly benefit from an extensive and standardized repository of experiments designed to evaluate GAN metrics. Until then, is crucial that GAN evaluation metrics are based on a solid theoretical foundation.
The current work does not compare SID to other established metrics, which is an important direction for future study, We conclude by discussing three other directions for future work:
Exploring other formulations of the skew term The subtle differences between SID) and FID have raised the question of how much redundant information the skew term encapsulates. We're considering whether our current formulation of the skew term contributes to this and ex- ploring alternative formulations for skew. For example, we could define a skew term using Mardia skewness (Mardia, 1970) or Kollo skewness (Kollo. 2008). Assuming we could formulate a (pseudo) metric using one of these measures of skewness, it would be interesting to see how it changes the performance of SID.
Exploring other methods for dimensionality reduction
We could also use random, isotropic, Gaussian projections to reduce the dimensionality of our features. To do this, we would simply create matrix U E Rexd, which takes the d-dimensional data and reduces it to k-dimensional data, by sampling from an isotropic Gaussian. This matrix is computationally inexpensive to construct and has several nice statistical properties. For example, Us is likely to be a projection matrix and unitary if l: is small enough (Wegner, 2021).
Identify other applications
GAN evaluation is not the only area where people features extracted from backbone architecture and assume that these features are normally distributed. There is also few-shot learning (see Hu et al. (2021) and Chobola et al. detection (see Lee et al. (2018) and Ahuja et al. (2019)). Both the potential speed-up
(2021)) and of applying PCA to the features and computing skewness could be of interest in these domains.
Broader Impacts
Generative models are powerful tools which offer both positive applications and opportunities for misuse. While this paper focused on the mathematical aspect of generative model evaluation, we recognize the importance of considering potential societal impacts of our work.
Evaluation measures like SID can be used to improve generative models, some of which might be used for nefari- ous purposes, such as generating fake social media profiles to deceive users. Additionally, all measures have their shortcomings, and there may be hidden biases in SID which have yet to be identified. We have not yet explored, for instance, how SID handles biases training data. These are important considerations not only for this work but for the broader field of generative model evaluation.
Acknowledgments
RM was partially supported by the NSF grant 00002904.
and by grant from the Simons Foundation MP-TSM-
References
Nilesh Ahuja, Ibrahima Ndiour, Trushant Kalyanpur, and Omesh Tickoo. Probabilistic modeling of deep features for out-of-distribution and adversarial detection, arXiv preprint 11X7:1199..11886 2019.
Anil Bhattacharyya. On measure of divergence between two multinomial populations. Sankhyā: the Indian Journal of Statistics, pp. 401-406, 1946.
Ali Borji. Pros and cons of GAN evaluation measures: New developments. Understanding, 215: 103329, 2022.
Computer Vision and Image
George Casella and Roger Berger. Statistical inference. Cengage Learning, 2021.
10

---

Published in Transactions on Machine Learning Research (04/2024)
Tomáš Chobola, Daniel Vašata, and Pavel Kordík. Transfer learning based few-shot classification using optimal trans- port mapping from preprocessed latent space of backbone neural network. In AAAI Workshop on Meta-Leerrnin and MetaDL Challenge, pp. 29-37. PMLR, 2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8188-8197, 2020.
Yadolah Dodge. Kolmogorov- Smimno test. The concise encyclopedia of statisties, pp. 283-287, 2008.
Ronald C Freiwald. An introduction to set theory and topology. Washington University in St. Louis, 2014.
Alison Gibbs and Francis Edward Su. On choosing and bounding probability metries. International Statistical Review, 70(3):419-4333 2002.
Clark R Givens and Rae Michael Shortt. A class of Wasserstein metrics for probability distributions. Michigan Mathematical Journal, 31(2)231-240, 1984.
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhardd Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local nash equilibrium. In NIPS, pp. 6626-6637, 2017.
Yuqing Hu, Vincent Gripon, and Stéphane Pateux. Leveraging the feature distribution in transfer-based few-shot learning. In International Conference on Artificial Neural Networks, pp. 487-499. Springer, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A stylc-based generator architecture for generative adversarial networks. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tönu Kollo. Multivariate skewness and kurtosis measures with application in ICA. Journal of Multivariate Analysis, 99(10)2232-2233 23388 2008.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distributt samples and adversarial attacks. Advances in Neural Information Processing Systems, 31, 2018.
Shaohui Liu, Yi Wei, Jiwen Lu, and Jie Zhou, An improved evaluation framework for generative adversarial networks.
arXiv preprint arxiv: 1eXXv:1803.07420 803.07474, 2018
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? A large-scale study. Advances in Neural Information Processing Systems, 31, 2018.
Lorenzo Luzi, Carlos Ortiz Marrero, Nile Wynar, Richard G Baraniuk, and Michael J Henry. Evaluating generative networks using Gaussian mixtures of image features. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 279-288, 2023.
Kanti V Mardia. Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3):519-530, 1970.
KV Mardia, JT Kent, and JM Bibby. Multivariate analysis. Acadelnic Press, Londres, 1979.
Alexander Mathiasen and Frederik Hvilshøj. Backpropagating through Fréchet inception distance. arXiv preprint arXiv:2009.140% 14075, 2020.
11

---

Published in Transactions on Machine Learning Research (04/2024)
Bert Mendelson. Introduction to topology. Courier Corporation, 1990.
James R. Munkres. Topology. Prentice Hall, Inc., 2 edition, January 2000. ISBN 0131816292.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. In International Conference on Machine Learning, pp. 7176-7185. PMLR, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural im- ages with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL
/ww.llllll.//// sttanforr..cc//oouuuuuuurrrr///p htwo200nnooouuuuumbbrrerr pdf.
Halsey Lawrence Royden and Patrick Fitzpatrick. Real analysis, volume 32. Macmillan New York, 1988.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in Neural Information Processing Systems, 31, 2018.
James Alexander Shohat and Jacob David Tamarkin. The problem of moments, volume 1. American Mathematical Society (RI), 1950.
John Andrew Swisher. Improved generative evaluation: Utilizing the manifold hypothesis. Bachelor's thesis, Harvard, 2022.
Anton Tsitsulin, Marina Munkhoeva, Davide Mottin, Panagiotis Karras, Alex Bronstein, Ivan Oseledets, and Em- manuel Müller. The shape of data: Intrinsic distance for data distributions. arXiv preprint 1rXiv:1905.11141, 2019.
K Tung. Flowers Dataset, 2020. URL hwwph:ttp///ce http::::/dooi.ooo /ttttp:://di...r//1.... 791 http://////.....////////////////
Cédric Villani. Topics in optimal transportation, volume 58. American Mathematical Soc., 2021.
Sven-Ake Wegner. Lecture notes on high-dimensional data. arXiv preprint arXiv:2101.05844 2021.
Jiqing Wu and Viktor Koelzer. Sorted eigenvalue comparison dBig: A simple alternative to dFID. In Workshop on Distribution Shifts, 2022. URL http:::/p////c. /ttp:://///eeeveeeeeeeeeeeeee wttpppffiidt....m Ag11mX.
A
Related work
The landscape of GAN evaluation measures has seen substantial growth since FID was introduced 2017. This discussion focuses on evaluation metries most closely related to ours; for comprehensive overview, see Borji (2022).
There relatively few research works that address the assumption that the Inception-v3 features are Gaussian. Swisher (2022) reduces the dimensionality of the features using autoencoder to be able to directly compute the optimal transport distance between the distributions in the latent space. This issue is also addressed in (Tsitsulin et al., 2019), which rather than using CNN embeddings builds a k-nearest neighbors graphs directly from the data, and then estimates the differences between those graphs. There is also a class-aware version FID (Liu et al., 2018) that tries to model different modes as Gaussians. Additionally, in (Luzi et al., 2023), the authors develop a new metric, WaM, to compare image features, using Gaussian mixture models (GMMs) instead. Since WaM uses GMMs, it generalization of FID.
Our work also shows that PCA be used to speed up the computation time of FID. The computation time of FID is slow due to both the term Tr( Tr(vE, E2) and computing the Inception-v3 embeddings (see Mathiasen & Hvilshøj (2020, Section 2)). There are several recent papers that address this issue. Mathiasen & Hvilshøj (2020) propose an algorithm to compute the term Tr(,,E2) by constructing a matrix with the same eigenvalues. Motivated by this work and applications of random matrix theory to deep learning, Wu Koelzer (2022) proposed comparing the sorted eigenvalues of the two covariance matrices. They show that this is much faster and appears converge faster than FID, requiring a smaller amount of data to estimate accurately.
12

---

Published in Transactions on Machine Learning Research (04/2024)
B Proofs of homeomorphisms
Proof of Example Let P by f()) Px gives the desired forward mapping. Essentially, given X, one can easily construct the exponential distribution pA by the pointwise definition in Equation (3).
(px A E (0, [c) be the distribution space. Then, f (0, 00) IP defined pointwise
Now we construct the inverse of f. Define g: (0, - IP by s(xx) - pp (0) = A. We see that
(fog)(pa) f(g(px)) = f(A) px
and
(go f(() = -(f(A)) = g(p))
implying that f and y inverses of each other.
Thus, exponential distributions are completely characterized by their parameter À and consequently are completely characterized by their mean"" 3-
Now we show that the bijection homeomorphism. Let e > 0 be given along with E (0, oo). Letting 6 262 22 + X, we see that for any X' E (0, 20) such that X we have
V% (Aexp(-Ax) - X eexx(-Xxx)) )² de
Ip^ Px)IL22(B0))
(x2exp( (Pexp(-2Az) - 2XX exp(-x(A X)) + exxx(-22X ) dr
2AX' X
2
X+ 2
X2
2AX X22
2(A + X)
A A XI
V2A + 2X
2X
2
-262 + 22 +1
2 26 - 2e
EEV +X²²
X+ - 22ve2 +\
2 2+2 - -2e +X
- 2 - 2
=6
Thus the mapping is continuous in one direction. Now, focusing the inverse, first note that must pick an 8 > 0 so that all px E IP satisfy
Ipx - PX = V.200 (Aesp(- - N exp(-Xxx)) dx
2
1 XI V2A + 2X +X
<8
We see that after we pick such 8 O, we will have
A XII < 2 28
13

---

Published in Transactions on Machine Learning Research (04/2024)
If we pick appropriate 8 > 0, that 28v c, then proof is done. To do this, square both sides and to solve the cubic equation:
8 10
4 0.
However, need not find the explicit formula for the correct ò 0. Although such formula can be found, it is very messy and not very enlightening. We must just verify that such a 5 > 0 exists. Note that if we choose = 0, then the left-hand side of the equation becomes negative. Hence, there must be a positive, real 5 > 0 that solves that equation (since it is cubic polynomial). Call this d Then, we have that
A XI < 20" V
desired.
Proof of Example Let {Pn, μ Rº E + Rº which takes Pμ,E and returns the mean μ.. We can get the mean from Pp.E by selecting R² which satisfies the following:
+ be the distribution space. First, we define function
sup Pr. (x)).
#ERr
We can see from Equation (4) that μa always satisfies this supremum. To uniqueness, note that
( n1) Σ (a1 p)) (z2 E (x2 (1)
implies that 21 = 222 = since is positive definite. So we define a function m : - IRP by m(Pu,)) =
arg max (x)). øER*
Next, we construct a function s IP + S that takes Pu.E and returns the covariance Σ. Note that due to our Pu, (m.s(((pp.))) Therefore, we focus on the
construction of m, we can easily determine the value of 2nx D simplified function
-
P
P
(=) - T
(8)
j=1
which can be tied back to Pp. via the relation P/,,x(()) 22og
Pn,x(1 + m)). In the above expression, [2>
Djy' the ij-th entry in I-1 Therefore, we first construct the simplified version of s and call 1/,, + E as follows.
We can simply obtain the diagonal of --1 from P/ Let æ; be the zero vector except that the i-th element Then, we see from Equation (8) that //,,x (2;)) = E Similarly, let Øi be the zero vector, except that the elements i-th and j-th are 1. Then, we from Equation (8) that ( E 2E,z² + Ezj- We rearrange this expression get Eny (p)))) - En -jJ') By doing this for each i,j E {1..... p}, we get Σ-1 5-1 and thus Σ.
We construct s as follows. First, we start Pa. and calculate μe from m. Second, we use Pa. E and μ to construct P/z, Lastly, we use p' descrihed above calculate E desired.
We have constructed functions which, when used together, map from P to IP 84 as desired.
C Invertible transformations applied to the Frobenius metric
We first recall the Theorem from 2.2 which shows that the third term in Equation (6), || 8/ 2/ is squared metric.
Theorem 1. For A, B X(n1,.... Typ C) we have that
d(A,Br B) =
Σ
a(AA)) - aB,))
de iE(1.....L) X... (l....e))
defines proper metric d.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Proof of Theorem 1. We first show that d is non-negative, symmetric, and that d(A, B) 0 if and only if A B. Clearly, d( A(A, B) 2 O for all A, B because the sum is over non-negative values. The symmetry is also clear from the definition because |ox (A)) if d(A, B) bijection, desired.
- - -<(,,] - (<(B,) - (A.)) 2 for each index If A then dl A, B) 0. Conversely, this implies that each term in the sum a (A:) (B,)) 2 which implies that A B because is
All that is left to show is that d satisfies the triangle inequality. So we introduce a new arry C E X(m1..... ny; C) for the proof. We write the vector a E C XI..X (and its elements a,) as vectorized (or flattened) versions of A. Similar definitions apply to B and C.
d(A, B) + (BB
Σ |a(Ai)- - a(Bb) +
aaBB)) a(C))
V
V Σ [a(a;) - a(b))
Ja(by) a(G))
|aa(a) a(b)ll2 + |a((b) a(c)ll2
[a(aa) a(c)ll2
(a(a;) - a(c)))
(a(A)) a(C)))
d(A, C),
where we denote a(a), ab),, and ((() as applied element-wise to the vectors a, b, and c. Moreover, the inequality comes from the 2-norm on vectors here. Thus, we have shown that by flattening these multi-dimensional arrays, we can use properties of vector metrics to prove the triangle inequality.
D Discussion of normalization
PCA can be computed by first computing the singular value decomposition (SVD) of the centered data matrix. Suppose
that > p and that the data matrix X E R2>P is centered. Then its covariance matrix is Σ - X X. If the
singular value decomposition yields X = U P V with U E Rax,, D E Rexp, and V E IRPXP then
Σ 1 VD²VT.
(9)
To perform PCA, split up V
[V] V2] and right multiply X by V1 B Here, the number of columns k of Vi is the dimension we are reducing to; equivalently, it is the number of principal components.
So, given two matrices of features X and Y, we compute the singular value decomposition of the reference data set
D
VT, we reduce X and Y by right multiplication by V1, the key point here being that we are using the same matrix Vi to reduce both X and Y. If we consider the covariance of X V1 we get
z. Z may be concatenation of X and Y, or may be larger. Writing U
0
Σ n-1
XV, n n
VT (VIDIVT + V.D2V,
1 D
n-1
since V,V, = I and V2 Vi = 0. Note that
1
P Σ
k Di Dilkes
Tr E
Dla but Tr ES
n n-
15

---

Published in Transactions on Machine Learning Research (04/2024)
implying that the scales of the two traces has now changed.
We conclude that to normalize between calculations of FID with different values of k. one should use the following transformation of X:
==
XV..
i= Dilee
This results in the following covariance:
E D D Dilkk
n --
11
that the trace of that quantity and of Σ are both
1 n-1 Σ
E
Additional skew experiments
Here we show some additional experiments regarding skew; i.e., the blur experiment (Figure 3) and PCA dimension- ality reduction on SID (Figure 4).
FID and the skew with the Gaussian blur experiment
4
4
FID
Skew
3
Skew term
2
1
0
0
0.2
0.4
0.6
0.8
Gaussian kernel a
Ground Truth
0=000 0.0
a= 0.2
a= 0.4.
= 0.8
1=
Figure 3: Skew tracks more with human perception than FID. Note that the blur levels on the images are quite low and undetectable until 0.4 and skew stays low for these undetectable noise levels, in contrast FID, These were typical images taken from the 50,000 samples used to calculate the terms.
F Using different models for feature extraction
In this section we investigate how using different architectures extract features from data affects SID. In Fig- ure 5 plot the behavior of SID four noise distortion experiments using Inception-v3, ResNet-18, ResNet-50,
16

---

Published in Transactions on Machine Learning Research (04/2024)
Salt and Pepper Noise
Added Gaussian Noise
800
d =512
d 5512
120
-a- d =256
d =256
e d =128
d =128
600
100
. -64
-64
=32
332
80
d 116
d 116
SID
SID
400
60
200
20
0.02
0.04
0.06
0.08
0.1
10
15
20
Gaussian standard deviation
Salt and Pepper noise p (%)
Rectangular Occlusions
Gaussian Blur
d 552
d =512
--- d =256
80
d =25
25
.*br d = =128
..d.. d = 128
- d =64
d =64
20
=** d -32
60
d -32
d 16
d 166
SID
15
SID
40
10
20
0.2
0.4
0.6
0.8
5
10
15
20
Gausian kernel a
Erased percentage (%
Figure 4: SID behaves similarly even when we project features down to lower dimensions via PCA. We show four types of distortions are applied to ImageNet images: added Gaussian noise, salt and pepper noise, Gaussian blur, and rectangular occlusions. For the Gaussian blur, we use kernel size of S. In each plot, SID is shown as function of the parameter controlling distortion, after reducing the embeddings to dimension E {512, 256, 128, 64,32, 16). All experiments are done using Inception-v3 feature extractor the entire (50K) ImageNet validation set.
and ResNeXt-101 (32x8d) feature extractors. The noise distortions pepper noise, Gaussian blur, and rectangular occlusions, as described in Section 4.2. All the features come from ImageNet as in Figures and 4. Additionally, dimensionality reduction is applied across all models to dimensions E (512, 256, 128, 64, 32, 16). although its worth noting that ResNet-18 features are already 512 dimensional. We see that the curve shapes are relatively similar across different settings, with some variations in the salt and pepper noise experiment. Overall, the plots show that SID is relatively robust with respect to dimensionality reduction and choice of feature extraction model.
additive Gaussian noise, salt and
17

---

Published in Transactions on Machine Learning Research (04/2024)
Added Gaussian Noise
Salt and Pepper Noise
Gaussian Blur
Roctangular Occlusions
800
SID (Inception-x))
80
d =512
600
d 256
20
400
d =128
40
50
10
=64
200
20
d =32
0
16
0.05
0.1
20
0.5
20
Standard deviation
Noise p (%)
Gausian kernel
Erased percentage (%)
Added Gaussian Noise
Salt and Pepper Noise
Gaussian Blur
Rectangular Occlusions
SID (ResNet-18)
400
15
d 5512
60
40
d 256
10
d =128
40
200
20
64
d =32
16
0.05
10
20
0.5
10
20
Standard deviation
Noise (%)
Gausian kernel
Erased percentage (%)
Added Gaussian Noise
Salt Pepper Noise
Gaussian
Rectangular Occlusions
SID (ResNet-50)
100
400
15
40
d =256
d 511
10
d 122
50
200
20
d =61
5
=32
d 16
0.05
0.1
10
20
0.5
10
20
Standard deviation
Noise (%)
Gausian kernel
Erased percentage (%)
SID (ResNeXt- 101 (32x8d))
Added Gaussian Noise
Salt and Pepper Noise
Gaussian Blur
Rectangular Occlusions
80
400
15
40
d 5512
60
300
d 256
10
d
40
200
20
d 64
20
100
5
322
0
d 16
0
0.05
0.1
20
0.5
10
20
Standard deviation
Noise (%)
Gausian kernel
Erased percentage (%)
Figure 5: SID behaves similarly on noise distortion experiments across different choices of feature extractors. We show four types of distortion applied to ImageNet images, using Inception-v3, v3, ResNet-18, 18, 50, and ResNeXt- 101 (32x8d) feature extractors.
G
Using SID on different datasets
We also study how the behavior of SID on noise distortion experiments is affected by changing the dataset. In Figure 5 we plot SID for the same four noise distortions (additive Gaussian noise, salt and pepper noise, Gaussian blur, and rectangular occlusions) on ImageNet, FFHQ, CelebA, and SVHN. All the features come from Inception-v3. As in our other experiments, we apply dimensionality reduction to dimensions d E (512, 256, 128, 64, 32, 16) for all the datasets. We see that all settings yield curves which increase the noise distortion increased, desired.
18

---

Published in Transactions on Machine Learning Research (04/2024)
We additionally perform the above experiments data from diffusion models. The results are shown in Figure 7. We used pretrained diffusion models* that generated data from the Stanford Cars dataset Krause et al. (2013), CelebA Liu et al. (2015), AFHQ Choi et al. (2020), and the Flowers dataset Tung (2020).
Added Gaussian Noise
Salt Pepper Noise
Gaussian
Rectangular Occlusions
800
SID (ImageNei) Net)
d 5512
100
600
20
60
256
400
d =28
50
10
=64
200
=32
0
d 16
0.1
10
20
0.5
0
10
20
Standard deviation
Noise (%)
Gausian kernel
Erased percentage (%)
Added Gaussian Noise
Salt and Pepper Noise
Gaussian Blur
Rectangular Occlusions
5512
200
SID (PFHQ)
400
30
d =26
40
20
=128
200
100
d =64
10
d =32
0
d =16
0
0.05
0.1
20
0.5
20
Standard deviation
Noise (%)
Gausian kernel
Erased percentage (%)
Addedd Gaussian Noise
Salt and Pepper Noise
Gaussian Blur
Rectangular Occlusions
200
600
=512
SID (CelebA)
80
8
=26
60
400
6
A =128
4
d =64
200
50
20
2
d =32
0
0
d 116
0
0.05
0.1
20
0
0.5
Standard deviation
Noise p (%)
Gousian kemell
Erused percentage (%)
Added Gaussian Noise
Salt and Pepper Noise
Gaussian Blur
Rectangular Occlusions
600
40
150
d 5512
SID (SVHN)
200
400
30
100
..... = 128
256
20
d =64
100
10
50
d 332
0
d 16
0.05
0.1
10
20
0.5
0
10
20
Standard deviation
Noise (%)
Gausian kernel
Erased percentage (%)
Figure 6: SID increases the noise distortion is increased, irrespective of the choice of dataset. We show results from ImageNet, FFHQ, CelebA, and SVHN, using the Inception-v3 feature extractor.
"The pretrained diffusion models from the repository: httpps///dstthhbb..mm//eeeeeeeee ffus
19

---

Published in Transactions on Machine Learning Research (04/2024)
Added Gaussian Noise
Salt and Pepper Noise
Gaussian Blur
Rectangular Occlusions
(DM Stanford Cars)
d =512
400
100
=56
100
WAA d = 128
200
50
=64
30
322
0
166
0.05
0.1
10
20
0.5
0
10
20
Standard deviation
Noise (%)
Gausian kernel
Erased percentage (%)
Added Gaussian Noise
Salt and Pepper Noise
Gaussian Blur
Rectangular Occlusions
SID (DM CelebA)
600
150
d 5512
150
40
256
100
400
100
d =28
=64
50
200
50
=32
0
116
0.05
0.1
10
20
0.5
0
10
20
Standard deviation
Noise (%)
Gausian kemee a
Erased percentage (%)
Added Gaussian Nuise
Salt and Pepper Noise
Gaussian Blur
Rectangular Occlusions
600
SID (DM AFTIQ)
150
150
S12
60
d 256
100
400
40
100
128
50
200
d =64
d =32
0
=66
0
0.05
0.1
20
0
0.5
10
Standard deviation
Noise p (%)
Gausian kemel o
Erased percentage (%)
Added Gaussian Noise
Salt and Pepper Noise
Gaussian Blur
Rectangular Occlusions
SID (DM Flowers)
200
512
200
400
30
150
d =26
20
100
Wo d 128
100
200
=64
10
50
d =32
0
=66
0
0.05
0.1
10
20
0
0.5 Gausian kemnel a
10
Standard deviation
Noise p (%)
Erased percentage (%)
Figure 7: SID behaves as expected with different datasets; increasing as we increase the noise distortions. Here we use data from four pretrained diffusion models trained on the Stanford Cars dataset, CelebA, AFHQ, and Flowers dataset. features were obtained with the Inception-v3 feature extractor.
20


Reviews:
[{"review_id": "JWTWmn9W96", "reviewer": "Reviewer_A5ec", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The article is a combination theoretical justification and experiments, with a good starting point motivation. The examples provided by the authors are easily to follow to understand the limitations of the Fréchet Inception Distance (FID). The authors also pointed out a few action items and directions for future work.\n\nSo FID assesses ‘how similar is a group of images relative to another group of images’, and the authors claim that the default assumption on the Gaussian distribution is not realistic. Thus, higher order moments like skewness are needed. \n\nAfter some experiments, the authors show that FID and SID are actually very much aligned, and the biggest difference is that the SID seems better matching the human perception than FID.\n\nDoes that mean the actually issue of FID is that when measuring group similarities, it does not take into account the human perception due to it does not using higher-order information? Or, do the authors intend to show using higher-order information can help (to some extent) evaluate/align with the human perception issue? More elaboration on this finding could help strengthen this article. \n\n\nFID is a computational more efficient metric comparing to SID. So if human perception is the major benefit for using SID, why not stick to FID and incorporating other human perception metrics? In actual evaluation, we typically look at multiple metrics not only FID, and subject evaluation could also be included during evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors studied the limitations of existing FID metric used for evaluating the GANs, and purpose toe use higher-order moments to access the quality of GAN-generated image features. The authors give a few examples on how existing FID metrics could fail, and they propose a revision to existing FID. The authors further consider incorporating PCA to the metric calculation to accelerate the computation. Evaluation shows that the proposed method (when combined with dimensionality reduction) is computationally and memory efficient, mostly aligns with FID, and is better in terms for human perception.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "On the dimensionality reduction effect:\n\nThe evaluation becomes computational and memory efficient when the dimension has been reduced from 2048 to 256. 10% drop in the accuracy can be a lot. I mean, considering the scenario that model A is slightly better than model B according SID (without dimensionality reduction). Is it possible that the merits diminished or even method B is better than method A after the dimensionality reduction?\n\n\n\nI'm a bit confused on the difference between SID and SFD. Does SID = SFD applied with Inception network?\n\n\n\nEvaluation is fairly limited. It should be tested for multiple models, on multiple datasets, and see how the evaluation agrees with existing metrics and publications.\n\n\n\nThe title is higher-order moments, but actually, the whole article is pretty constrained to be no more than the 3rd moment.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gUrvYSGeKI", "reviewer": "Reviewer_MLqA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. . **Theoretical Rigor**: Solid mathematical foundation\n2. **Empirical Validation**: Effective demonstration of SID's advantages over FID through experiments.\n\n**Weaknesses:**\n\n1. **Limited Scope of Experiments**: More diverse datasets and more experiments on new generative models such as diffusion models could further validate SID's effectiveness.\n2. **Complexity and Practicality**: Implementation complexity and computational cost compared to FID needs clarification.\n3. **Comparative Analysis**: A more comprehensive comparison with other metrics could strengthen the argument.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the Skew Inception Distance (SID), a novel metric for assessing GAN-generated image quality. SID advances beyond the Fréchet Inception Distance (FID) by accounting for non-Gaussian feature distributions, incorporating skewness into evaluation. Key contributions include:\n\n1. **Theoretical Foundation**: Establishes SID as a pseudometric on probability distributions, providing a robust mathematical basis for its application.\n2. **Improved Accuracy**: Demonstrates through experiments that SID aligns more closely with human perception than FID in certain scenarios.\n3. **Efficiency Enhancement**: Introduces the use of PCA to accelerate computations for both FID and SID.\n4. **Broader Applicability**: Suggests potential for SID's use in areas beyond GAN evaluation.\n\nThis work contributes to the field by offering a more nuanced and potentially accurate tool for evaluating the realism and quality of GAN-generated images.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested Changes:**\n\n1. **Broaden Experimental Validation**: Include additional datasets to enhance SID's empirical validation. [Critical]\n2. **Clarify Implementation**: Provide detailed implementation guidelines and computational cost comparison with FID. [Critical]\n3. **Extended Comparative Analysis**: Compare SID with other metrics beyond FID for a more comprehensive evaluation. [Strengthening]\n4. **Real-World Applications**: Explore and document potential applications of SID in real-world scenarios. [Strengthening]\n5. **Robustness Testing**: Conduct robustness tests against various types of noise and image distortions. [Strengthening]\n6. **Inclusion of Diffusion Model Experiments**: Conduct experiments using diffusion models to evaluate SID's performance in a broader range of generative models. This would not only strengthen the paper but also provide insights into SID's applicability across different types of generative approaches. [Critical]\n\nThese changes are aimed at enhancing the paper's rigor, relevance, and practical applicability.", "broader_impact_concerns": "**Broader Impact Concerns:**\n\nWhile the paper focuses on a technical advancement in evaluating GANs, it's essential to address potential ethical implications in a broader impact statement. This includes:\n\n1. **Misuse of GANs**: Enhanced evaluation tools like SID could inadvertently aid in creating more realistic deepfakes or misleading content.\n2. **Bias in Generated Images**: The paper should discuss how SID might handle biases inherent in training data, which can lead to unfair or biased outputs.\n\nIncluding a thorough discussion of these aspects would ensure a responsible approach to the technology's broader societal impacts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lRsN6BWdeM", "reviewer": "Reviewer_vu1z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\n\n+ The paper is well written. I can follow it easily.\n+ Some theoretical analyses are provided.\n+ The proposed SID metric has clear definitions and the authors also detailedly introduce the reason to use the PCA and its impacts on final results. \n\n**Weakness**:\n\n+ The proposed SID looks to be an incremental extension of FID.\n+ This submission only compares with FID. So could the authors also make some comparisons with other popular metrics? This can better present the empirical advantages of the proposed metric. \n+ The submission needs to measure the alignment with human perception. But in Figure 2, I am a bit confused about why to conclude \"Skew tracks more with human perception than FID\". Could the authors provide more details? And a user study might help here for a solid conclusion.\n+ I am also worried about the experiment part. It seems that the authors only perform the comparisons with the Inception-v3 features. How about the performance with other backbone networks like ResNet-50 or ViT? Since SID is a straightforward extension of FID, it is almost necessary to conduct comprehensive experiments to more convincingly demonstrate the empirical advantages of SID.\n+ The authors only apply the corrupted images to measure the differences between SID and FID. I wonder about the quantitative differences for really generated images. For example, the authors can use GAN or diffusion models to generate 1000 images and then compute FID \\& SID. It is important to indicate that SID is statistically meaningful by comparing over a large group of generation results across different benchmarks.\n+ In Section 4.1, could the authors further elaborate how to conclude \"features are skewed after dimensionality reduction using PCA\"? I know some tests using Mardia skewness test and Kolmogorov–Smirnov hypothesis tests are conducted, could the authors provide more details to make papers more self-contained?\n+ In the legend of Figure 1, I can guess that the **d** (e.g. d=16) indicates the dimension. But I think it would be better if its meaning can be clearly specified in the figure caption.\n+ Maybe the paper could be improved by adding a limitation section. Some discussions about the failure cases are also welcome.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors develop a new metric, named Skew Inception Distance (SID), for the image generation evaluation. Compared with the Fréchet Inception Distance (FID), the SID metric can account for the importance of third-moments in image features for the skewed Inception-v3 features. Some evaluations are performed to show the experimental advantages over FID.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weakness part.", "broader_impact_concerns": "Although I don't observe any major concerns regarding impact, it would enhance the paper if the authors incorporate brief discussions on the social implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new metric to measure the output quality of generative models. The proposed method - Skew Inception Distance (SID) - is meant to improve on FID by considering third moments (skew) in addition to the first two moments, better matching human perception.

Overall, the reviewer comments have mostly been addressed (including additional experiments) or flagged as limitations within the paper. As such, the reviewers are in agreement, leaning accept, as the claims in the paper are supported. The main concerns are with respect to the extent of novelty.

The final recommendation is to accept the paper to TMLR as is.
Decision: Accept (poster)