Sorted eigenvalue comparison $d_{\mathsf{Eig}}$: A simple alternative to $d_{\mathsf{FID}}$
Authors: Jiqing Wu
Viktor Koelzer
Venue: Submitted to ICLR 2023 (2023)
Keywords: Distribution shift
FID
eigenvalue comparison
random matrix theory

Abstract: For $i = 1, 2$, let $\mathbf{S}_i$ be the sample covariance of $\mathbf{Z}_i$ with $n_i$ $p$-dimensional vectors. First, we theoretically justify an improved Fréchet Inception Distance ($d_{\mathsf{FID}}$) algorithm that replaces np.trace(sqrtm($\mathbf{S}_1 \mathbf{S}_2$)) with np.sqrt(eigvals($\mathbf{S}_1 \mathbf{S}_2$)).sum(). With the appearance of unsorted eigenvalues in the improved  $d_{\mathsf{FID}}$, we are then motivated to propose sorted eigenvalue comparison ($d_{\mathsf{Eig}}$) as a simple alternative:  $d_{\mathsf{Eig}}(\mathbf{S}_1, \mathbf{S}_2)^2=\sum_{j=1}^p (\sqrt{\lambda_j^1} - \sqrt{\lambda_j^2})^2$, and $\lambda_j^i$ is the $j$-th largest eigenvalue of $\mathbf{S}_i$. Second, we present two main takeaways for the improved $d_{\mathsf{FID}}$ and proposed $d_{\mathsf{Eig}}$ . (i) $d_{\mathsf{FID}}$: The error bound for computing non-negative eigenvalues of diagonalizable $\mathbf S_1 \mathbf S_2$  is reduced to $\mathcal{O}(\varepsilon) \|\mathbf S_1 \| \|\mathbf S_1 \mathbf S_2 \|$, along with reducing the run time by $\sim25\%$. (ii) $d_{\mathsf{Eig}}$: The error bound for computing non-negative eigenvalues of sample covariance $\mathbf S_i$ is further tightened to $\mathcal{O}(\varepsilon) \|\mathbf S_i \|$, with reducing $\sim90\%$ run time. Taking a statistical viewpoint (random matrix theory) on $\mathsf{S}_i$, we illustrate the asymptotic stability of its largest eigenvalues, i.e., rigidity estimates of $\mathcal{O}(n_i^{-\frac{1}{2}+\alpha})$. Last, we discuss limitations and future work for $d_{\mathsf{Eig}}$.

Under review as a conference paper at ICLR 2023
SORTED EIGENVALUE COMPARISON dEig: A SIMPLE ALTERNATIVE TO dFID
Anonymous authors Paper under double-blind review
ABSTRACT
For i = 1,2, let S, be the sample covariance of Z; with n, p-dimensional vectors. First, theoretically justify improved Fréchet Inception Distance (dFID) al- gorithm that replaces np.trace(sqrtm( S2)) with np.sqrt(eigvals(S) S2) (2)).sm... With the appearance of unsorted eigenvalues in the improved dEID, are then motivated to propose sorted eigenvalue comparison (dEig) as a simple alternative:
(2) = Ej=(( x3)², and X3 is the j-th largest eigenvalue of S; Second, we present two main takeaways for the improved dFID and proposed deig (i) dFID: The error bound for computing non-negative eigenvalues of diagonalizable S₁S, is reduced S S2 along with reducing the time by 25%. (ii) dEig: The error bound for computing non-negative eigenvalues of sample covariance S; further tightened (E ||| || with reducing 90% run time. Taking a statistical viewpoint (random matrix theory) on S, we illustrate the asymptotic stability of its largest eigenvalues, ie., rigidity estimates of On, ata) Last, we discuss limitations and future work for deig.
INTRODUCTION
Import nuпpy
2 from scipy. linalg import oigvala, cigvalsh
3 The square of improved drib
def dFID dFID (meanl, covl, mean2, cov2)
eigval eigvals
ccv2)
Round computst
arrora rif exist)
Lead negative eigeavalues
to
9
eigual (aigual < 0] =
10
dir meanl mean2 dif.dot ap.trace (cov] cov2)
11
12
return res 2 ap.sgrt teigval) s)m()
13
14 # The square of proposed dex
dacgl scm2):
16
Sorted etgenvalues, eigvall (igvalshhssmm)]
18
eigvall[eigvall
19
eigvel2 eigvelsh (scm2)
20
(eg
21
dir - ap, sqet (eigvall) ap.agrt (eigva12)
zeturn dif.dot (dif)
Figure I: Python codes for the square of improved dFID and proposed deig-
In the image domain, it is of great interest to analyze the distribution shift between two collections of data entries (Wiles et al. 2021 1 Borji 2019). On one hand, this is driven by the increasing awareness about the violation of the assumption of 'identical distribution' between training and (real-world) test datasets (Wu et al. 2022b). As for instance illustrated in the leaderboard of WILDS (Koh et al., 2021 Sagawa al. 202 Y many algorithms suffer from performance degradation and fail to generalize to heterogeneous testing settings. On the other hand, the importance of assessing distribution shift has been recognized with the rise of generative adversarial nets (GAN) (Goodfellow et al. 2014 Heusell

---

Under review as a conference paper at ICLR 2023
al. 2017). The rapid development of GAN variants (Karras et al. 2019 2020b urges reliable and accurate metric(s) to assess the discrepancy between generated and real images (Borji 2019).
To objectively assess GAN models, researchers have proposed plethora of evaluation scores
including Inception Score (Salimans et al., 2016), Kernel Inception Distance (dkID) (Biikowski et al.
2018), and Precision/Recall (Kynkäänniemi al. 2019 Sajjadi et al. (018)) (please also see (Borji for in-depth review). Among various scores, Fréchet Inception Distance (dFID) (Heusel 701771 is arguably the most widely-used metric for benchmarking GAN performance (Parman et al. 2022). This is mainly due to the favorable theoretical property of being a mathematical
2019
2022)
al.
metric (Dowson & Landau 1982) and practical property of being well-correlated with perceived image quality (Sajjadi et 2018). Meanwhile, Chong & Forsyth (2020) argued that dFID is a biased estimator and Kynkäänniemi al. (2022) observed its undesirable sensitivity towards fringe features or classes. Despite these weaknesses, dFID currently remains the goldd standard' for GAN evaluation and continuously attracts broad attention. In recent study, |Mtthiasnn Hvilshøj (2020) proposed
to compute eigenvalues rather than square root of matrix in dEID. We view this as promising simplification and improvement, nonetheless a precise theoretical analysis has not been performed and therefore becomes the starting point of this paper.
The study of random matrix theory (RMT), with an emphasis on understanding the properties of (random) eigenvalues (Paul & Aue 2014). has brought novel insights in the domain of deep learning (Liao & Couillet 2018). Pastur 2002 Baskerville et al. 2022). among which [Seddik et al.] (2020) analyzed deep learning representations of GAN generated images through the lens of eigenvalues of their sample covariance matrix (SCM). Driven by the need to efficiently quantify the distribution shift between two collections of heterogeneous data entries, we propose to compare sorted eigenvalues (dEig) as simple alternative to dFID- Our contributions are summarized as follows:
For i = 2, let S, be the sample covariance of Z; = (zf,..., 2/. with ni p-dimensional vectors.
(dFID) We articulate the fact that S,S, is diagonalizable and has non-negative eigenvalues. This allows us theoretically justify an improved algorithm of dEID. i.e., by replacing the unique principal square root of a matrix with the element-wise square root of its eigenvalues. Therefore, the error bound for computing its eigenvalues reduced to O(s) S [ S2 reducing the run time by- ~ 25%.
(dEig) Since S, is symmetric positive semidefinite, the error bound for computing its non- negative eigenvalues is further tightened to O())] along with reducing 90% run time. From the viewpoint of random matrix theory (RMT), we demonstrate the asymptotically stable behavior of the largest eigenvalues (spikes).
THE IMPROVED dFID
(Linear Algebra) Notation: Lower case Roman or Greek letters (e.g., 8, Y, Y, denote scalars, bold lower case letters (e.g., 2, (1) denote vectors, and bold upper case letters (e.g.. Q, U, Z, A) denote matrices. is matrix transpose, ||.|| is L norm, S denotes asymptotically less than.
2.1 PRINCIPAL SQUARE ROOT OF MATRIX
Without loss of accuracy, we discuss dFID through the lens of linear algebra. More specifically, scalars, vectors and matrices discussed in the section are deterministic, while a statistical viewpoint on these objects will be later introduced in the proposed dEig section. For i = 1, 2, let Z, [zj..... 2%, be collection of n. p-dimensional vectors. For simplicity, we assume sample mean 18, 2 2
throughout Sec. Accordingly, S, = _ ZT denotes the sample covariance matrix (SCM) of Z,- We start the discussion with revisiting standard the definition(s) of dFID (Givens & Shortt 1984), then we elaborate the properties of principal square root the key computational challenge of dFID-
2.1.1 (
Definition 1. Let St be the SCM of Z, and w.l.o.g. S₁ is non-singular, then we define
drin (S, 2
Trace + S2 2(S (
(1)
2

---

Under review as a conference paper at ICLR 2023
To compute the Trace() of dFID, first need to clarify the symbol in Eq. As mentioned in (Dowson Landau 1982). denotes the positive (or principal (Higham 2008 square rool of matrix S such that S S S, and 'principal' specifies the square root(s) S with non-negative eigenvalues. In general, the square root of matrix may neither exist nor be unique (Higham 2008). Consider now the special case where S is symmetric positive semidefinite (PSD), then we have
Theorem 2. (Principal square root) Let symmetric PSD S be decomposed as S = QAQ", where is an orthogonal matrix and A is a diagonal matrix with non-negative eigenvalues, then S Q :T is the unique principal square root of S.
Based on the definition of St, is easy that S S. Importantly, 5 is unique in the sense that if there exists another symmetric SPD 5 such that 5: S, then 5 Ss. With Thm. in hand and given the fact that S and S2 symmetric PSD, make the following claim.
Corollary 3. S S2ST is symmelric PSD matrix and therefore has unique principal square root.
Claim. Accordingly, Tra S257 )+) (Fq.) 1) can be derived after eigenvalue decomposition sug- gested in Thm. [2] As computational routine, this formulation is nevertheless undesirable. Because we need to call the eigenvalue decomposition function twice, which potentially increases computa- tional time and error risk. Instead, we seek for another equivalent formulation of Eq. Givens & Short 1984p.
2.1.2 (race((S,S2)*)
Lemma 4. Following the specifications of S; in Eq. then we have
(S. Trace(S, S2 2(5))))
(2)
Because of non-singular S₁, it is not difficult to see that eigenvalues of S S.St and S₁S, are identical, and the corresponding eigenvectors are identical up to invertible linear transformation
SÍ (or S ). Due to the fact that S S is symmetric PSD, then we have Corollary 5. S,S, is a diagonalizable matrix with non-negative eigenvalues and therefore has unique principal square root.
Remark Note that least one of S and S2 should be non-singular or the null space S; should be contained in that of S₂ (Dowson & Landau 1982). If S1 is singular, then the above discussions remain the same after switching the role of S and non-singular S2.
mathematically equivalent the element- wise square root of cigenvalues of S2 Since S,S, and S S.S² have identical eigenval- ues and the trace of a diagonalizable matrix is the sum over its eigenvalues, then we have Tra Sz)+ = Trace((S} S25)) ) and prove Lem. Importantly, this rigorously justifies the workaround algorithm of dFID that computes the element- square root of eigenvalues, which bypasses the expansive computation of the square root matrix.
Claim. Consequently, eigenvalues of (2)
2.2 ELEMENT-WISE SQUARE ROOT OF EIGENVALUES
Before substituting the square root component of dEID. let take a step back and re-examine its widely-used implementation ecipy.lingale Iingallg.grtm(([ In a nutshell, the underlying computational routine is blocked Schur algorithm (Björck & Hammarling 1983 Deadman et al.) 2012). which includes two phases: Schur decomposition (schur()) and solving (triangular) Sylvester equation. For computing Trace() of p x diagonalizable matrix (S, S2) we show the latter phase is redundant.
https://github.ee htt.hh////gtoooooo//aaaaaaaraa/a ficaan--id/// fi

---

Under review as a conference paper at ICLR 2023
2.2.1 NUMERICAL ERROR BOUND
Corollary (Schur decomposition) Let the diagonalizable matrix S,S₂ be decomposed as QUQ'. Here, Q an orthogonal matrix and U an upper triangular matrix. Then Trace((S.S2))) Σ)=1 Vugz, where U1.....,, Upp are diagonal entries of U.
This equation is derived from the fact that diagonal entries of U are exactly tion-neaaiveeeiieevvllss eigenvalues of Sy, As immediate consequence, it suffices to compute schur() for obtaining Trace((S{ (2)) By default, schur() simultaneously computes U and Q. In our case, we only want to compute diagonal entries of U. This leads to more speedy eigvals() that shares the same core routine as schur(.. Eventually, we replace the standard pythonic implementation np. (p.ttace((qrm(() with (erriivvll()..smmdde (See Fig. [orr more details and Mathiasen & Hvilshøj (2020) for reference). Such a series of algorithmic simplification allows us to propose (strictly) tighter error bound compared the original case w.E.. sqrtm().
Error bound of eigvals(. As discussed in (Anderson et al. 1999.. for the computed eigenvalue 7 and eigenvalue 7/ of S2S₁ we have [/g - S O(e)s S1 where is machine epsilon. The remaining task compute 8y- Since if my is the right eigenvector for 7₁ then the left eigenvector
v ST Because of 8j == |0] ST 1 my j I.....P. the (asymptotic) error bound for computing eigenvalue % can be formulated as
||S| 1 112 we have s, I || 112 || For
1g - % x
Moreover, if want compute eigenvalues of the p p SCM S; that is symmetric PSD, we can utilize the eigvalsh() with lower run time and obtain a tighter error bound.
Error bound of eigvalsh(. For ..... . P. the error bound for computing eigenvalue Xj of S, can
be formulated as (Anderson et al. 1999
Xi < (c)
(4)
2.2.2 EIGENVALUE COMPARISON
Now, we discuss an important variant of dEID when S and S2 commute, i.e., S,S2 = S2S1. Corollary 8. (Unsorted eigenvalue comparison) Let S1.2 two SCMs that are simultaneously diagonalizable by orthogonal matrix Q, then
(SI, Trace((S) -
-(v) 1232
(5)
j=l
where X3 is the j-th eigenvalue of S: w.r.l. Q.
Under such special case where S and S₂ share the same eigenbasis, driD is reduced to computing the Fuclidean distance between unsorted eigenvalues. Motivated by this reduction, we propose to compare sorted eigenvalues as a simple alternative to dFID-
Definition 9, (Sorted eigenvalue comparison) Let S1.2 be two SCMs, then we define
S2)
ΣV X3)2
where X; is the j-th largest eigenvalue of S,. Accordingly, dEig is a pseudometric on the set of SCMs with order p.
Note that Si and S2 in Eq. do not necessarily commute. Instead of eigvals( used for computing eigenvalues of non-symmetric S (Eq. 2 dEig can be obtained with a more numerically stable and
faster eigvalsh(,, which is customized to compute X; of symmetric S. As a pseudometric, dEig satis- fies non negaaiviti symmetry and triangular inequality, while SCMs need not to be indistinguishable regarding deig. Following the convention, deig and dFID scores reported the following are always the square of dEig and dEID resp.
4

---

Under review as a conference paper at ICLR 2023
10M
vise
knen
ando
wis
san
cood
2000
20000
1918
M00
vitan
M010
anto
Jonn
2000 10e0 PM
(Orocle comparison) The 4hpp and day soores computied between sample covariance 60K, 40%,
Sk ground covariance.
190
1500
359
80
199
100
5008
40W
5000
5060
1800
5000
1808
2000
2009
1000
580
(leaf-world comparison (The dlyp dag computed sumple
40k, sample
dypo
dim
wottnn
actur
nigudds
wedheals
sigxalsh
16304
1101
(1
969.4 (2
854.16
100.17
66,03
(0.57)
4096
48 (0.04)
15.58 (0,57)
14,47 (0.11)
2.77 (2.00)
2.78 (9,000
2048
2.34 10 02121
1.00 (0.02)
1.59 (0,03)
0.49 (2:00)
0.20 (8.00)
The dupp deg obtained difirrent implementatioon variants.
Figure 2: The toy studies of multivariate Gaussian distribution (0, E). Here, all the experiments are computed with Intel(R) 19-9940X CPU @ 3.30GHZ and repeated with four random seeds. Since the coefficient of variance std/mean /mean < 0.01 for both dEig and dFID, we only report the mean score in Plot a and b. Besides, sqrtmO, schur() and eigvals( achieve identical numerical results for computing dFID up to negligible rounding error. So do svdvals() and eigvalsh() for computing dEig. Therefore, only two curves are presented in Plot a, b.
2.2.3 TOY STUDIES: dEig IS MORE RELIABLE THAN dFID.
Mathematical equivalence between dEig and dRID. For proof of principle, we conduct toy studies with multivariate Gaussian data. Concretely, we construct non-negative diagonal entries of p- dim covariance matrix E with
while keeping the off-diagonal entries zero. By multiplying Σ and X, np.random.randn(p. ri:), we obtain n, Gaussian data entries Y, = E that are drawn from (0, E). Then we compare S = Y, to ground-truth Σ (Fig. (a) and to S2 1 Y.Y (Fig. b)). Following above theoretical discussions, instantiate nz Eq. of dEig with sqrtmó, schur() and eigvals(). Because S; is symmetric SPD, implement Eq.2 of dFID with svdvals() and eigvalsh(). Throughout our experiments, we notice that the results of implementation variants are identical up very small rounding errors. Therefore, we experimentally confirm the validity of improved dEID and the equivalency between svdvals() and eigvalsh(. Because identical (sample) covariances simultaneously diagonalizable, we have dEig dFID in theory. Since S 2 Sy ~ E with sufficient amount of data, we expect dEig s dFID z 0 in practice.
Numerical difference between dEig and dFID- When comparing S to E. Fig. 2 (a) shows that dEig and dEId have a comparable trend of decreasing scores with a growing number of data entries (5k+500 50k). This indicates that both dEig and dFID are meaningful metries and can converge their theoretical limit. When comparing S1 to S2, Fig. ( illustrates that dEig is more resistant to the data size difference. contrast to dFID. it suffices to use a smaller amount of data to achieve good estimation for dEig. Arguably, dEig represents more reliable score than dFID due to the fact that 1) dEig demonstrates favorable convergence curves that are overall closer to O, and 2) comparison with the standard dFID (Eq. (2). dEig (Eq. 6) is more faithful routine approximate Eq. 5 the simplified dFID for our toy setting.
Run time. When comparing different variants for implementing deig and dFID, Fig. 2 (c) shows 18% 32% reduction of run time by replacing sqrim() with eigvalsO, and we further reduce the run time by 85% 94% when utilizing eigvalsh(). As a result, it is beneficial to apply the improved deig and proposed dEID for computing distribution shifts, especially in the high dimensional cases such as p 16384. From now on, dEig and dEID are computed with eigvals() and eigvalsh() by default.
5

---

Under review as a conference paper at ICLR 2023
3 THE PROPOSED d'ig
(Probability) Notation: Sans serif lower case letters (e.g., x, z, A) denote random variables, sans serif bold lower case letters (e.g., x. y, (1) denote random vectors, and sans serif bold upper case letters (e.g.. X. Y, Z, Q S) denote random matrices. x denotes asymptotic equivalence, N is the set of positive integers, E is expectation and P is probability distribution.
Following the computational analysis on deterministic eigenvalues, we here investigate the statis- tical facets of deig from the viewpoint of RMT. Given a SCM, researchers have keen interest in understanding the asymptotic behavior of the largest eigenvalues (spikes) (Izenman) 2021). This is motivated by the observation that spikes reflect the direction of largest variance and reserve the most critical information (Perry et al.) (018). Since spikes also dominate the computation of deig. it is important to analyze their asymptotic behavior in our study.
3.1 ASYMPTOTICALLY STABLE BEHAVIOR
In this section, cigenvalue. vector and matrix are non-deterministic and indicate random eigenvalue (variable), random vector and random matrix unless stated otherwise. Firstly, we recall canonical case where X = (X1..... X,, is a random matrix with IID entries. For j = 1,.... P and k = 1..... let n x p and ylk be the IID entries of X satisfying E xx1l2 = 1 and E|x;ss 1 S cm for all fixed m E we further assume Exjx 0. As shown in one of the pioneer studies (Marčenko) & Pastur 1967). the asymptotic eigenvalue density of Xx can be well characterized with limiting Stuelfjes transform. However, the assumptions of IID entries and diagonal covariance structure are very stringent and do not reflect real-world data statistics. Regarding GAN assessment concrete example, samples drawn from x, are usually representations obtained with the penultimate layer (pool3) of an Inception V3 model (Szegedy al., 2016). Therefore, X1k ......... ..ppk. Xpk of x; can be dependent and have a more general covariance structure Σ. To resolve the gap, we assume E satisfies the stability condition as in (Bao et al. 2015 Condition 1.1 (iii) and propose to investigate
N₂. Similar to Sec. [
= E X, linear transformation of X.
2 X be the r largest eigenvalues Q = 1YYT, then for any j = 1, . we can find deterministic 6, such that for any (small) a > and (big) 3 > O, we have
Theorem 10. (Case of zero expectation: Y = Σ $ X) Fix r and let X1 2...>
PP - 0,1 2 n 8ta) S Ca.3n 8
for some constant C independent of n.
Discussion. Here, Thm. 10 is a direct result of the local density law (Knowles & Yin 2017).
Note that #. LEYY = Σ holds true and we impose linear dependency among XIR, Xyk of X to approximate real-world scenario. In the meantime, xI,... Xn remain identically distributed, which reflects the key fact that data entries such as image representations of GAN model are drawn from the same probability distribution. As learned representations commonly have non-zero expectations m of O, we further introduce a deterministic rank-1 matrix M me. to model this scenario. Here, e= (1,...,1) 11) and m = du satisfying d = p and || = 1. For Z = M + Σ we have - M + Σ Fix and let A1 2 ... xy be the largest eigenvalues S zzz then for any j 1, . . can find deterministic 0j such that for any (small) a >> and (big) 3 > O, we have
Lemma 11. (Case of non-zero expectation:
P(AA- - 0 n àta)= S ca.an
(8)
for some constant Ca independent of nt,
Lemma 2.2), the eigenvalue counting functions of Q and S <, X for 2, 3..,,.. T. Then, the rigidity estimation of R 3,......, ...,, r can be obtained by considering 0, := 0, and applying Thm. 10 As to the case of j = 1, we consider Eq. 8 for (2 À as a conjecture and leave the proof for future work. Together
Discussion. As shown in Bai 1999 differ by most 1. Thus, Az+1
p'
for j = 2,
with discussions Sec. 2 we illustrate both the numerical and asymptotic stability of dEig.
Remark 12. Similar to Louart & Couilled 2018 Remark 0.I), the above lemma suggests a (abusive)
definition of SCM S = IzZ' without subtracting the mean expectation. Taking S as the input of dEig. we show that it feasible for dEig to quantify the distribution shift in follow-up GAN studies.
6

---

Under review as a conference paper at ICLR 2023
3.2 GAN STUDIES: dEig SIMPLE ALTERNATIVE TO dFID-
Recently, Parmar et al. 2202) discovered surprising subtleties of image pre-processing steps for downstream GAN evaluation. To faithfully benchmark the GAN performance of state-of-the-art (sota) models, the authors published new APIs to reproduce the evaluation results. Hence, the implementation of our GAN experiments is built on top of these APIs. Next, we summarize four key aspects of GAN evaluation that we examine in this study.
4 Scores. Similar to Parmar et al. 2022), we take two widely-used scores dFID and dKID as baselines. Then, we investigate two variants of the proposed metric: (S,, S2) Ej=(vA /xz>2
E - V2322 m- - m2l1 The X3 of the former are cigenvalues of S., and X; of the latter are cigenvalues of S; 1 m.m.. where m, is the sample mean. Differing from toy settings of Gaussian distribution (0, Σ) that lead to deig R dFID R 0 with sufficient data, do not have such theoretical limit or ground-truth score in GAN studies, As workaround, we consider dFID to be the 'gold standard' score for analyzing dEig. Without loss of accuracy, we take dKID x 10sh and dEig x 10 for clearer comparisons.
(Eq. and (St,
3 Models. To illustrate the strength of dEig for challenging cases, we investigate three sota GAN models and probe their nuances when visual evaluations are non-trivial: StyleGAN2 with the recommended Config (Style2) (Karras al.] 2020a). StyleGAN3 with translation equivariance Config (Style3t) and with translation and rotation equivariance Config (Style3r) (Karras et al. 2021).
3 Interpolations. Following the practice of Parmar et al. (2022). we also present results that influenced by different image interpolations such as Clean (Clean), PyTorch_eegaee (Py legacy and SensorFlow_legayy (TF legaey..
5 Datasets. Lastly, we run thorough comparisons on commonly-used datasets including FFHQ, AFHQ, and LSUN (Horse, Church, Cat categories) for GAN model training. For each dataset, we generate 100k fake images and repeat each experiment 4 times by randomly sampling a given number of image entries from 100k fake images.
- drp chiday 1D Siz -0-if
Srywer
Styl
0
Oes
Inpoo
legoy Oser
Halp
SUm 4000 5000 5000 1900 2003
50to 4ee 1909 2009
PHO datased) The quantitaivv scores computed with and b0k GAN generated images.
[ITHQ datssel) The dogo and fag
acores computed with 50k. 31 CAN ganerated images.
Anp
48
(AFHO dataset) quontitative soores computed with generated Images.
(AFHO dataset) The dap dog soores computed generated Images.
Hole
Chack
Home
Church
drp
3 3614:00001
0.00:00.00
0100:0001
6727--0.00
1185-0018
608-0.007
O. G.700.0.010
1.277.0.004
4.211.0.00
Get
Som 4000
2000 (980) 1900
cono 1900 19c0e Ho0M B0w
Saoo M00e som 1000 1960 080
imagas.
1. F.SSUN dataselj The dyys and di.
acores computed with 50k- Sk GAN genersted imagee.
Horo, results obtained with Style2 model Clean Config.
Horo, results obtained with Stylod model under Clean Config.
Figure 3: The main results of GAN studies. Here, 70k, 15803 and 50k real images for FFHQ AFHQ and LSUN datasets resp. are applied to compute the reported scores.
In the following, we discuss the main results of our GAN studies. As displayed in Fig. 3 (a, b), dEig and diID show similar evaluation curves and correlate well with each other in terms of different combinations of models and interpolations. When observing the convergence curve with an increasing
7

---

Under review as a conference paper at ICLR 2023
amount of GAN generated images Fig. 3 f), observe identical behavior in the toy studies. That is, deig is more favorable than dFID in the sense that suffices to use a small amount of image entries to obtain a good estimation for deig. Similar claims can be made for the LSUN dataset. As shown in Fig. (c), dEig illustrates comparably increasing scores from the Horse to the Cat category, indicating less satisfying GAN generation results for the Cat images. Meanwhile, the convergence speed remains faster for dEig compared to dFID (Fig. 3 (t)). With regard to d'Egg' the variant of sorted eigenvalue comparison show less consistency with the gold standard dFID (See Fig. 3 (a, b)) and is less desirable in our GAN studies. Based on the investigations of the four key aspects and theoretical advantages of dEig discussed above, the proposed dEig represents simple alternative to dFID. By applying dEig GAN model evaluation, we take critical step towards a more comprehensive analysis of high-dim distribution shift between two collections of image entries.
FFH0
Ekyeewatie
Blpeweeter
APHO
Bipeniake
Eeeeweter
Hamse
Bipanvkke
Bigewecter
Charch
Biperiake
Bigeweeter
Cel
Bipenndar
Bernwoter
3.99
207.8166:197
0.29
0.0
14:35-200
4844--1 BN
0011
13.660004
2.34
6.000011
0.00
2.3ae001
1.04
43740005
720tdd
divoure
0/06
189
8.00:000 D40
2013
1:500000
0000
0000
3 3.2200015
28'5
0.011
Eddted
4.Nwnoms
2.008
1033
3.200000
0006
2.7t00oss
001e
2 25300012
2.20
DACCY
0080
2031
24400004
000E
24470001
0013
1.200000
2.00
2.44000t
0,122
2.79
2:1000014
4440000
Figure 4: The eigenvalue fluctuation and eigenvector similarity for 10 spikes of dEig- Here, all the experiments are obtained with Style2 under Clean configuration. The eigenvalue fluctuation (standard deviation) is obtained by repeating experiments with random seeds. Also, we report the largest cosine similarity between the i-th largest eigenvector of GAN images and its counterpart of real images. The indicates that the largest cosine similarity is not obtained between the the i-th largest eigenvectors of GAN and real images.
As displayed in Fig. we report the eigenvalue and eigenvector behaviors for the largest spikes. The reported cutoffs were determined by the dominant percentage (> 80%) taken by these spikes compared to the complete spectrum. Notably, the 10 largest spikes present small fluctuations (std) obtained with four random seeds, which serves complementary evidence to support the theoretical rigidity estimations discussed in Thm. 10 and Lem. 11 Except for the few cases marked with the largest cosine similarity is mostly obtained with the i-th largest eigenvector for both GAN and real images. If we decompose the distribution shift to scale shift (eigenvalue shift) and rotation shift (eigenvector shift), such results suggest that the dominant eigenvector shift is only determined by the cosine of the angle between them, and is not influenced by eigenvector permutation. By weighing the estimation challenges of eigenvectors, dEig makes a meaningful trade-off that only takes eigenvalue differences into account.
legacy
Jegacy
Style3t
StyleBr
Style2
StyleSt
StyleGr
Style2
StyleSt
StyleBr
Stylc2
deID
3510=0..02
3.43:0.042
3.1.0.01
2.77=0.07
3.088-0.41
2.866.0.06
2.77+0.022
3.03640.087
2.857
deig
1.9.0.08
2.22.0.061
2.657+0.073
1.627+0.060
1.883+0.063
2.513+0.056
1.625x0.005
1.883+0.043
0.507
Figure 5: The nuance between dFID and dEig- Here, all the experiments are conducted with the FFHQ dataset.
Lastly, we report a nuanced case when comparing dFID and dEig. Fig. 5 shows that the face general- ization performance w.n.t. dEig tends to be improved from Style2 to Style3r and Style3t, which not compatible with dFID. By imposing the translation and rotation equivariance in StyleGAN3, Karras jtt al. 2202)) reported anti- anti-aliasing improvements over StyleGAN2 by resolving the 'texture sticking' supported by the decreasing dEig scores. However, due to
issue. Such clear visual improvements the lack of ground-trus -rutt, whether such a correlation between visual improvements and drig supports the effectiveness of deig remains inconclusive.
8

---

Under review as a conference paper at ICLR 2023
4
RELATED WORK
Fast dFID- The role of eigenvalues played in drio has been firstly noticed in the study of Fast Fréchet Inception Distance (Mathiasen & Hvilshøj] (020). When n1 < p. the researchers suggested to compute the eigenvalues of nı x ni matrix / Z2 Znj Zi supervise the model training. Accordingly, the differences between our study and Fast dFID lie the fact that we do not assume in & p and we compute S mu Z, Z Zg Zg instead. Since a more precise justification w.r.t eigenvalues was not present in Mathiasen & Hvilshoj 2020), the key contributions of our theoretical analysis on dFID come from articulating the unique principal square root, diagonalizable S S2 with non-negative eigenvalues and proposing a tight asymptotic error bound.
Seddik et al. (220)). From a RMT viewpoint, Seddik et al. (020) studied the SCM of GAN image representations and argued that such representations behave asymptotically as if they are drawn from a Gaussian mixture. Following this insight, we further show that comparing sorted eigenvalues of SCMs is useful and efficient for measuring nigh-dimensional distribution shift, which is novel and distinct contribution by our theoretical study.
5
LIMITATIONS AND FUTURE WORK
5.1 EIGENVECTOR
In contrast to the improved dFID that implicitly takes eigenvectors of S; into account via the matrix multiplication S2S the proposed dEig only measures the eigenvalue difference. Admittedly, the exclusion of eigenvectors in dEy is mainly due to the disencouraging properties such as more loose numerical error bound (Anderson et al.) 1999) and more strict conditions for distribution estimation (Knowles Yin 2013). Nevertheless, eigenvectors carry plausibly critical information and should be carefully examined in subsequent work.
5.2 FUTURE STUDIES: deig MAY BE MORE COMPREHENSIVE AND INFORMATIVE THAN dFID-
Similar to existing measurements, dEig remains scalar-valued score for measuring high-dimensional distribution shifts. A more comprehensive quantification is still missing for applications in both the natural and medical image domains. Due to the inherent data heterogeneity and critical im- plications for real-world application, facilitating in-depth analysis of distribution shifts underlying high- high-dimenssonal images (or representations) is of key importance to support the development and application of high-quality data science approaches, e.g., in the medical domain (Yue et al. 2020 Cios & Moore, 2002.. In such a scenario where inaccurate analysis can have severe consequences, existing scalar-valued scores including dEig is not sufficient. To resolve this issue, direct follow-up on dEig is to individually compare the eigenvalue difference along each dimension. Naturally, the scalar-valued dEig is decomposed a more complete overview of data heterogeneity. In addition, the deig builds the bridge between the classical principal component analysis (PCA) (Abdi Williams, 2010) and latent semantic understanding (Shen & Zhou 2021 Härkönen et al.
multi-ddimensionl vector-valued measurement and enables
2020).
Taking cancer studies as an example(Fremond dimensional analysis with dEig could pave a promising way towards precise risk stratification by validating well-established and proposing novel features with prognostic importance in complex medical images. This can be concretely supported with biologically interpretable visualization examples generated by perturbing the largest iigervalue(sSecigenvettooo interest. This approach could thus be used to control for inherent variance in existing data repositories and generate prototypical examples of disease states such as highly aggressive tumors in radiological pathological time series.
al. 2022, Wu al. 2022a), the fine-graines multi-
in a given dataset of
In combination of comprehensive quantification and biologically meaningful visualization, dEig thus adds valuable tool for future work in the natural and medical image domains.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433-459 2010.
Edward Anderson, Zhaojun Bai, Christian Bischof, L Susan Blackford, James Demmel, Jack Don- garra, Jeremy Du Croz, Anne Greenbaum, Sven Hammarling, Alan McKenney, et al. LAPACK users' guide. SIAM, 1999.
Zhidong Bai. methodologies in spectral analysis of large dimensional random matrices, a review. Statistica Sinica, 9, 1999.
Zhigang Bao, Guangming Pan, and Wang Zhou. Universality for the largest eigenvalue of sample covariance matrices with general population. The Annals of Statistics, 43(1):382 -2122221, 2015.
Nicholas P Baskerville, Diego Granziol, and Jonathan P Keating, Appearance of random matrix theory in deep learning. Physica A: Statistical Mechanics and its Applications, 590:126742, 2022.
Mikolaj Binkkowki,, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mnd gans. arXiv preprint arXiv: 01 401, 2018.
Åke Björck and Sven Hammarling. A schur method for the square root of matrix. Linear algebra and its applications, 52:127-140, 1983.
Ali Borji. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding, 179:41-66 65, 2019.
Ali Borji. Pros and cons of gan evaluation measures: New developments. Computer Vision and Image Understanding, 215:103329, 2022.
Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6070-6079, 2020.
Krzysztof J Cios and G William Moore. Uniqueness of medical data mining. Artificial intelligence in medicine, 26(1 12:1:2224444 2002.
Edvin Deadman, Nicholas Higham, and Rui Ralha. Blocked schur algorithms for computing the matrix square root. In International Workshop on Applied Parallel Computing, pp. 171-182. Springer, 2012.
DC Dowson and BV Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):550-455 1982.
Sarah Fremond, Viktor Hendrik Koelzer, Nanda Horeweg, and Tjalling Bosse. The evolving role of morphology in endometrial cancer diagnostics: From histopathology and molecular testing towards integrative data analysis by deep learning. Frontiers Oncology, 12, 2022.
Clark Givens and Rae Michael Shortt. class of wasserstein metrics for probability distributions. Michigan Mathematical Journal, 31(2):231-24 1984.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems, 33:9841-9850. 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update nule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Nicholas J Higham. Functions of matrices: theory and computation. SIAM, 2008
10

---

Under review as a conference paper at ICLR 2023
Alan Julian Izenman. Random matrix theory and its applications. Statistical Science, 36(3):421-443 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4401-4410 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 12111-1-12144, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020b.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.
Antti Knowles and Jun Yin. Eigenvector distribution of wigner matrices. Probability Theory and Related Fields, 155(3):53-552,, 2013.
Antti Knowles and Jun Yin. Anisotropic local laws for random matrices. Probability theory and related fields, 169, 2017.
Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, et al. Wilds: A benchmark of in-the-wild distribution shifts. International Conference on Machine Learning, pp. 5637-5664.
PMLR, 2021.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fr) 'echet inception distance. arXiv preprint 2.XXi:2203306026 2022.
Zhenyu Liao and Romain Couillet. The dynamics of learning: random matrix approach. In International Conference on Machine Learning. pp. 3072-3081. PMLR, 2018.
Cosme Louart and Romain Couillet. Concentration of measure and large random matrices with an application to sample covariance matrices. arXiv preprint arXiv: arXiv:/805.08295, 2018.
Vladimir A Marčenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Mathematics of the USSR-Sbornik, 1(4):457, 1967.
Alexander Mathiasen and Frederik Hvilshøj. Backpropagating through fr) 'echet inception distance. arXiv preprint arXiv:2009. / /4075, 2020.
Gaurav Parmar, Richard Zhang, and Jun-Yar Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11410-11420 2022.
Leonid Pastur. Eigenvalue distribution of large random matrices arising in deep neural networks: Orthogonal case. arXiv preprint arXiv:2201.0454 2022.
Debashis Paul and Alexander Aue. Random matrix theory in statisties: A Deview.JJurnig of Statistical Planning and Inference, 150:1-29, 2014.
Amelia Perry, Alexander S Wein, Afonso S Bandeira, and Ankur Moitra. Optimality and sub- optimality of pca i: Spiked random matrix models. The Annals of Statistics, 46(5):2416-245 2018.
11

---

Under review as a conference paper at ICLR 2023
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga. Henrik Marklund, et al. Extending the wilds benchmark for unsupervised adaptation. arXiv preprint arXiv:2112.05000 2021.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in Neural Information Processing Systems, 31, 2018.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
Mohamed EI Amine Seddik, Cosme Louart, Mohamed Tamaazousti, and Romain Couillet. Random matrix theory proves that deep learning representations of gan-data behave gaussian mixtures. In International Conference on Machine Learning, pp. 8573-8582, PMLR, 2020.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1532-1540, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 28 2818-2 1-28626, 2016.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebuffi, Ira Ktena, Taylan Cemgil, et al. A fine-grained analysis distribution shift. arXiv preprint 10. / 2021.
Jiqing Wu, Nanda Horeweg, Marco de Bruyn, Remi Nout, Ina Srggenliemk-cchulle Ludy CHW Lutgens, Jan J Jobsen, Elzbieta M van der Steen-Banasik, Hans W Nijman, Vincent THBM Smit, et al. Automated causal inference application to randomized controlled clinical trials. Nature Machine Intelligence, 2022a.
Jiqing Wu, Inti Zlobec, Maxime Lafarge, Yukun He, and Viktor H Koelzer. Towards iid representation learning and its application on biomedical data. arXiv preprint arXiv:2203.0083 00332, 2022b.
Lin Yue, Dongyuan Tian, Weitong Chen, Xuming Han, and Minghao Yin. Deep learning for heterogeneous medical data analysis. World Wide Web, 2355):275--273, 2020.
12


Reviews:
[{"review_id": "roB7I-zXjm", "reviewer": "Reviewer_RbUs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study two similiarty distances for distribution shift, $d_{FID}$ (Frechet Inception Distance) and the prposed $d_{Eig}$ (Sorted Eigenvalue Comparison). The authors first showed by slightly modifying the $d_{FID}$ algorithm, the computational time for $d_{FID}$ can be improved. Then, the authors provide better upper bounds for this method. Finally, inspired by this algorithm, the authors propose $d_{Eig}$ and justify its stability by eigenvalue rigidity.", "summary_of_the_review": "The paper is clear to follow, but however the main contribution of the paper is unclear. The first improvement trick seems to be not that hard, I found it hard to believe that no one has ever implemented in such way to save computational time--it would be more convincing that some well used library has been using $np.trace(sqrtm(S_1S_2))$ as an evidence. The second point: by comparing to upper bounds, it is not enough to say that one algorithm is better than the other; you need a lower bound or PAC-type of result to justify that. Finally, despite arguing $d_{Eig}$ is more stable alternative than $d_{FID}$, the empirical studies didn't show this. (Say in Fig 3, if you look at the relative error in the table, $d_{FID}$ has the smallest errors)\n\nI agree with the authors that $d_{Eig}$ seems to be computationally cheaper, but however it seems to lack the desired mathematical properties in evaluation. As the contribution of the paper isn't clear for me, I tend to reject the paper for ICLR.", "strength_and_weaknesses": "Strength: $d_{Eig}$ is much cheaper to compute than $d_{FID}$.\nWeakness:\n1. The trick to accelerate $d_{FID}$ seems not hard.\n2. Comparing to upper bounds. (See Summary of Review)\n3. $d_{FIG}$ in experiment isn't more stable and it lacks of some desired properties.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "myGMeiMx2Hp", "reviewer": "Reviewer_28R5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed an improved Fréchet Inception Distance estimator (d_FID), as well as a sorted eigenvalue distance estimator (d_Eig), between two sample covariance matrices (SCMs). \nComputational error bounds were established for these two methods, with a huge saving in the running time.\nThe authors then provided a few (statistical) arguments from random matrix theory to characterize the behavior of the largest eigenvalues of SCMs in Section 3.1.\nThe proposed d_Eig is then applied, as a simple alternative to d_FID, in Section 3.2, as the scores of GANs.\n", "summary_of_the_review": "Detailed comments: \n\n* some derivations in Section 2 are somewhat elementary. These are not enough to be the major contribution of the paper.\n* Section 2.2.1: error bound of eigvals(): in the first sentence of this paragraph, s_j is not yet defined? or perhaps I missed something？\n* Section 2.2.2 considers a very special case: I am wondering if there exists any practical situation where such a special case holds.\n* Theorem 10 holds under the stability condition (Bao et al., 2015, Condition 1.1 (iii)) which should be stated explicitly. Also, it remains unclear whether Theorem 10 is a novel result.\n* I do not understand why it is of interest to consider the \"spike\" behavior in Sec 3.1, and in which context.\n* some numerical results, e.g., in Figure 2(c) and in Figure 3 are hardly visible: Not sure if they should be put in the appendix.\n* in Section 3.2, the concrete application in GAN is considered. However, it remains unclear how the proposed improvement is important from a GAN perspective.\n* No proof is given: this makes me feel that the results presented in the paper are all existing results. And no code is released for the numerical experiments.", "strength_and_weaknesses": "**Strength**: the paper is in general clearly written. And the problem under study is of significance.\n\n**Weaknesses**: some efforts are needed to better highlight the contribution of this paper. See my detailed comments below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cNtmdUxgEiM", "reviewer": "Reviewer_7tys", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents two formulas to compute the Fréchet Inception Distance (FID) between real/generated datasets. \nThe authors provide some worst-case error analysis coupled with practical details and eigenvalue sorting. Numerical \nexperiments demonstrate the effectiveness of the proposed schemes. ", "summary_of_the_review": "See my answer to \"Strength and weaknesses\".", "strength_and_weaknesses": "Before I post my summary, I want to suggest the authors to flip the orientation of the x-axis. It took me several minutes until I realize that the x-axis increases along the left direction. Normally, you would expect the number of samples to increase along the right direction, and the opposite is really confusing.\n\nThe main strength of the paper is the experimental evaluation where the authors demonstrate benefits of the eigenvalue-based formulation. In particular, d_{eig} can perform well with fewer data samples than d_{fid}. Nonetheless, my main concern is that a lot of the things discussed in the paper are straightforward. For example, there is no need to compute the eigenvalues of S_1S_2 using a non-symmetric eigenvalue solver (this is claimed as a drawback). This seems to be one of the main reasons why the proposed d_{eig} does better; but when S_1 is SPD, the pencil (S2,S_1^{-1}) is also SPD, and there is no need to use a non-symmetric eigenvalue solver. \nMoreover, computing the square root of a matrix product is asymptotically equivalent to that of an eigenvalue decomposition, although I do agree that computing eigenvalues seems cheaper indeed; especially if someone uses the definition of d_{fid} in (1) which requires way too many floating-point operations. Section 2.2.2 holds when the two covariance matrices commute. Is this happening in practice? If not, the definitions in 2.2.2 become heuristics. Are these heuristics the ones reported in the experiments? I can tell that this seems to be the case in Section 3.2, in which case the authors do not compute the exact FID. \n\nOverall, I think the paper is well-written and clear. While I support the idea, many of the modifications the authors present are relatively incremental (for example, (2) is well known in the literature).  \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MrteuxDYd7", "reviewer": "Reviewer_6Rmw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies variations on the Frechet Inception Distance $d_{FID}$.\nFirst, the authors show how we can implement between two datasets by using\nmore numerically stable eigenvalue subroutines instead of a full\nmatrix square root method. This leverages the fact that the covariance matrices\nfor the two sets of samples is PSD. This observation leads to practical running\ntime improvements. Second, the authors then propose the \"Sorted eigenvalue\ncomparison\" distance, which is motivated by FID when the two covariance matrices\nare diagonalized by the same orthgonal matrix $Q$. The remainder of this work\nexplore the implications of using $d_{Eig}$ more broadly.", "summary_of_the_review": "This work studies a clean mathematical distances and begins to explore (1) how\nit can be computed faster in practice and (2) variants that also fast to\ncompute while being theoretically sound. I do not recommend that it be accepted\nto ICLR 2023 given the overall quality of the paper, arguments, and experiments.\nThe work is interesting, but it needs to be developed quite a bit further\nbefore being accepted to a conference of this caliber.", "strength_and_weaknesses": "**Strengths:**\n- The Frechet inception distance (FID) is widely used.\n  Heusel et al., (NeurIPS 2017) proposed using FID for GANS, which has kicked off a massive amount of subsequent research on distances between distributions that are useful in practice.\n- The derivation of $d_{Eig}$ as an alternative to $d_{FID}$ is a nice contribution and could be studied more deeply.\n\n**Weaknesses:**\n- When introducing FID, it would also be valuable to draw connections to the\n  Wasserstein distance-based definition (and equivalently couplings).\n- The runtime reductions of 25% and 90% need more context (e.g., does this come\n  from an improve constant factor in the running time, or is it an artifact of\n  your data/implementation/machine?).\n- A more detailed discussion about the differences between `scipy.linalg.sqrtm`\n  and `np.linalg.eigvals` under the hood is needed? Don't both compute the Schur decomposition?\n- Paper organization: Introduce all notation in a single section. Splitting\n  between the beginning of Section 2 and Section 3 is non-standard. I also\n  recommend putting all empirical results at the end of the paper.\n- Figure 2a is a good experiment, but I think these plots would be easier to\n  parse if the dimension increased from left to right, and if the number of\n  samples on the x-axis increased from left to right (i.e., reverse everything).\n  Moreover, I think language like \"Approximation quality\" or \"Numerical stability\"\n  would be more helpful than \"Oracle comparison\".\n- [page 6] The paper could benefit from a couple more sentences between Theorem\n  10 about why the stability condition in Bao et al. (2015) is reasonable.\n- The paper could benefit from fewer experiments, where each\n  experiment has a little more substance and deeper theoretical backing (or\n  connections to experiments in other related works).\n\n**Suggestions:**\n- [page 1] The notation $\\lambda^{(1)}_{j}$ would better differentiate $S_1$ and $S_2$.\n- [page 2] Suggestion: \"the proposed $d_{FID}$ section\" --> use \\Cref{} to give actual section.\n- [page 2] It would be useful to explicitly say $S_{i} \\in \\mathbb{R}^{p \\times p}$ to\n  make the orientation of your vectors clear.\n- [page 3] suggestion: Add citation for the uniqueness property in Theorem 2.\n- [page 3] suggestion: Add proof of Lemma 4 to the appendix for completeness.\n- [page 4] The notation $\\lesssim O(\\epsilon)$ is redundant/imprecise.\n- [page 5] suggestion: Format the parenthesis in Eq (5) so that they're at\n  least as tall as the sqrt symbols.\n- [page 8] All of the tables should use tabular environments instead of Numbers\n  screenshots.\n- [page 9] suggestion: Discuss related works as last section of the\n  introduction.\n- [page 9] typo: \"as an example(Fremond et al.,...)\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject