The Bures Metric for Taming Mode Collapse in Generative Adversarial Networks
Authors: Hannes De Meulemeester
Joachim Schreurs
Michaël Fanuel
Bart De Moor
Johan Suykens
Venue:  (2021)
Keywords: Generative Adversarial Networks
Deep Learning
Neural Networks

Abstract: Generative Adversarial Networks (GANs) are performant generative methods yielding high-quality samples. However, under certain circumstances, the training of GANs can lead to mode collapse or mode dropping, i.e. the generative models not being able to sample from the entire probability distribution. To address this problem, we use the last layer of the discriminator as a feature map to study the distribution of the real and the fake data. During training, we propose to match the real batch diversity to the fake batch diversity by using the Bures distance between covariance matrices in feature space. The computation of the Bures distance can be conveniently done in either feature space or kernel space in terms of the covariance and kernel matrix respectively. We observe that diversity matching reduces mode collapse substantially and has a positive effect on the sample quality. On the practical side, a very simple training procedure, that does not require additional hyperparameter tuning, is proposed and assessed on several datasets.  

Under review as a conference paper at ICLR 2021
THE BURES METRIC FOR TAMING MODE COLLAPSE IN GENERATIVE ADVERSARIAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Generative Adversarial Networks (GANs) are performant generative methods yielding high-quality samples. However, under certain cireumstances, the training of GANs can lead to mode collapse or mode dropping, i.e. the generative models not being able to sample from the entire probability distribution. To address this problem, we use the last layer of the discriminator as feature map to study the distribution of the real and the fake data. During training, we propose match the real batch diversity to the fake batch diversity by using the Bures distance between covariance matrices in feature space. The computation of the Bures distance can be conveniently done in either feature space or kernel space in terms of the covari- ance and kernel matrix respectively. We observe that diversity matching reduces mode collapse substantially and has positive effect on the sample quality. On the practical side, a very simple training procedure. that does not require additional hyperparameter tuning, is proposed and assessed on several datasets.
INTRODUCTION
In several machine learning applications, data is assumed to be sampled from an implicit probability distribution. The estimation of this empirical implicit distribution is often intractable, especially in high dimensions. To tackle this issue, generative models are trained to provide an algorith- mic procedure for sampling from this unknown distribution. Popular approaches are Variational Auto-Encoders proposed by Kingma & Welling (2014), Generating Flow models by Rezende & Mohamed (2015) and Generative Adversarial Networks (GANs) initially developed by Goodfellow et al. (2014). The latter are particularly successful approaches to produce high quality samples, especially the case of natural images, though their training is notoriously difficult. The vanilla GAN consists of two networks: generator and discriminator. The generator maps random noise, usually drawn from multivariate normal, fake data in input space. The discriminator estimates the likelihood ratio of the generator network to the data distribution. It often happens that GAN generates samples only from few of the many modes of the distribution. This phenomenon is called 'mode collapse'.
Contribution. We propose BuresGAN: a generative adversarial network which has the objective function of vanilla GAN complemented by additional term, which is given by the squared Bu- res distance between the covariance matrix of real and fake batches in a latent space. This loss function promotes matching of fake and real data in feature space IR. so that mode collapse is reduced. Conveniently, the Bures distance also admits both feature space and kernel based expression. Contrary other related approaches such as in Che al. (217)) or Srivastava et al. (2077). the architecture of the GAN is unchanged, only the objective is modified. A variant called alt-BureGGAN, which is trained with alternating minimization, achieves competitive performance with a simple training procedure that does not require hyperparameter tuning or an additional reg- ularization such as gradient penalty. We empirically show that the proposed methods are robust when it comes to the choice of architecture and do not require an additional fine architecture search. Finally, an extra asset of BuresGAN is that it yields competitive or improved IS and FID scores compared with the state of the art on CIFAR- 10 and STL-10 10 using ResNet architecture.
Related works. The Bures distance is closely related to the Fréchet distance (Dowson & Lan- dau 1982) which is 2-Wasserstein distance between multivariate normal distributions. Namely,

---

Under review as a conference paper at ICLR 2021
the Fréchet distance between multivariate normals of equal means is the Bures distance between their covariance matrices. The Bures distance is also equivalent to the exact expression for the 2- Wasserstein distance between two elliptically contoured distributions with the same mean as shown in Gelbrich (1990) and Peyré et al. (201)). Noticeably, the Fréchet Inception Distance score (FID) popular manner to assess the quality of generative models. This score uses the Fréchet distance between real and generated samples in the feature space of a pre-trained inception network as it is explained in Salimans et al. a (2016) and Heusel al. (2017).
There exist numerous works aiming to improve training efficiency of generative networks. For mode collapse evaluation, we compare BuresGAN the most closely related works. GDPP-GAN (Elfeki et al. (019)) and VEEGAN (Srivastava et al.) (017)) also try to enforce diversity in 'latent' space. GDPP-GAN matches the eigenvectors and eigenvalues the real and fake diversity kernel. In VEE- GAN, an additional reconstructor network is introduced to map the true data distribution to Gaussian random noise. In a similar way, architectures with two discriminators are analysed by Nguyen et al.) (017), while MADGAN (Ghosh et al. 2018 uses multiple discriminators and generators. A differ- ent approach taken by Unrolled-GAN (Metz et al. (017) which updates the generator with respect to the unrolled optimization of the discriminator. This allows the training
be adjusted between using the optimal discriminator in the generator's ' objective, which is ideal but infeasible in practice. Wasserstein GANs (Arjovsky et al.) a (2017] Gulrajani al., (017) leverage the 1 I-Wasserstein distance to match the real and generated data distributions. In MDGAN (Che et al.] 2017). a regularization added to the objective function, so that the generator can take advantage of another similarity metric with more predictable behavior. This idea is combined with a penalization of the missing modes. More recent approaches redueing mode collapse are variations of WGAN (Wu et al., 2018). Entropic regularization has been also proposed in PresGAN (Dieng et al. 2020). while met- ric embeddings were used in the paper introducing BourGAN (Xiao et al. 2018). A simple packing procedure which significantly reduces mode collapse was proposed in PacGAN (Lin et al. 2018 that we also consider hereafter in our comparisons.
METHOD
GAN consists of a discriminator D Rd - R and a generator G R Rd which are typically defined by neural networks and parametrized by real vectors. The value D(x) gives the probability
that x comes from the empirical distribution, while the generator G maps a point in the latent space Rf to point in the input space Rd The training of GAN consists in solving
min max Ex [log D(=)] + Em-p, 1og(1 D(x))).
(1)
G D
by alternating two phases of training. In equation [ the expectation in the first term is over the empirical data distribution pa, while the expectation in the second term is over the generated data distribution Pg. implicitly given by the mapping by G of the latent prior distribution NO0, Te). is common to define and minimize the discriminator loss by
VD = - [lg D(æ)] - Ea-r, [log(1 - D(x))].
(2)
In practice, it is proposed in Goodfellow al. (0144) to minimize generator loss Va - -E (log (=G(=)))
rather than the second term of equation D for an improved training efficiency.
Matching real and fake data covariance. To prevent mode collapse, we encourage the generator to sample fake data of similar diversity to the real data. This is achieved by matching the sample covariance matrices of the real and fake data respectively. Covariance matching and similar ideas were explored for GANs in Mrouch et al. (2017) and [lleki et al. (2019). In order to compare covariance matrices, we propose to use the squared Bures distance between positive semi-definite é x matrices (Bhatia et al.) 2011).110 i.e.,
B (A,
min (EO())
|/2 11/2 Tr(A+ B 2(A BAb)
Being a Riemannian metric on the manifold of positive semi-definite matrices Massart & Ab- sil 2020), the Bures metric is adequate to compare covariance matrices. The covariances
2

---

Under review as a conference paper at ICLR 2021
defined in feature space associated to the discriminator. More precisely, the last layer of the discriminator, denoted by Φ(x) E R/, defines a feature map, namely D(z) = a(w d(x)), where w is the weight vector of the last dense layer and a is the sigmoid function. We use the normalization Φ(x)) = (z))) after the centering of Φ(x). Then, we define a co- variance matrix as follows: C(p) Ee (()))((()))))))))))))))))))))))) J. For simplicity, we denote the real data and generated data covariance matrices by Ca = C(pa) and C C(pg). respectively. Our proposal is to replace the generator loss by VG + AB(Ca, Cg)² The value = 1 was pro-
found to yield good results in the studied datasets. Two specific training algorithms posed. Algorithm | deals with the squared Bures distance as an additive term to the generator loss, while alternating training is used in Algorithm 2 and does not introduce extra parameter.
Algorithm 1: BuresGAN
Algorithm 2:
Alt- Bl-BBureGGAN
Sample real and fake batch;
Sample real and fake batch:
Update G by minimizing
Update G by minimizing B(Ć,, C₁)²; Update G by minimizing Va;
VG + AB(C,, Cg)²: Update D by maximizing VV:;
Update D by maximizing VVD:
The training described in Algorithm is analogous to the training of GDPP GAN, although the ad- ditional generator loss is rather different. The computational advantage of the Bures distance is that admits two expressions which can be evaluated numerically in a stable way. Namely, there is no need calculate gradient update through an eigendecomposition.
Feature space expression. In the training procedure, real x! and fake data (9) with i = 1, 1,....b
sampled respectively from the empirical distribution and the mapping of the normal distribution N Le) by the generator. Consider the case where the batch size b is larger than the feature space dimension. Let the embedding of the batches in feature space be 4. {
(a)
Φ(2, (at) )
Rix with a = d.g. g. The covariance matrix of one batch in feature space is C = T5 T T6 where e is the & eo-norallized centered feature map of the batch. Numerical instabilities can be avoided by adding a small number, e.g. 14, to the diagonal elements of the covariance matrices, so that, in practice, we only deal with strictly positive definite matrices. From the computational perspective, an interesting alternative expression for the Bures distance is given by
(4) whose computation requires only one matrix square root. This identity can be obtained from Lemma 1 Note that analogous result is proved in Oh et al. (2020).
B (Ca,, C,)² Tr(Ca + C. 2(C,
Lemma 1. Let A and B be f x f symmetric positive semidefinite matrices and let B = YTY Then, we have: (i) AB is diagonalizable with nonnegative eigenvalues, and (ii) Tr((AB)²) = TY)(Y
Kernel based expression. Alternaively, if the feature space dimension f is larger than the batch size b, it is more efficient to compute B(Ca, Cg) thanks to b x b kernel matrices: Ka T
Kg = $ T and Kdg T Then, we have the kernel based expression
B(Ca, cg (Ka Kg
(5)
which allows to calculate the Bures distance between covariance matrices by computing matrix square root of a b x matrix. This a consequence of Lemma 2
Lemma 2. The matrices X" XY Y and X T are diagonalizable with nonnegative eigen- values and share the same non-zero eigenvalues.
Connection with Wasserstein GAN and integral probability metrics. The Bures distance is proportional to the 2- Wasserstein distance W2 between two ellipically contoured distributions, with the same mean (Gelbrich 1990). For instance, in the case of multivariate normal distributions, we have
B (), = min E (x.)aa X YI s.t. X ~ N((,,)) A) and ~ (O, B),
For simplicity, we omit the normalization by 5 in front of the covariance matrix.
3

---

Under review as a conference paper at ICLR 2021
where the minimization is over the joint distributions π. More precisely, in this paper, we make the approximation that the implicit distribution of the real and generated data in the feature space R/ (associated to ((x)) are elliptically contoured with the same mean. Under different assumptions, the Generative Moment Matching Networks (Ren et al.) 2016 Li et al. 2017) work in the same spirit, but the Kantorovich dual formula for the I-Wasserstein distance: Wi B) = seLip J fd 3), where a, 3 are signed measures. Generalizations of such integral formulae are called integral probability metrics (see for instance [Binkowski et al. (2018)). Here, f is the discriminator, so that the maximization over Lipschitz functions f plays the role of the maximization over discriminator parameters in the min-max game of equation [ Then, in the training procedure, this maximization alternates with a minimization over the generator parameters.
different approach to match covariance matrices. On the contrary, WGAN uses
We can now discuss the connection with Wasserstein GAN. Coming back to the definition of Bures- GAN, we can now explain that the 2-Wasserstein distance provides an upper bound on an integral probability metric. Then, we assume that the densities are elliptically contoured distributions in feature space, the use of the Bures distance to calculate W/2 allows to spare the maximization over the discriminator parameters - and this motivates why the optimization of only influences updates of the generator in Algorithm [ and Algorithm 2 Going more into detail, the 2-Wasserstein dis- tance between two probability densities (w.r.t. the same measure) is equivalent to a Sobolev dual norm, which can be interpreted as an integral probability metric. Indeed, let the Sobolev semi- norm || (f IIV (())) dx)a/2, Then, its dual norm over signed measures is defined as || = fl dv. It is then shown in Peyre (2018) and Peyré et al. (2019) that there exist two positive constants 0₁ and C2 such that
ei cilla -1 S W2(a. S c2lla- 81
Hence, the 2-sasserseiin distance gives an upper bound on an integral probability metric.
Algorithmic details. The matrix square root equation and equation 5 is obtained thanks to the Sewton-Schultz algorithm which is inversion free and can be efficiently calculated on GPUs since it involves only matrix products. In practice, we found 15 iterations of this algorithm to be sufficient for the small scale datasets, while 20 iterations were used for the ResNet examples. A small regularization term 1e- 14 is added for stability. The latent prior distribution is ( Ie) with f = 100 and the parameter in Algorithm is always set to X = 1. In the tables hereafter, we indicate the largest scores in bold, although we invite the reader to also consider the standard deviation.
3 EMPIRICAL EVALUATION OF MODE COLLAPSE
The BuresGAN and alt-BureSGAN performances on synthetic data, artificial and real images are compared with the standard DCGAN (Salimans et al. 2016), WGAN-GP, MDGAN, Unrolled GAN, VEEGAN, GDPP and PacGAN. We want to emphasize that the purpose of this experiment is not to challenge these baselines, but to report the improvement obtained by adding the Bures metric to the objective function. It would be straightforward to add the Bures loss to other GAN variants, as well as most GAN architectures, and we would expect improvement in mode coverage and generation quality. In the experiments, we notice that adding the Bures loss to the vanilla GAN already significantly improves the results.
A low dimensional feature space (f = 128) is used for the synthetic data so that the feature space formula in equation is used, while the dual formula in equation 5 is used for the image datasets (Stacked MNIST, CIFAR-10, CIFAR-100 and STL-10) for which the feature space is larger than the batch size. The architectures used for the image datasets are based on the DCGAN (Radford) et al. 2016). while results using ResNets are given in Section 4 All images are scaled in between and before running the algorithms. Additional information on the architectures and datasets given in Appendix. The hyperparameters of other methods are typically chosen as suggested in the authors' reference implementation. The number of unrolling steps in Unrolled GAN is chosen to be 5. For MDGAN, both versions are implemented. The first version, which corresponds to the mode regularizer, has hyperparameters A = 0.2 and X = 0.4, for the second version, which corresponds to manifold diffusion training for regularized GANS, has À 10 WGAN-GP uses À = 10.0 and Peritic = 5. All models are trained using Adam (Kingma & Ba 2015) with 3 = 0.5, B2 = 0.999 and learning rate 10-3 for both the generator and discriminator. Unless stated otherwise, the batch
4

---

Under review as a conference paper at ICLR 2021
size is taken to 256. Examples of random generations of all the GANs are given in Appendix. Notice that in this section we report the results achieved only at the end of the training.
3.1 ARTIFICIAL DATA
Synthetic. The ring is mixture of eight two-dimensional spherical Gaussians in the plane with means 2.5 x (cos((2R) (2x/8))), sin(((z and std 0.01 for E (1.....8), .8.. The 2D-grid a mixture of 25 wo-diieesional isotropic normals in the plane with means separated by 2 and with standard deviation 0.05. All models have the same architecture, with é = 256 following Elfeki et al. (2019), and are trained for 25k iterations. The evaluation is done by sampling 3k points from the generator network. A sample counted as high quality if it is within 3 standard deviations of the nearest mode. The experiments are repeated 10 times for all models and their performance is compared in T
BuresGANs consistently capture all the modes and produces the highest quality samples. The train- ing progress of the Alt-BuresGAN is shown on Figure I where we observe that all the modes early on in the training procedure, afterwards improving the quality. The training progress of the other GAN models listed in Table is given in Appendix. Although BuresGAN training times are larger than most other methods for this low dimensional example, we show Appendix D.T that Bures- GAN scales better with the input data dimension and architecture complexity.
Step
Step 2k
Step 4k
Step 6k
Step 8k
Step 25k
Figure 1: Figure accompanying Table I the progress of Alt-BuresGAN on the synthetic examples. Each column shows 3k samples from the training of the generator in blue and 3k samples from the true distribution in green.
Grid with 25 modes
Ring with 8 modes
Nb modes
% in 3a
time (s)
Nb modes
% 3a
time (s)
GAN
22.9(4)
76(13)
32(0.2)
7.4(2)
76(25)
32(0.4)
WGAN-GP
24.9(0.3)
77(10)
225(1)
7.1(1)
9(5)
224(1)
MDGAN-vI
21(3)
49(18)
65(0.2)
6.5(2)
65(26)
65(0.5)
MDGAN-v2
25(0)
68(11)
81(0.2)
5(3)
20(15)
81(0.8)
UnrolledGAN
19.7(1)
78(19)
98(0.2)
8(0)
77(18)
98(0.7)
VEEGAN
25(0)
67(3)
49(0.3)
8(0)
29(5)
49(0.5)
GDPP
20.5(5)
(9(23)
392(25)
7.5(0.8)
73(25)
404(5)
PacGAN2
23.6(4)
(5(28)
43(0.4)
8(0)
81(15)
43(0.6)
Alt-BuresGAN
25(0)
84(1)
298(0.6)
8(0)
84(6)
298(1)
BuresGAN
25(0))
82(1)
279(0.4)
8(0)
82(4)
278(1)
Table I: Experiments on the synthetic datasets with a GPU Nvidia Quadro P4000. Average(std) over 10 runs. All the models are trained for 25k iterations and the total training time indicated in seconds.

---

Under review as a conference paper at ICLR 2021
Stacked MNIST. The Stacked MNIST dataset is specifically constructed to contain 1000 known modes. This is done by stacking three digits, sampled uniformly at random from the original MNIST dataset, each in different channel. The BuresGAN models are compared to the other methods and are trained for 25k iterations. The results with different batch sizes can be found in Table 2 For the evaluation of the performance, number of captured modes measures mode collapse and the KL divergence, which also measures sample quality. The mode of each generated image is identified by using standard MNIST classifier which is trained up to 98.43% accuracy on the validation set (see Appendix), and classifies each channel of the fake sample. The same classifier is used count the number of captured modes. The metrics are calculated based on 10k generated images for all the models. Interestingly. for most models, an improvement is observed the quality of the images - KI. divergence and terms of mode collapse number of modes attained as the size of the batch increases. For the same batch size, architecture and iterations, the image quality is improved by BuresGAN, which is robust with respect to batch size and architecture choice. The other methods show higher variability over the different experiments. WGAN-GP has the best single run performance with discriminator with convolutional layers and has average superior performance when using discriminator with convolutional layers (see Table Table[2]jin Appendix) but sometimes fails to converge when increasing the number of discriminator layers by along with increasing the batch size. MDGANv2, VEEGAN, GDPP and WGAN-GP often have an excellent single run performance. However, when increasing the number of discriminator layers, the training of these models has a tendency to collapse more often as indicated by the large std. Vanilla GAN is one of the best performing models in the variant with layers. This indicates that, for certain datasets, careful architecture tuning can be more important than complicated training schemes. A lesson from Table 2 is that BuresGAN's mode coverage does not vary much if the batch size increases, although the KL divergence seems to be slightly improved. Generated samples from the Alt-BuresGAN are given in Figure 2 Since VEEGAN and PacGAN papers use a different setup, we also report in Table 12 the specifications of the different settings.
follow Metz al. (2017) and use the following metries: the
Stacked MNIST
CIFAR-10
CIFAR-100
9
9
2
4
$
7
7
7
9
5
4 9
3
7
4
7
6
0
7
4
0
3
2
4
8
3 4
4
8
9
9
4
8
4
Figure 2: Generated samples from trained Alt-BureGGAN, with DCGAN architecture
As it was observed by previous papers such as in Lin et al. (2018) and Xiao et al. (2018), even the vanilla GAN can achieve an excellent mode coverage on certain architectures. Since the results of Table 2 do not determine a clear winner, we repeated similar experiment with more challenging architecture (reported in Table T0p which includes convolutional layers for both the generator and discriminator in an analogous way to the experiment proposed in Srivastava et al. (2017). The leading methods of Table 2 are considered for this experiment. Then, we observe Table that vanilla GAN and GDPP collapse for this architecture. Compared with the results of Table the difference between the methods is more significant: WGAN-GI yields the best result and is followed by the BuresGAN models. However note that our empirical results indicate that WGAN-GP is sensitive to the choice of architecture and hyperparameters and its training time is also longer as can be seen from Table 19 in Appendix.

---

Under review as a conference paper at ICLR 2021
Nb modes(t)
KL div.(L)
Batch size
64
128
256
64
128
256
GAN
993.3(3.1)
995.4(1.7)
998.3(1.2)
0.28(0.02)
0.24(0.02)
0.21(0.02)
WGAN-GP
980.2(57)
(38.3(219)
785.1
0.26(0.34)
1.05(1)
1.6(2.4)
3 conv. layers
MDGAN-vl
233.8(250)
204.0(202)
215.5(213)
5.0(1.6)
4.9(1.3)
5.0(1.2)
MDGAN-v2
299.9(457)
300.4(457)
200.0(398)
4.8(3.0)
4.7(3.0)
5.5(2.6)
UnrolledGAN
934.7(107)
874.1(290)
884.9(290)
0.72(0.51)
0.98(1.46)
0.90(1.4)
VEEGAN
971.2(10.3)
687.9(117)
395.6(166)
0.33(0.05)
2.01(2.61)
3.52(2.61)
GDPP
894.2(298)
897.1(299)
997.5(1.4)
0.92(1.92)
0.88(1.93)
0.20(0.02)
PacGAN2
989.8(4.0)
99..3(4.8)
897.7(299)
0.33(0.02)
0.29(0.04)
0.87(199)
BuresGAN
993.5(2.7)
996.3(1.6)
997.1(2.4)
0.29(0.02)
(.25(0.02)
0.23(0.01)
Ali-BuresGAN
995.0(2.4)
995.6(3.6)
995.5(2.4)
0.28(0.03)
0.27(0.02)
(.24().02)
Table KL--divergence between the generated distribution and true distribution for an architecture with 3 conv. layers for the Stacked MNIST dataset. The number of counted modes assesses mode collapse. The results are obtained after 25k iterations and we report the average(std) over runs.
Nb modes(1)
KI. d(v.
GAN
21.6(25.8)
5.10
WGAN-GI GP
999.7(0.6)
0.11(0.006)
4 conv. layers
MDGAN-v2
729.5(297.9)
1.76(1.65)
UnrolledGAN
24.3(23.61)
4.96(0.68)
VEEGAN
816.1(269.6)
1.33(1.46)
GDPP
33.3(39.4)
4.92(0.80)
PacGAN2
972.4(12.0)
0.45(0.06)
BuresGAN 989.9(4.77
0.38(0.06)
Alt-BuresGAN
990.0(4..9
0.33(0.04)
Table 3: Stacked MNIST experiment for an architecture with 4 conv. layers. All the models are trained for 25k iterations with batch size of 64, learning rate of 2e-4 for Adam and normal latent distribution. The evaluation is over 10k samples and we report an average(std) over 10 runs.
3.2 REAL IMAGES
Metrics. The image quality is assessed thanks to the Inception Score (IS), Fréchet Inception Dis- tance (FID), Inference via Optimization (IvO) and Sliced Wasserstein Distance (SWD). The latter was also used in [lleki et al. (2019) and Karras et al. (2017) to evaluate the quality of images as well as the severity of mode-collapse. In a word, SWD evaluates the multiscale statistical similarity between distributions of local image patches drawn from Laplacian pyramids. A small Wasserstein distance indicates that the distribution of the patches is similar, thus real and fake images appear similar both appearance and variation at this spatial resolution. IvO (Metz al. 2017) measures mode collapse by comparing real images with the nearest generated image. It relies on an optimiza- tion procedure within the latent space to find the closest generated image given test image, and returns the distance which can be large in the case of mode collapse. The metrics are calculated based on 10k generated images for all the models.
CIFAR datasets. In Table 4 we evaluate the performance on the 32 x 32 x 3 CIFAR datasets. While all models are trained for 100k iterations, the best performance observed for BuresGAN and Alt-BuresGAN in terms of image quality, measured by FID and Inception Score. and in terms of mode collapse, measured by SWD and IvO. We also notice that UnrolledGAN, VEEGAN and WGAN-GP have difficulty converging to satisfactory result for this architecture. This in contrast to the 'simpler' synthetic data and the Stacked MNIST dataset, where the models get comparable performance to BuresGAN and Alt-BureSGAN. In Arjovsky al. (2017). WGAN-GP achieves a very good performance on CIFAR-I0 with a ResNet architecture which is considerably more complicated than the DCGAN used here. Therefore, results with a Resnet architure reported in Section 4 Also, for this architecture and number of training iterations, MDGAN-v1 and MDGAN- v2 did not converge to meaningful result in our simulations.
7

---

Under review as a conference paper at ICLR 2021
CIFAR-I0
CIFAR-100
IvO(L)
18(1)
FID(I)
SWD(
IvO(U)
18(1)
FID(L)
SWD())
GAN
0.30(0.06)
(.62().22)
59(8.5)
3.7(0.9)
0.37(0.10)
5.2(1.1)
91.7(66)
7.8(4.9)
WGAN-GP
0.49(0.24)
2.01(0:47)
291(87)
8.3(1.9)
0.54(0.27)
1.2(0.5)
283(113)
9.7(2.5)
UnrolledGAN
0.36(0.08)
3.1(0.6)
148(42)
9.0(5)
0.44(0.10)
3.2(0.7)
172.9(40)
13.1(9.2)
VEEGAN
0.36(0.08)
2.5(0.6)
198(33.5)
12.0(3)
0.35(0.08)
2.8(0.7)
177.2(27)
12.8(3.9)
GDPP
(.29().05)
6.76(0.27)
62.1(5.5)
4.1(1.1)
(.37()
6.9(0.2)
65.018)
4.4(1.9)
PacGAN2
1.00(0.35)
(.51(0.18)
60.61(5.9)
2.95(1)
0.97(0.35)
5.6(0.1)
59.9(5.2)
4.0(18)
BuresGAN
0.30(0.03)
6.34(0.17)
43.7(0.9)
2.1(0.6)
0.36(0.08)
6.5(0.1)
47.2(1.2)
21((1.)
Alt-BuresGAN
(.29))
6.23(0.07)
45.4(2.8)
1.7(0.9)
0.33((0.5)
6.4(0.1)
49.4(3.4)
1.8(0.6)
Table Generation quality on CIFAR-10 and CIFAR-100 with DCGAN architecture. Average(std) over 10 runs. 100k iterations for each. For improving readability, SWD score was multiplied by 100.
CIFAR-100 dataset consists of 100 different classes and is therefore more diverse. Compared to the original (IFAR-10 dataset, the performance of the proposed GANs remains almost the same, with small increase IvO. An exception is vanilla GAN, which shows higher presence of mode collapse as measured by IvO and SWD.
STL-100 The STL-10 dataset includes higher resolution images of size 96 x 96 3. The best performing models from previous experiments are trained for 150k iterations. Samples of generated images from trained Alt-BuresGAN are given on Figure 3 The metries are calculated based on 5k generated images for all the models. Compared to the previous datasets, GDPP and vanilla GAN are rarely able to generate high quality images on the higher resolution STL-10 dataset. Only BuresGANs are capable of consistently generating high quality images well preventing mode collapse, for the same architecture.
Figure 3: Images generated by BuresGAN with a ResNet architecture for CIFAR-10 (left) and STL- 10 (right) datasets. The STL-10 samples are full-sized 96 x 96 images.
Timings. The computing times for these datasets are in Appendix. For the same number of iter- ations, Balt-JBureSGAN training time is comparable to WGAN-GP training for the simple data in Table T For more complicated architectures, (alt-) Bures)Br scales better and the training time was observed to be significantly shorter with respect to WGAN-GP and several other methods.
4 HIGH QUALITY GENERATION USING A RESNET ARCHITECTURE
As noted by Lucic et al. (018). a fair comparison should involve GANs with the same architec- ture, and this why restricted in paper classical DCGAN architecture. It is natural to
8

---

Under review as a conference paper at ICLR 2021
IvO(L)
IS(1)
FID())
SWD())
GAN
0.50(0.15)
2.9(1.8)
237(54)
12.3(4.1)
GDPP
0.46(0.09)
3.3(2.2)
232(84)
8.2(4.0)
PacGAN2
0.97(0.05)
4.7(1.5)
(66
8.1(4.3)
BuresGAN
0.44(0.05)
7.6(0.3)
109(7)
2.3(0.3)
All-BuresGAN
0.45(0.04)
7.5(0.3)
110(4)
2.8(0.4)
Table 5: Generation quality on STL-10 with DCGAN architecture. Average(std) over runs. 150k iterations for each. SWD score was multiplied by 100 for improving readability.
question the performance of BuresGAN with ResNet architecture. Hence, we trained BuresGAN on the CIFAR-I0 and STL-10 datasets by using the ResNet architecture taken from Gulrajani al. (017)). In this section, the STL-10 images are rescaled to 48 x 48 x 3 according the procedure described in MMyato al. (2018) Lee al. 2 Wang al. 2 ), that the comparison of IS and FID scores with other works is meaningful. Note that BuresGAN has no parameters to tune, except for the hyperparameters of the optimizers. The results are displayed in Table 6 where the scores of state-of-ther In contrast with Section we report here the best performance achieved at any time during the train- ing, averaged over several runs. To the best of our knowledge, our method achieves a new state of the art inception score on STL-10 and is within a standard deviation of state of the art on CIFAR- 10 using a ResNet architecture. The FID score achieved by BuresGAN is nonetheless smaller than the reported FII) scores for GANs using a ResNet architecture. A visual inspection
unconditional GAN models with ResNet architecture are also reported.
the generated images in Figure g shows that the high inception score is warranted, the samples clear, diverse and often recognizable. BuresGAN also performs well on the full-sized STL-10 data set where an inception score of 11.11 + 0.19 and an FID of 50.9 0.13 is achieved (average and std over runs).
CIFAR-10
STL-10
IS (f)
FID (L)
IS (T)
FID (1)
WGAN-GP ResNct
(Gulrajani ct al. 2017]
7.86(0.07)
18.8t
InfoMax-Gan GANN [Lee
2019
8.08(0.08)
17.14(0.20)
8.54(0.12)
37.49(0.05)
SN-GAN ResNeT Mryato
2018
8.22(0.05)
21.7(0.21) /
9.10(0.04)
40.1(0.50)
ProgressiveGAN Karras el al. 2017
8.80().05)
CR-GAN (Zhang et al. 2020
8.4
14.56
NCSN (Song & Ermon
2019
8.87(0.12)
25.32
Improving MMD GAN (Wang al. 2019
8.29
16.21
9.34
37.63
WGAN-div (Wu et al. 2018)
18.1
BuresGAN ResNet (Ours)
8.81(0.08)
(2.91(0.40)
9.67(0.11)
31.42(1.01)
Best achieved IS and FID, using a ResNet architecture. Results with an asterisk are quoted from their respective papers (std in parenthesis). BuresGAN results were obtained after 300 itera- tions and averaged over 3 runs. The result indicated with are taken from Wu al. (2018). For all the methods, the STI.110 images are rescaled to 48 x 48 x 3 contrast with Table
Table 6:
CONCLUSION
In this work, we discussed an additional term based on the Bures distance whichpromotes a matching of the distribution of the generated and real data in feature space R The Bures distance admits both a feature space and kernel based expression, which makes the proposed model time and data efficient when compared to state of the art models. Two training procedures are proposed: Algorithm deals with the squared Bures distance as an additive term to the generator loss, while an alternating training is used in Algorithm 2 so that no extra parameter is introduced. Our experiments show that the proposed methods are capable of reducing mode collapse and, on the real datasets, achieve a clear improvement of sample quality without parameter tuning and without the need for regularization such as a gradient penalty. Moreover, the proposed GANs show a stable performance over different architectures, datasets and hyperparmeters.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Net- works. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Rajendra Bhatia, Tanvi Jain, and Yongdo Lim. On the definite matrices. Expositiones Mathematicae, 37(2):165 191, 2019.
distance between positive
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying. MMD GANs. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjie Li. Mode Regularized Gener- ative Adversarial Networks. In Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2017.
Adji B. Dieng, Francisco J. R. Ruiz, David M. Blei, and Michalis K. Titsias. Prescribed generative adversarial networks. arxiv:1910.04302, 2020.
D.C Dowson and B.V Landau. The Fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis, 12(3):450- - 455, 1982.
Mohamed Elfcki, Camille Couprie, Morgane Riviere, and Mohamed Elhoseiny. GDPP: Learning Diverse Generations using Determinantal Point Processes. In Proceedings of the 36th Interna- tional Conference on Machine Learning (/CML),2015
Matthias Gelbrich. On a Formula for the L2 Wasserstein Metric between Measures on Euclidean and Hilbert spaces. Mathematische Nachrichten, 147(1):185-21) 1990.
Arnab Ghosh, Viveka Kulharia, Vinay Namboodiri, Philip H.S. Torr, and Puneet K. Dokania. Multi-agent diverse generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Infor- mation Processing Systems 27, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of Wasserstein GANs. In Advances in neural information processing systems 31, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to Local Nash Equilibrium. In Advances in Neural Information Processing Systems 30, 2017.
Yoopyo Hong and Roger A. Horn. The Jordan Canonical Form of a Product of a Hermitian and a Positive Semidefinite Matrix. Linear Algebra and its Applications, 147:373 386, 1991.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Diederik P Kingma and Max Welling. Auto-encoding Variational Bayes. In Proceedings of the International Conference Learning Representations (ICLR), 2014.
Kwot Sin Lee, Ngoc-Trung Tran, and Ngai-Man Cheung. Infomax-gam gan: Mutual information maxi- mization for improved adversarial image generation. In NeurIPS 2019 Workshop on Information Theory and Machine Learning, 2019.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabas Poczos. MMD GAN: Towards Deeper Understanding of Moment Matching Network. Advances in Neural Informa- fion Processing Systems 30, 2017.
10

---

Under review as a conference paper at ICLR 2021
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. In Advances in Neural Information Processing Systems 31. 2018.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In Advances in neural information processing systems, pp- 700-709, 2018.
Estelle Massart and P.-. Absil. Quotient Geometry with Simple Geodesics for the Manifold of Fixed-Rank Sesidive-Semidefiiit Matrices. SIAM Journal on Matrix Analysis and Applications, 41(1):171-198, 2020.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled Generative Adversarial Networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2018
Youssef Mroueh, Tom Sercu, and Vaibhava Goel. McGan: Mean and Covariance Feature Matching GAN. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative adversarial nets. In Advances in Neural Information Processing Systems, volume 30, 2017.
Jung Hun Oh, Maryam Pouryahya, Aditi Iyer, Aditya P. Apte, Joseph O. Deasy, and Allen Tannen- baum. A novel kernel Wasserstein distance on Gaussian measures: An application of identifying dental artifacts in head and neck computed tomography. Computers in Biology and Medicine, 120: 103731, 2020.
Gabriel Peyré, Marco Cuturi, et al. Computational Optimal Transport. Foundations and Trends in Machine Learning, 11(5-6):355-607, 2019.
Rémi Peyre. Comparison between W22 distance and / norm, and Localization of Wasserstein distance, ESAIM: COCV, 1489-144991101 2018
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
Yong Ren, Jun Zhu, Jialian Li, and Yucen Luo. Conditional Generative Moment-Matching Net- works. In Advances in Neural Information Processing Systems 29, 2016.
Danilo Jimenez. Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. Proceedings of the 32th International Conference on Machine Learning (ICML), 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs. In Advances in neural information processing systems 29, 2016.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11918-11930, 2019.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing Mode Collapse in GANs using Implicit Variational Learning. In Advances in Neural Information Processing Systems 30, 2017.
Wei Wang, Yuan Sun, and Saman Halgamuge. Improving MMD-GAN training with repulsive loss function. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Jiqing Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, and Luc Van Gool. Wasserstein Diver- gence for GANs. In Proceedings of European Conference on Computer Vision (ECCV), 2018.
11

---

Under review as a conference paper at ICLR 2021
Chang Xiao, Peilin Zhong, and Changxi Zheng. BourGAN: Generative Networks with Metric Em- beddings. Advances in Neural Information Processing Systems 32, 2018.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. Proceedings of the International Conference on Learning Representations (ICLR), 2020.
A PROOFS
Proof of Lemma (i) is consequence of Corollary 2.3 in (Hong & Horn 1991). (ii We now follow (Oh al., 2020). Thanks to (i), we have AB PDP- where and the columns of P contain the right eigenvectors of AB. Therefore, Tr((A)) Then, Y AAY is clearly diagonalizable. Let us show that it shares its nonzero eigenvalues with AB. We have ABP PD, so that, by multiplying of the left by Y, holds that (Y AY YPD. b) Similarly, suppose that we have the eigenvalue decomposition YAY Q QA. Then, we have BAYTQ Q y QA with = YTY. This means that the non-7ero eigenvalues of Y AY are also eigenvalues of BA.. Since A and B are symmetric, this completes the proof.
nonnegative diagonal
Tr(D=/2),
Proof of Lemma 2 Y Y.
The result follows from Lemma | and its proof, where A X and B
B DETAILS OF THE THEORETICAL RESULTS
Let A and B be symmetric and positive semi-definite matrices. Let A = U diag( (VAUT where U and À obtained thanks to the eigenvalue decomposition A U 'dia(A)U" We show here that the Bures distance between A and B is
B (A, B)² = min // B1/2U Tr(A + B 2(4 B/b) (EE(())
(6)
where (E)) is the set of f orthonormal matrices. We can simplify the above expression as follows
min UEO(E)
||A /2/2 Tr(A) Tr(B) max TTA4/23//2UU) UEO(E)
since T A2/2) 122)) Let the characteristic function of the set of orthonor- mal matrices be f(U) (U) that is, f(U) = U (()) and +o0 otherwise.
Lemma 3. The Fenchel conjugate of f(U) (U))(U) is (*(M) = || [| where the nuclear norm is || Tr(VMT M) and U, M EXE
Proof. The definition of the Fenchel conjugate with respect to the Frobenius inner product gives
(*) = sup TrUU M) S(U) = max Tr(U MM. UEQ(F)
UERiXE
Next we decompose M as follows: M = WWVT where W, V E (()) are orthogonal matrices and Σ is l x & diagonal matrix with negative diagonal entries, such that MM WwW and M M = V. Notice that the non zero diagonal entries of E are the singular values of M. Then,
max Tr M) max Tr(WEV = max Tr(EU',, UEO(E)
(EO(E)
D'EQ(E)
where we renamed U VTU. Next, we remark that Tr(EU') Tr (ia(()))")) Since by construction, Σ is diagonal with non negative entries the maximum is attained at U = 1l. Then, the optimal objective is Tr(E)
M).
By taking M A we obtain equation Notice that the role of and can be exchanged in equation 6 since U is orthogonal.
12

---

Under review as a conference paper at ICLR 2021
C TRAINING DETAILS
C.I
SYNTHETIC ARCHITECTURES
Following the recommendation in the original work (Srivastava al., 2017), the same fully- connected architecture is used for the VEEGAN reconstructor in all experiments.
Layer
Output
Activation
Layer
Output
Activation
Input 256
Input 2
Dense 128
tanh
Dense
128
tanh
Dense 128
tanh
Dense 128
tanh
Dense
2
Dense
Table 7: The generator and discriminator architectures for the synthetic examples.
Layer
Output
Activation
Layer
Output
Activation
Input
2
Input
Dense
128
tanh
Dense 128
tanh
Dense
128
tanh
Dense
256
Dense 256
tanh
Normal 256
Table 8: Respectively the MDGAN encoder model and VEEGAN stochastic inverse generator ar- chitectures for the synthetic examples. The output of the VEEGAN models samples drawn from a normal distribution with scale and where the location learned.
C.2 STACKED MNIST ARCHITECTURES
Layer
Output
Activation
BN
Layer
Output
Activation
BN
Input
100
Input
28, 28, 3
Dense
12544
ReLU
Yes
Conv
14, 14, 64
Leaky ReLU
No
Reshape
7, 7, 256
Conv
7,
Leaky ReLU
Yes
Conv"
7, 128
ReLU
Yes
Conv
4, 4, 256
Leaky ReLU
Yes
Conv'
14, 14, 64
ReLU
Yes
Flatten
Conv'
28, 28, 3
ReLU
Yes
Dense
Table 9: The generator and discriminator architectures for the Stacked MNIST experiments. The BN column indicates whether batch normalization is used after the layer or not. For the experiments with 2 convolution layers in Table 21, the final convolution layer removed in the discriminator.
13

---

Under review as a conference paper at ICLR 2021
Layer
Output
Activation
BN
Layer
Output
Activation
BN
Input
100
Input
28, 28, 3
Dense
2048
ReLU
Yes
Conv
14, 14, 64
Leaky ReLU
No
Reshape
2, 2, 512
Conv
Leaky ReLU
Yes
Conv
256
ReLU
Yes
Conv
256
Leaky ReLU
Yes
Conv'
ReLU
Yes
Conv
2,2,
Leaky ReLU
Yes
Conv'
14, 14, 64
ReLU
Yes
Flatten
Conv
28, 28,
ReLU
Yes
Dense
Table 10: The generator and discriminator architectures for the Stacked MNIST experiments with layers for both generator and discrimintor. The BN column indicates whether batch normalization used after the layer or not.
Layer
Output
Activation
BN
Input
28, 28, 3
Conv
14, 14,
ReLU
Yes
Conv
7, 7, 64
ReLU ReLU
Yes
Conv
7,7, 128
Yes
Flatten
Dense
100
Table 11: The MDGAN encoder model architecture for the Stacked MNIST experiments. The BN column indicates whether batch normalization is used after the layer not.
Parameters
BuresGAN
PacGAN
VEEGAN
GDPP
learning rates
x 10
& 2 x 10
2 10TH
2 10
X 10
learning rate decay
no
no
no
no **
Adam }1
0.5
0.5
0.5
0.5
Adam B2
0.999
0.999
0.999
0.9
iterations
25000 & 4
20000
?
15000*
disc. conv. layers
4
4
3
gen. conv. layers batch size evaluation samples (i.e., 7. dimension) z distribution
& 4
4
4
3
64, 128, 256 & 64 10000
64
64
64
26000
26000
26000
100
100
100
128
AO TT)
unif[-1, 1]
uniff 1,
unif 11,
The hyperparameters and architectures of the stacked MNIST experiment performed by the different methods are given here. This represents our best efforts at providing comparison between the different parameters used. The asterisks ** indicate that the parameters where obtained by the Github repository. Notice that GDPP paper also used 30000 iterations for training DCGAN and unrolled GAN (indicated by "). BuresGAN's column refers to the settings of experiment of Table 2 and Table 3 respectiivlly, for which the different values are separated by the symbol &.
Table 12:
14

---

Under review as a conference paper at ICLR 2021
C.3 CIFAR-10 AND 100 DCGAN ARCHITECTURES
Layer
Output
Activation
BN
Layer
Output
Activation
BN
Input
100
Input
32, 32, 3
Dense
16384
ReLUU
Yes
Conv
16, 16, 64
Leaky ReLU
No
Reshape
8, 8, 256
Conv
8, 8, 128
Leaky ReLU
Yes
Conv'
128
ReLU
Yes
Conv
4, 4, 256
Leaky ReLU
Yes
Conv'
16, 16, 64
ReLU
Yes
Flatten
Conv'
32, 32,
ReLU
Yes
Dense
Table 13: The generator and discriminator architectures for the CIFAR-10 and CIFAR-100 experi-
ments. The BN column indicates whether batch normalization used after the layer or not.
Layer
Output
Activation
BN
Input
32, 32,
Conv
16, 16,
ReLU
Yes
Conv
8, 64
ReLU ReLU
Yes
Conv
8, 8, 128
Yes
Flatten
Dense
100
Table 14: The MDGAN encoder model architecture for the CIFAR- and CIFAR-100 experiments.
The BN column indicates whether batch normalization is used after the layer not.
C.4 STL-10 DCGAN ARCHITECTURES
Layer
Output
Activation
BN
Layer
Output
Activation
BN
Input
100
Input
96, 96, 3
Dense
36864
ReLU
Yes
Conv
48, 48, 64
Leaky ReLU
No
Reshape
12, 12, 256
Conv
24, 24, 128
Leaky ReLU
Yes
Conv"
12, 12, 256
ReLU
Yes
Conv
12, 12, 256
Leaky ReL.U
Yes
Conv"
24, 24, 128
ReLU
Yes
Conv
6, 6, 512
Leaky ReLU
Yes
Conv'
48, 48, 64
ReLU
Yes
Flatten
Conv*
96, 96,
ReLU
Yes
Dense
Table 15: The generator and discriminator architectures for the STL-10 experiments. The BN col- umn indicates whether batch normalization is used after the layer not.
15

---

Under review as a conference paper at ICLR 2021
Layer
Output
Activation
BN
Input
96, 96,
Conv
48, 48, 3
ReLU
Yes
Conv
24, 24, 64
ReLU
Yes
Conv
12, 12, 128
ReLU
Yes
Conv
12, 12, 256
ReLU
Yes
Flatten
Dense
100
Table 16: The MDGAN encoder model architecture for the STL-10 experiments. The BN column indicates whether batch normalization is used after the layer or not.
C.5 RESNET ARCHITECTURES
For CIFAR-IO, we used the ResNet architecture from the appendix of Gulrajani et al. (2017) with minor changes as given in Table 17 We used an initial learning rate of 5e-4 for CIFAR- 10 and STL-10. For both datasets, the models are run for 200k iterations. For STL-10, used similar architecture that is given in Table 18
Layer
Kernel Size
Resample
Output Shape
Input
128
Dense
200
Reshape
200
4
ResBlock
[3 x
X 2
up
200 x 8 x 8
ResBlock
up
200 16 16
ResBlock
3
x
3
x
2
up
200 x 32 x 32
Conv, tanh
32 x 32
Layer
Kernel Size
Resample
Output Shape
ResBlock
3
3
X 2
Down
200 x 16 x 16
ResBlock
x 2
Down
200
8
ResBlock
3 3
x 2
Down
200 x x 4
ResBlock
x
200 X X 4
ResBlock
3 x
x
200 4 x 4
ReLu, meanpool
200
Dense
Table 17: The generator (top) and discriminator (bottom) ResNet architectures for the CIFAR-10 experiments.
16

---

Under review as a conference paper at ICLR 2021
Layer
Kernel Size
Resample
Output Shape
Input
128
Dense
200 6 6
Reshape
200 x x 6
ResBlock
33
x
up
200 12 12
ResBlock
3 x 3
x
up
200 x 24 x 24
ResBlock
x
up
200 48
48
Conv, tanh
3 3
3 x 48 x 48
Layer
Kernel Size
Resample
Output Shape
ResBlock
Down
200 x 24 24
ResBlock
3
x 2
Down
200 x 12 x 12
ResBlock
x
Down
200
6
ResBlock
3
3
x 2
200 x 6 x 6
ResBlock
x
3
x 2
200 x x 6
ReLu, meanpool
200
Dense
Table 18: The generator (top) and discriminator (bottom) ResNet architectures for the STL-10 periments. For the experiment with full-sized 96x96 images, an extra upsampling block was added the generator.
D ADDITIONAL EXPERIMENTS
D.1 TIMINGS
The timings per iteration for the experiments presented in the paper listed in Table 19 Times are given for all the methods considered, although some method do not always generate meaningful images for all datasets. They measured for 50 iterations after the first iterations, and the average number of iterations per second is computed. The fastest method is the vanilla GAN. BuresGAN has similar computation cost GDPP. We observe that BuresJBN is significantly faster compared to WGAN-GP. In order to obtain reliable timings, these results were obtained on the same GPU Nvidia Quadro P4000, although, for convenience, the experiments on these image datasets were executed on machine equipped with different GPUs.
stacked MNIST
CIFAR-10
CIFAR-100
STL-100
GAN
0.54
0.65(0.02)
0.64(0.0008)
6.00(0.01)
WGAN-GP
2.99(0.004)
3.41(0.009)
3.41(0.006)
36.5(0.03)
UnrolledGAN
1.90(0.002)
2.17(0.003)
2.18(0.004)
21.99(0.06)
MDGAN-v1
1.24(0.002)
1.47(0.001)
1.47(0.002)
13.35(0.03)
MDGAN-v2
1.66(0.002)
1.98(0.002)
1.98(0.002)
18(0.03)
VEEGAN
0.56(0.006)
0..6
0.65(0.004)
6.10(0.03)
GDPP
0.69(0.02)
0.80(0.02)
0.80(0.02)
7.46(0.03)
PacGAN2
0.77(0.006)
0.91(0.007)
0.91(0.007)
8.02(0.008)
BuresGAN
0.72(0.02)
0.82(0.001)
0.82(0.0008)
7.6(0.03)
Alt-BuresGAN
0.98()
1.15(0.007)
1.15(0.007)
10.10(0.03)
Table 19: Average time iteration in seconds for the convolutional architecture. Averaged over runs, with std parenthesis. The batch size is 256. For Stacked MNIST, we use discriminator architecture with 3 convolutional layers.
17

---

Under review as a conference paper at ICLR 2021
D.2 BEST INCEPTION SCORES ACHIEVED WITH DCGAN ARCHITECTURE
The inception scores for the best trained models are listed in Table 20 For the CIFAR datasets, the largest inception score significantly better than the for UnrolledGAN and VEEGAN. This is the same for GAN and GDPP on the STI-10 dataset, where the methods often converge to bad results. Only the proposed methods are capable of consistently generating high quality images over all datasets.
CIFAR-10
CIFAR-100
STL-10
GAN
5.92
6.33
6.13
WGAN-GP
2.54
2.56
UnrolledGAN
4.06
4.14
VEEGAN
3.51
3.85
GDPP BuresGAN
6.21
6.32
6.05
6.69
6.67
7.94
Alt-BuresGAN
6.40
6,48
7.88
Table 20: Inception Score for the best trained models on CIFAR-10, CIFAR-100 and STL-10, with a DCGAN architecture (higher is better).
D.3 INFLUENCE OF THE NUMBER OF CONVOLUTIONAL LAYERS FOR DCGAN ARCHITECTURE
Also, we provide in Figure 21 results with DCGAN architecture including only 2 conv. layers for the discriminator in contrast to Table which uses 3 conv. layers.
modes(()
KL div.())
Batch size
64
128
256
64
128
256
GAN
970.5(5.8)
972.7(6.4)
979(3.5)
0.47(0.04)
0.44(0.02)
0.41(0.03)
WGAN GG
996.7(1.6)
997.5(0.9)
998.1(1.5)
0.25(0.01)
0.22(0.01)
0.21(0.05)
2 conv, layers
MDGAN-VV
115.9(197)
260.9(267)
134.3(157)
5.5(1.4)
4.9(1.7)
5.8(0.9)
MDGAN-v2
698.1
898.4(299)
599.2 188)
2.2(3.0)
0.86(1.9)
2.8(3.2)
UnrolledGAN
953.5(11)
971.4(4.8)
966.2(17.3)
0.71(0.06)
0.60(0.04)
0,58(0.10)
VERGAN
876.7(290)
688.5(143)
775-9(386)
0.92(1.6)
1.9(2.4)
1.54(2.2)
GDPP
974.4(3.3)
978.2(7.6)
980.5(6.0)
0.45(0.02)
0.43
0.41(0.03)
PacGAN2
969.8(6.9)
971.1
977.9(4.3)
0.54(0.04)
0.51
0.48(0.02)
BuresGAN
973.2(1.3)
979.9(4.0)
981.1(4.9)
0.36(0.02)
0.30(0.02)
0.25(0.01)
Ali-BuresGAN
975.4(6.8)
978.2(5.4)
980.2(3.0)
0.37(0.02)
0.30(0.01)
0.28(0.01)
Table 21: KL-divergence between the generated distribution and true distribution (Quality, lower is better). The number of counted modes indicates the amount of mode collapse (higher better). 25k iterations and average and std over 10 runs. Same architecture as in Table 2 with discriminator with 2 convolutional layers.
E ADDITIONAL FIGURES
18

---

Under review as a conference paper at ICLR 2021
Step 1k
Step 2k
Step
Step
Step 8k
Step 25k
GAN
WGAN-GP
MDGAN-v1
MDGAN-v2
UnrolledGAN
VEEGAN
GDPP
PacGAN2
BuresGAN
Alt-BureSGAN
Figure 4: The progress of different GANs on the synthetic ring example. Each column show 3000 samples from the training generator blue with 3000 samples from the true distribution in green.
19

---

Under review as a conference paper at ICLR 2021
Step 1k
Step 2k
Step
Step
Step 8k
Step 25k
GAN
WGAN-GP
MDGAN-v1
MDGAN-v2
UnrolledGAN
VEEGAN
GDPP
PacGAN2
BuresGAN
Alt-BureSGAN
Figure 5: The progress of different GANs on the synthetic grid example. Each column show 3000 samples from the training generator blue with 3000 samples from the true distribution in green.
20

---

Under review as a conference paper at ICLR 2021
GAN
WGAN-GP
MDGAN-VI
3
0
4
9
H
4
1
6
4
0
7
5
8
9
9
8
2
9
9
0
0
0
7
7
2
8
9
4
4
4
7
9
3
8
3
5
8
8
9
8
9
7
8
7
5
9
7
7
3
7
2
4
3
0
9
2
9
4
7
7
8
9
5
6
9
3
3
3
8
4
B
7
4
6
8
0
MDGAN-v2
UnrolledGAN
VEEGAN
0
5
2
1
9
9
9
9
$
9
0
A
4
4
0
6
S
9
0
$
9
7
9
8
3
0
8
4/
8
7
7
9
6
9
3
6
&
9
4
0 9
7
4
2
3
S
9
4
4
9
6
3
A
4
0
8
0
5
4
0
7
7
&
4
3
8
6
3
4
7
4
9
9
6
8
8 7 0 4
4
4
9
2
K
9
6
4
4
7 8
3 9 4
5 3
9
4
8
9
8 6 4
S
1
9
4
2
6
0
6
4
3
8
7
Z
1
4
2
4
6
7
2
3
0
7
9
4
9
4
2
GDPP
PacGAN2
BuresGAN
9
3
7
4
7
4
9 2
4
9
6
9
0
4
4
9
10
9
2
4
1
4
0
77
&
7
7
7
8
4
8
4
4
5
2
2
9
9
4
7
2.
5
8
4
3
5
47
Q
7
3
36
9
7
2
4
X
5 /
9
4
9
4
6
K
5
5
3
7
5
2
6
8
9 9 0
9 7 3 4
3
0
4
8
7
K
7
5
4
4
66
0
5
3
5
6
1
8 6
8
3
3
4
6
5
6
5
g
4
0
9
9 4
4
3
4
7
4
9
5
9
5
7
7
3
0
7
8
0
9
8
3
4
9
9
6
Alt- Alt-BuresGAN
9
30
7
2
4
8
4
7
{
0h
7
7
9
5
4
0
B
f
6
3
8
9
C
8
Y
6
7
6
O
M
Z
4
6
J
7
3
8
3
5
0
1
8
1 9
3
4
4
6
9
4
Figure 6: Generated images for the Stacked MNIST dataset. Each model is trained with 3 layers and mini-batch size 256. Each square shows 64 samples from the trained generator.
21

---

Under review as a conference paper at ICLR 2021
GAN
WGAN-GP
MDGAN-vI
MDGAN-v2
UnrolledGAN
VEEGAN
CDPP
PacGAN2
BuresGAN
Alt- At-BuresGAAN
Figure 7: Generated images for CIFAR-10 using a DCGAN architecture. Each square shows 64 samples from the trained generator.
22

---

Under review as a conference paper at ICLR 2021
GAN
WGAN-GP
MDGAN-vI
MDGAN-v2
UnrolledGAN
VEEGAN
CDPP
PacGAN2
BuresGAN
Alt- B--BursGGNN
Figure 8: Generated images for CIFAR-100 using DCGAN architecture. Each square shows 64 samples from the trained generator.
23

---

Under review as a conference paper at ICLR 2021
GAN
GDPP
PacGAN2
BuresGAN
Figure 9: Generated images for STL-10 using DCGAN architecture. Each square shows 64 sam- ples from the trained generator.
24


Reviews:
[{"review_id": "nArCN7vUuag", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nSummary: This paper proposes to use the squared Bures distance in discriminator feature space to match the generated and real distributions. This proposed method does not require any modification to the network architectures, and is easy to implement. The proposed method produces good empirical results with simple generator architectures in synthetic and real datasets. \n\nReason for score: While I find strong interest in the proposed method, the way in which it is presented in this paper does not permit me to fairly judge the merits of this method. The experiments are not thorough and the quality of writing is subpar. Thus, I vote for reject on this paper.\n\nPros. \nThis method introduced in this paper is theoretically sound. The idea of imposing additional distribution matching in the feature space of the discriminator is an interesting direction that should warrant more studies. Extending beyond the current scope, the use of the adversarial network as both a discriminator and a feature extractor has great potential as a research direction.\n\nCons\n1. The choice of synthetic experiments may be too easy to discern difference between methods. Most GANs get similar results on the 2D grid experiments as the bottleneck is mostly in the modelling capacity of the generator at expressing non-linearities. Previous works have also achieved perfect mode coverage on stacked-mnist with similar network architecture (https://arxiv.org/pdf/1712.04086.pdf). \n2.The Bures metric is one among many metrics for comparing covariance matrices. Ablation on the choice of distance metric should be performed. \n3. Higher performing methods with similar network capacities have been left out in the performance tables. This is counter-productive to the purpose of establishing context. Very few recent methods evaluate DCGAN on cifar10 or STL as the architecture is too limiting for these complex datasets. Only inception scores are reported on resnet experiments. Thorough experimentation and evaluation with a modern architecture on RGB datasets would greatly help the case for this paper.\n\n4. The writing in this paper requires significant rework to reach publication quality. Grammatical mistakes and convoluted sentences are too frequent and significantly detracts from the idea being presented.\n\nMinor comments\n“...images, although, the training...” run-on sentence\n“; and a discriminator” what follows a semicolon must be a full sentence.\n“issue – the ‘mode collapse’ – appears” -> remove “the”\n“complemented by a additional term” -> “an additional term”; I stopped tracking grammar mistakes after this point.\n“MDGAN (Li et al., 2017),” -> MMD-GAN, MDGAN is (Che et al., 2017).\nIn table 1, does “time” column correspond to total training time of 25k iterations? Why is BuresGAN so much slower than MDGAN here but faster in iteration time according to appendix D table 16?\n\n[Post rebuttal]\nI've done a complete re-read of the updated manuscript. I am not convinced with the efficacy of the proposed method in solving mode drops beyond what has already been achieved in the literature. In particular, there is no guarantee that matching covariances within the feature space of a NN prevents mode dropping, as the NN (discriminator) itself must abstract away visual information to perform discrimination. The quality of writing has not improved significantly either. I am now more confident with my original assessment. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JEFcAOT1OIG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main strengths of the work are,\n* The proposed idea is relatively novel, although similar ideas were explored in (Mroueh et al., 2017; Elfeki et al., 2019).\n* The paper uses a computationally efficient expression (4) for the Bures distance. However, this expression has been proposed in prior work (Oh et al., 2020).\n* The paper discusses connections with Wasserstein GAN and integral probability metrics. In particular, shows that the proposed distance os proportional to the 2-Wasserstein distance.\n* Detailed information on the architectures and datasets is given in the Appendix, aiding reproducibility.\n* The paper is very well written and easy to follow.\n\nMy main concerns are with the evaluation,\n* Experiments on synthetic data: The following two papers outperform the proposed approach [1,2], especially in terms of number of high quality samples on both the Ring and Grid sets.\n* Experiments on CIFAR-10 with DCGAN architecture - Both [1,2] again outperform the proposed approach in terms of the the FID metric. The order of the IvO score also does not seem to match prior works e.g. VEEGAN (Table 2), [1,2]. Please clarify the exact procedure used to compute the IvO scores.\n* Experiments on CIFAR-10/STL-10 using ResNet architecture: The FID scores should be also reported for fair comparison with the state of the art. More importantly: the IS scores of best performing methods on CIFAR-10: ProgressiveGAN (Karras et al., 2017) and NCSN (Song & Ermon, 2019) are to reported for STL-10. It is unclear whether the proposed approach really achieves a new state of the art inception score on STL-10. Comparison with ProgressiveGAN (Karras et al., 2017) and NCSN (Song & Ermon, 2019) on STL-10 must be performed.\n* The cost of the computing the Bures distance in terms of training time in comparison to simpler losses like Hinge loss [3] or gradient penalty loss of WGAN-GP should be clarified. It is unclear whether the additional resources required (if any) justly the limited performance gain of the proposed method.\n\n\n[1] Mixture Density Generative Adversarial Networks, CVPR 2019.\n\n[2] \"Best-of-Many-Samples\" Distribution Matching, NeurIPS Workshop, 2019.\n\n[3] Spectral Normalization for Generative Adversarial Networks, ICLR 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ef-s870nQL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update:\n\nThank the authors for providing the additional results and updating the paper. Since the comparison to one state-of-the-art method has been added (though some other state-of-the-arts are still missing) and the benefit is shown across different settings, I increase the score from 5 to 6.\n\n(One suggestion: I would recommend you to highlight the changes in the revision with a different color, so that readers can identify the changes easily.)\n\n----\n----\nThe paper addresses the mode collapse issue in GANs. More specifically, the paper proposes to add a regularization which matches the Bures distance between the covariance matrices of the features of real and generated data. The paper demonstrates the performance across different datasets and architectures.\n\nOverall, the paper has several merits:\n* Experiments across a wide range of datasets and architectures.\n* A detailed comparison of training time.\n\nHowever, I cannot recommend this work for acceptance at this point, mainly because the paper did not compare with the state-of-the-art (and some widely-used) methods for fighting mode collapse, and the improvements on the benchmark datasets are rather weak. The details are below.\n* The paper does compare with several GANs on the standard benchmark datasets for evaluating mode collapse (e.g. 2D-grid, 2D-ring, stacked MNIST). However, the baselines are rather weak and old. Even in that case, the scores of (Alt-)BuresGAN in Table 1 and Table 2 are just similar to (or sometimes even worse than) those baselines. In fact, there are many other newer and/or better methods that have shown to be outperforming the baselines the paper considered by a large margin, and some of them are already been out for years (e.g. [1,2,3,4]). But the paper did not compare with those methods. Therefore, it is unclear at all whether (Alt-)BuresGAN is useful or not.\n* More specifically, on the stacked MNIST dataset, for example, many methods can dramatically improve the DCGAN baseline from ~100 modes (a relatively poor score) to 1000 modes (the best possible score on this dataset) [1,3,4]. The experiments in your paper seem to build on a better DCGAN architecture (993.3 modes already). But even in that case, (Alt-)BuresGAN only achieves 995.0 modes, and in fact, this score is within std to the DCGAN baseline. Also, on 2D-grid and 2D-ring datasets, some methods can achieve much better results and improvements than yours both quantitatively and qualitatively [2]. I understand that possibly the hyper-parameters and architectures in (Alt-)BuresGAN and those papers are different, so we cannot directly conclude that (Alt-)BuresGAN is worse than [1,2,3,4]. But these results do raise critical concerns about the performance of (Alt-)BuresGAN in fighting mode collapse, compared with the state-of-the-art methods.\n\nBesides this point, I also have some other questions/suggestions:\n* You use the features from the last layer of the discriminator. Why do you choose the last layer? How the performance would be if you are using other layers?\n* \"Algorithmic details\" paragraph on page 4: the regularization term is already mentioned before, so you might consider removing it here.\n* In the same paragraph, you might want to move \"In the tables hereafter, we indicate the largest scores in bold if they differ from lower scores by at least one std\" to the experimental section, because it is for result presentation, not for your algorithm.\n* In fact, it is unclear to me if you are using this rule to mark the bold numbers. For example, in table 1, 84(6) and 22.9(4) shouldn't be marked as bold according to this rule. The same problem exists in all other tables in the paper. To me, how you mark the bold numbers seems random.\n\nIn conclusion, the paper does have some merits, but also have some critical problems, especially lacking the comparisons to the state-of-the-art methods, which makes it hard to judge the contribution of the method. **However, I am happy to adjust the scope if the authors can provide evidence regarding comparisons to the state-of-the-art methods during the rebuttal.**\n\n[1] Lin, Zinan, et al. \"Pacgan: The power of two samples in generative adversarial networks.\" Advances in neural information processing systems. 2018.\n\n[2] Xiao, Chang, Peilin Zhong, and Changxi Zheng. \"Bourgan: Generative networks with metric embeddings.\" Advances in Neural Information Processing Systems. 2018.\n\n[3] Belghazi, Mohamed Ishmael, et al. \"Mine: mutual information neural estimation.\" arXiv preprint arXiv:1801.04062 (2018).\n\n[4] Eghbal-zadeh, Hamid, Werner Zellinger, and Gerhard Widmer. \"Mixture density generative adversarial networks.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DuO35RmK-ye", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Update after reading the authors' rebuttal:\nThe revision of the manuscript was much improved. However, the lack of controlled experiments does not convince me. This paper proposes a new penalty to deal with mode collapse, and the authors claimed that it could be easily added to any existing GAN variants. The authors may need to provide a controlled experiment to show their claim, e.g., what happens if adding their penalty to some GAN variants, fixing the same setting. \nOne good point from the new revision is that BuresGAN can be competitive with the state-of-the-art baselines, with some slight changes in the network architectures. I suggest the authors to test their penalty with other GAN variants to stronger support their claim.\n\n--------------------------------------------\nThis paper concerns the problem of mode collapse in Generative Adversarial Networks (GANs). A new measure (Bures distance) is investigated to overcome mode collpase in GANs. Bures distance can help us to measure the similarity of two possitive semi-definite matrices and was proposed before. This paper proposes a penalty to the generator loss to encourage the diversity of fake data to match the diversity of real data. To do this, the last layer (providing the representation for each input) of the discriminator is used to define the diversity in input. Bures distance is then used to define the similarity between the diversity of real data with that of fake data, leading to the novel method called BuresGAN. Four datasets are used for their evaluation and comparison with 7 baselines. The experimental results seem to be promising.\n\nPros:\n- Bures distance is an interesting metric and promising to deal with mode collapse.\n- The experimental results are promising.\n\nCoins:\n- Unclear context: mode collapse is a challenging problem. There exist various approaches to deal with this problem, such as using more generators, more discriminators, using different losses, or penalty. However, this paper does not provide an extensive overview of the existing literature on mode collapse. As a result, it is unclear about the context of this paper and the significance of their contributions. The authors should provide an extensive summary and then place their work in a clear context.\n- Unclear significance: The authors use different network architectures for different methods in their experiments, e.g., MDGAN often uses architectures which are different from other methods. Such different architectures make us hard to see which components (e.g. architecture, loss, penalty) really contribute to the success/failure of a method. Also, it is unclear whether the good performance of BuresGAN comes from the new penalty or not. The authors should design a controlled experiment to see the practical effect of their new penalty compared with other penalties, such as by fixing the shared architectures for generator and discriminator and their losses.\n- Baselines: some other types of baselines for dealing mode collapse should be included, e.g. multi-generator or multi-discriminator based methods. Such a comprison will provide more evidents to see the significance of the proposed penalty. Some examples are MAD-GAN [Ghosh et al., 2018] and D2GAN [Nguyen et al., 2017].\n\nMinor comment:\n- The results of some baselines, e.g., MDGAN, VEEGAN, UnrolledGAN, are sometimes not very good as reported in Table 3. Why this happened? Was it because of the use of default settings or unconvergence when training? \n\nReference:\n- Nguyen, T., Le, T., Vu, H., & Phung, D. (2017). Dual discriminator generative adversarial nets. In Advances in Neural Information Processing Systems (pp. 2670-2680).\n- Ghosh, A., Kulharia, V., Namboodiri, V. P., Torr, P. H., & Dokania, P. K. (2018). Multi-agent diverse generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8513-8521).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose the Bures metric (a distance between covariance matrices of the last feature layer of a discriminator) as an extra loss to mitigate mode collapse. The metrics bears some similarity to the covariance term in FID, and builds upon a number of GAN papers that augment GAN losses with differences in covariances between real and generated data. As the reviewers noted, the authors did an admirable job of performing an apples-to-apples comparison with other GAN alternatives, and use a number of metrics to demonstrate their results. Unfortunately, the most extensive comparisons usedthe DCGAN architecture, which is now 2-3 years too old for a potential reader to ascertain how well the proposed method would work on her problem. Moreover, the reviewers identified discrepancies in the baselines of those the experiments, as the numbers reported in this paper seemed to indicate poorer performance and the numbers reported in the original papers.

During the rebuttal phase, the authors demonstrated that these methods also perform well with using ResNet architectures on CIFAR-10 and STL-10, and the method is competitive with more recent models. As noted by the reviewers, however, these new comparisons are not as extensive and controlled as those that used DCGAN. Furthermore, results on more difficult datasets, such as ImageNet, are missing.

Had the extensive experiments used ResNets instead of DCGAN, or if the authors demonstrated promising results on ImageNet, I would recommend acceptance. Unfortunately, I think the audience for this paper in 2020-2021 would be relatively limited, so I have to recommend rejection.
Decision: Reject