Ensembles of Generative Adversarial Networks for Disconnected Data
Authors: Lorenzo Luzi
Randall Balestriero
Richard Baraniuk
Venue:  (2021)
Keywords: GANs
ensembles
disconnected data

Abstract: Most computer vision datasets are composed of disconnected sets, such as images of different objects. We prove that distributions of this type of data cannot be represented with a continuous generative network without error, independent of the learning algorithm used. Disconnected datasets can be  represented in two ways: with an ensemble of networks or with a single network using a truncated latent space. We show that ensembles are more desirable than truncated distributions for several theoretical and computational reasons. We construct a regularized optimization problem that rigorously establishes the relationships between a single continuous GAN, an ensemble of GANs, conditional GANs, and Gaussian Mixture GANs. The regularization can be computed efficiently, and we show empirically that our framework has a performance sweet spot that can be found via hyperparameter tuning. The ensemble framework provides better performance than a single continuous GAN or cGAN while maintaining fewer total parameters. 

Under review as a conference paper at ICLR 2021
ENSEMBLES OF GENERATIVE ADVERSARIAL NETWORKS FOR DISCONNECTED DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Most computer vision datasets are composed of disconnected sets, such images of different objects. We prove that distributions of this type of data cannot be represented with continuous generative network without error, independent of the learning algorithm used. Disconnected datasets can be represented in two ways: with an ensemble of networks or with a single network using truncated latent space. We show that ensembles are more desirable than truncated distributions for several theoretical and computational reasons. We construct a regularized optimization problem that rigorously establishes the relationships between a single continuous GAN, an ensemble of GANs, conditional GANs, and Gaussian Mixture GANs. The regularization can be computed efficiently, and we show empirically that our framework has a performance sweet spot that can be found via hyperpa- rameter tuning. The ensemble framework provides better performance than single continuous GAN or cGAN while maintaining fewer total parameters.
INTRODUCTION
Generative networks, such as generative adversarial networks (GANs) (Goodfellow et al. 2014) and variational autoencoders (Kingma & Welling 2013). have shown impressive performance in generating highly realistic images that were not observed in the training set (Karras et al. 2017]
2019a However, even state of the art generative networks such as BigGAN (Brock et al. 2018
generate poor quality imagery if conditioned on certain classes of ILSVRC2012 (Russakovsky et al. 2015). We argue that this is due to the inherent disconnected structure of the data.
In this paper, we theoretically analyze the effects of disconnected data on GAN performance. By disconnected, we mean that the data points are drawn from an underlying topological space that is disconnected (the rigorous definition is provided below in Section 3.T.. As intuitive example, consider the collection of all images of badgers and all images of zebras. These two sets are disconnected, because images of badgers do not resemble images of zebras, and modeling the space connecting these sets does not represent real images of animals.
We rigorously prove that one cannot distribution perfectly under the disconnected data model. Because generative networks are continuous, they cannot map a connected latent space (R into the disconnected image space, resulting in the generation of data outside of the true data space. In related work, (Khayatkhoei et al. 2018) has empirically studied disconnected data but does not formally prove the results in this paper. addition, the authors use a completely unsupervised approach to attempt to find the disconnected components as a part of learning. In contrast, we use class labels and hence work in the supervised learning regime.
single continuous generative network learn data
Our suggested approach to best deal with disconnected data is to use ensembles of GANs. We study GANs in particular for concreteness and because of their widespread application; however, our methods can be extended to other generative networks with some modification. Ensembles of GANs are not new, e.g., see (Nguyen al. 2017 Ghosh et al. 2018 Tolstikhin et al. 2017: Arora et al. 2017), but there has been limited theoretical study of their properties. We prove that ensembles can learn the data distribution under the disconnected data assumption and study their relationship to single GANs. Specifically, we develop a first-of-its-kin theoretic framework that relates single GANs, ensembles of GANs, conditional GANs, and Gaussian mixture GANs. The framework makes

---

Under review as a conference paper at ICLR 2021
it easy e.g.. develop regularized GAN ensembles that encourage parameter sharing, which show outperform eGANs and single GANs.
While primary focus here is theoretical insight, also conduct range of experiments to
demonstrate empirically that the performance (measured in terms of FID (eusel et al. 2017). MSE to the training set (Metz et al.) 2016). Precision, and Recall (Sajjadi al. 2018) increases when use an ensemble of WGANs over a single WGAN on the CIFAR- 10 dataset (Krizhevsky & Hinton 2009). The performance increase can be explained in terms of three contributing factors: 1) the ensemble has more parameters and hence has higher capacity to learn complex distributions, 2) the ensemble better captures the disconnected structure of the data, and 3) parameter sharing among ensemble networks enables successful joint learning, which we observe can increase performance.
We summarize our contributions as follows:
We prove that generative networks, which continuous functions, cannot learn the data dis- tribution if the data is disconnected (Section 3.2). The disconnected data model is defined in Section 3.1 where we argue that it is satisfied in many common datasets, such MNIST, CIFAR- 10, and ILSVRC2012. Restricting the generator to a disconnected subset of the domain is one solution (Section 3.3), but we study better solution: using ensembles.
We demonstrate how single GANs and ensembles that ensembles are able to learn the true data distribution under our disconnected data model (Section (.2).. Finally, we demonstrate that there is equivalence between ensembles of GANs and common architectures such as cGANs and GM-GANs due to parameter sharing between ensemble components (Section 4.3).
related (Section
We then prove
We empirically show that, in general, an ensemble of GANs outperforms single GAN (Sec- tion (.I). This is true even if we reduce the number of parameters used in an ensemble so that it has fewer total parameters than single GAN (Section(522) Finally, we empirically show that parameter sharing among ensemble networks leads to better performance than a single GAN (Section 5.3) or even cGAN (Section 5.4.
BACKGROUND AND RELATED WORK
2.1
GENERATIVE ADVERSARIAL NETWORKS (GANS)
GANs are generative neural networks that use adversarial loss, typically from another neural network (Goodfellow et al. 2014). In other words, a GAN consists of two neural networks that compete against each other. The generator G R RR is neural network that generates p- dimensional images from an f-dimensional latent space. The discriminator neural network which is trained classify between the training set and generated images. As compositions of continuous functions (Goodfellow et al. 20166), both G and D are continuous.
RR - (0, 1) is
G has parameters 0g E R where |0gl is the possibly infinite cardinality of 0g. Similarly, D has parameters θn e RR@01. The latent, generated, and data distributions are Pz, Pa. and Px, respectively. We train this network by solving the following optimization problem:
min max V(Og, 0p) DG On
minmax Eu Ea 0G 0n
[log D(x)] + Ez-P, [log(1 D(G(=)))
(I)
Here we write min and max instead of minimize and maximize for notational compactness, but we referring an optimization problem. The objective of this optimization is to learn the true data distribution, i.e., Pg = Px- Alternatively, we can use the Wasserstein distance instead of the typical cross-entropy loss: V(Oa, 0n) = Em- Px D(x) Ez- D(G(z)) restricted to those Oc, Op which force D to be 1-Lipschitz as done in the WGAN paper (Arjovsky el al. 2017). Thus, we will use V denote either of these two objective functions.
2.2 GANS THAT TREAT SUBSETS OF DATA DIFFERENTLY
Ensembles of GANs. Datasets with many different classes, such ILSVRC2012 (Russakovsky al.] 2013). are harder to learn in part because the relationship between classes is difficult to quantify. Some models, such as AC-GANs (Odena al. 2017), tackle this complexity by training different
ctt al.
2

---

Under review as a conference paper at ICLR 2021
models different classes of data in a supervised fashion. In the AC-GAN paper, the authors train 100 GANs on the 1000 classes of ILSVRC2012. The need for these ensembles is not theoretically studied or justified beyond their intuitive usefulness.
Several ensembles of GANs have been studied in the unsupervised setting, where the modes or disconnected subsets of the latent space are typically learned (Pandeva & Schubert 2019 Hoang et al. | Khayatkhoei et al. 2018) with some information theoretic regularization as done in (Chen al. 2016). These are unsupervised approaches which we do not study in this paper. Models such as SGAN (Chavdarova & Fleuret, 2018) and standard GAN ensembles (Wang et al., 2016) use several GANs in part to increase the capacity or expressibility of GANs. Other ensembles, such as Dropout-GAN (Mordido et al. 2018), help increase robustness of the generative network.
Conditional GANs (cGANs). Conditional GANs (Mirza & Osindero 2014) attempt to solve the optimization problem in ① by conditioning on the class y. one-hot vector. The generator and discriminator both take y as an additional input. This conditioning can be implemented by having the latent variable be part of the input, e.g., the input to the generator will be [z y"'" instead of just z.
Typically, conventional eGANs have the following architecture modification. The first layer has an additive bias that depends on the class vector y and the rest is the same. For example, consider multilayer perceptron, with matrix W in the first layer. Converting this network be conditional would result in the following modification to the matrix in the first layer:
W conditional
æ y
=
W
B
Wx + By = Wx+ + B.k
Hence, we can think of B as matrix with columns B. E (1...,, K} being bias vectors and W being the same before. We pick bias vector B., based on what class we are conditioning on but the other parameters of the network are held the same, independent of k. This is done to both the generator and the discriminator. Some eGANs condition on multiple layers, such as BigGAN (Brock et al. 2018), or on different types of layers, such as convolutional layers, but our formulation here extends clearly to those other architectures.
Gaussian Mixture GANs (GM-GANs). The latent distribution Pz is typically chosen to be either uniform, isotropic Gaussian, or truncated isotropic Gaussian (Goodfellow al., 2014; Radford et al., 2015; Brock et al., 2018). We are not restricted to these distributions; research has been conducted extending and studying the affect of using different distributions, such as mixture of Gaussians (Ben- Yosef & Weinshall 2018 Gurumurthy al. 2017).
3 CONTINUOUS GENERATIVE NETWORKS CANNOT MODEL DISTRIBUTIONS DRAWN FROM DISCONNECTED DATA
3.1 DISCONNECTED DATA MODEL
We begin by introducing a new data model that accounts for disconnected data. Typical datasets with class labels satisfy this model; we provide additional examples below.
Definition 1 (Disconnected data model). We assume that the data lies on K disjoint, compact sets X' Rd he {1, (1...., K} so that the whole data lies the disjoint union of each component: UK , X X. Moreover, we assume that each component X₁ is connected (Rudin 1964). We then draw data points from these sets in order to construct our finite datasets.
In Definition [ we let each X₂ be compact in order to remove the degenerate case of having two components X'. and X, that are arbitrarily close to one another, which is possible if we only assume that X closed and disjoint. that is the case, there are trivial counter-examples (see the appendix) to the theorems proved below.
Lemma 1. X a disconnected set, and X, is disconnected from Xx for j k.
Disconnected datasets are ubiquitous in machine learning (Khayatkhoei et al.) 2018 Hoang et al. 2018 Pandeva & Schubert 2019). For example, datasets with discrete labels (typical in classification problems) will often be disconnected. We study this disconnected data property, because generative networks are unable learn the distribution supported on such dataset, show below.
3

---

Under review as a conference paper at ICLR 2021
3.2 CONTINUOUS GENERATIVE NETWORKS CANNOT REPRESENT DISCONNECTED DATA DISTRIBUTION EXACTLY
In this section, prove that, under Definition | continuous generative networks cannot learn the true data distribution exactly due to model misspecification.
Suppose that (S2, F, P²) is probability space with Pz being the distribution of the random vector : S2 - R We assume that Ps is equivalent to the Lebesgue measure A. This just means that {(=(A)) 0 if and only if P= (A) for example, which is commonly used as a latent distribution in GANs (Arjovsky et al. 2017). The transformed (via the generative network G) random vector a G the original probability measure Pz but is defined on the induced probability space (S',
F2,
for any set A E F. This is true for Gaussian distribution,
E. is determined by
Theorem 1. If G can generate from multiple components of X (say X1 and X2). then the probability of generating samples outside of X is positive: Pa R x) 0. Otherwise we can only generate from one component (say X1), J. then (x( = 0 for x E K3..
The continuity of G' the fundamental reason why Theorem is true. A continuous function cannot map a connected space to a disconnected space. This means that all generative networks must generate samples outside of the dataset the data satisfies Definition
Suppose that our data is generated from the true random vector Taata - IR using the probability distribution Px. Also, suppose that we learn Pg by training a generative network.
Corollary 1. Under Definition ? we have that d(Pa, Px) for any distance metric d and any learned distribution Pg.
that learning the data distribution will incur irreducible error under Definition I because our data model and the model that we are trying to train do not match. Hence, we need to change which models we consider when we train in order to best reflect the structure of our data. At first thought a discontinuous G might be considered. but that would require training G without backpropagation. Instead, we focus on restricting G to a discontinuous domain (Section 3.3 and training an ensemble of GANs (Section 4.1 as two possible solutions.
From Corollary
3.3 RESTRICTING THE GENERATOR TO A DISCONNECTED SUBSET OF THE LATENT DISTRIBUTION
In this section, study how after training. Suppose that we train a generator G on some data so that G(R') ツ X. Therefore, we actually generate points from the true data distribution. We know that the distributions cannot be equal because of Theorem] [ implying that if we restrict the domain of to the set = G (x' then G(Z) X. The next theorem shows how the latent distribution is related to restricting the domain of G.
remove the irreducible error in Theorem I from our models
Theorem (Truncating the latent space reduces error). Suppose that Pz(Z) > and let the generator G learn a proportionally correct distribution over X. In other words, there exists real number c E R so that
Po(A)= cPx(() A AEF, X.
Then, we use the truncated latent distribution defined by (B) = 0 for all B e F that satisfy B o 7 Ø. This allows to learn the data distribution exactly, i.e.
(A) = Px((A Ae
We write Pat. because, by truncating the latent distribution, we effectively restrict G to the domain Z. Theorem shows that if we learn the data distribution approximately by learning a proportional distribution, then in (Rudin) 1964). Z must be disconnected, which implics that a disconnected latent distribution is a solution to remove the irreducible error in Theorem
learn the true data distribution by truncating latent distribution. By 4.22
Although Theorem 2 suggests that we truncate the latent distribution, there are several limitations with this approach. First, the latent distribution cannot be truncated without knowing closed form expression for Pg. Second, we may learn the disconnected set Z by training a mixture distribution for Pz as is done in (Ben-Yosef & Weinshall] 2018 Gurumurthy et al. 2017). The problem with
4

---

Under review as a conference paper at ICLR 2021
this is that the geometric shape of Z is restricted to be spherical or hyperellipsoidal.. Third, before truncating the latent space, we need to train a generative network to proportionally learn the data distribution, which is impossible to confirm.
Given these limitations, we introduce the use of ensembles of generative networks in Section 4.1 This class of models addresses the issues above follows. First, we will not need to have access to Pa in any way before or after training. Second, knowing the geometric shape of Z is no longer an issue because each network in the ensemble is trained on the connected set Xr instead of the disconnected whole X. Finally, since the k-th network will only need to learn the distribution of Xs. we reduce the complexity of the learned distribution and do not have confirm that the distribution learned is proportionally correct.
4 ENSEMBLES OF GANS AND PARAMETER SHARING
We demonstrate how to train ensembles of GANs practically and relate ensembles to single GANs, cGANs, and GM-GANs. We focus on feedforward (Goodfellow et al. 2016) GANs in this section for concreteness; therefore, we study an ensemble of discriminators as well as generators.
4.1 ENSEMBLE OF GANS VS. A SINGLE GAN
Given ensemble of GANs, we will write G3 R - R" the k-th generator with parameters 0G. IR 9cl for A: E {1...., K}. where K is the number of ensemble networks. We assume that each of the generators has the same architecture, hence 0c.| = 10c,1 for all i,j; thus we drop the subscript and write 0ccl. Likewise, we write D3 IP [0. 1] for the k-th discriminator with parameters 0D2 Rlonl since the discriminators all have the same architecture. The latent distribution is the same for each ensemble network: Pz. The generated distributions will be denoted Po..
For concreteness, assume that K is the number of classes in the data; for MNIST, CIFAR-10 10, and ILSVRC2012, K would be 10. 10, and 1000, respectively. If K is unknown, then an unsupervised approach (Hoang et al. 2018 Khayatkhoei et al. 2018) can be used. Define the parameter π R such that _x=1 Tx = 1. We then draw a one-hot vector using the k-th generator if the k-th component of y is 1. Hence, have that generated sample is given by æ Gz(z). This ensemble of GANs is trained by solving
Cat(Tr) randomly and generate a sample
min max (00.. Op.)
(2)
Oc, Gp.
for l E (l,..., ......,,K Kj. Note that with an ensemble like this, the overall generated distribution Pa _k=1 Ty Pa, (x) is mixture of the ensemble distributions. This makes comparing single GAN to an ensemble challenging; for example, consider comparing a Gaussian to a mixture of Gaussians.
In order to compare single GAN an ensemble of GANs, define new hybrid optimization
K
K
min 0al
max 0D
V(OG,, OD,) ) s.t.
10D - 0D. Et
s.t.
0c,
0G. &
DD.....pDg
0DK
where 1 1 denotes the t, "norm," which counts the number of non-zero values in vector. Thus, t > serves as a value indicating how many parameters are the same across different networks, which is more general than having tied weights between networks (Ghosh et al. 2018). We penalize the parameters because it is convenient, although is not equivalent to, penalizing the functions themselves. This is true because Oc. OG We analyze the behavior of (3) as we vary t in the next theorem.
G2 GG = 0 but the converse is not true.
Theorem 3. Let G and D be the generator and discriminator network in GAN. Suppose that for k E [l,..., K} we have that Gk and Dr have the same architectures as G and D, respectively. Moreover, assume that (x(X))= = Px (Zu) for all j, k. Then,
i) Suppose that t 2 max KK K-1 0pl 0cl| Then for all is (1....,K} have that (0&,, Oz.) is solution to 3) if and only if (07, Ob.) is solution 2
5

---

Under review as a conference paper at ICLR 2021
ii Suppose that t 0. Then have that (0 (D) is solution to for each e {1,.... K} if and only if (0; (D) is a solution to 77.
Informally, Theorem shows that, when O, we essentially have single GAN, because all of the networks in the ensemble have the same parameters. If is large then we have an unconstrained problem such that the ensemble resembles the one in Equation 2 Therefore, this hybrid optimization problem trades off the parameter sharing between ensemble components in way that allows us to compare performance of single GANs with ensembles.
Unfortunately, Equation 3 is a combinatorial optimization problem and computationally in- tractable. Experimentally, relax Equation (3
the following
K
K DD
min 0aj 0G,
max 0D
Σ
V(OG.,00)
A
1D.1 1
0c 0G lla
(4)
ODK
in order to promote parameter sharing and have almost everywhere differentiable regularization term that we can backpropagate through while training. Although Equation (4) is a relaxation
of Equation (3) we still have the same asymptotic behavior when we vary À as when vary / as shown in Appendix
4.2 OPTIMALITY OF ENSEMBLES FO GANs
This next theorem shows that if we are able to learn each component's distribution, Pxa. then an ensemble can learn the whole data distribution Px.
Theorem 4. Suppose that G is the network that generates X₂ for each k (1,...: K}, i.e.
P Px. Under Definition] we can learn each G by solving (2) with V being the objective
function in Equation
We know from (Goodfellow al., 2014) that globally optimal solution is achieved when the distribution the generated images equals Px- Hence, this theorem has an important consequence: Training ensemble of networks is optimal under our current data model.
is important to note that the condition "Gę is the network that generates X is necessary but not too strong because we may have distribution that cannot be learned by generative network or that our network does not have enough capacity to learn. We do not care about such cases however, because are studying the behavior of generative networks under Definition
4.3 RELATION OF ENSEMBLES OF GANS TO OTHER GAN ARCHITECTURES
Relation to eGANs. We compare a cGAN to an ensemble of GANs. Recall from Section 2.2 that cGAN has parameters O and OD that do not change with different labels but there matrices Bg and Bn that do depend on the labels. Specifically, they solve the optimization problem Theorem with the additional constraint that the only parameters that be different are the biases in the first layer. For other variants of cGANs a similar result applies.
Theorem 5. A cGAN is equivalent to an ensemble of GANs with parameter sharing among all parameters except for the biases in the first layer. Moreover, the optimization can be modified so that it is equivalent to the cGAN optimization problem.
Relation to GM-GANs. Another generative network that is related to ensembles is the GM-GAN.
The first layer in GM-GANs transforms the latent distribution from isotropic Gaussian into a mixture of Gaussians. This new layer plays similar role the B in the cGAN comparison above, meaning that GM-GANs solve the optimization problem (3) with the additional constraint that the only parameters that can be different the parameters in the first layer.
Theorem 6. GM-GAN is equivalent parameters except for the first layer. Moreover the optimization in (3) can be modified so that is equivalent to the GM-GAN optimization problem.
ensemble of GANs with parameter sharing among all

---

Under review as a conference paper at ICLR 2021
FID
Average MSE
Precision
Recall
0.3
0.8
400
0.6
0.6
200
0.2
0.4
0.4
0.1
0.2
0.2
500
1,000
0
500
1,000
0
500
1,000
500
1,000
epoch
epoch
epoch
epoch
Ensembles of WGANs with fewer total parameters than a single WGAN perform better on CIFAR 10. We do not have to sacrifice computation to achieve better performance, we just need models that capture the underlying structure of the data. The dotted red line is the baseline WGAN, the solid blue line is the equivalent ensemble, and the dashed black line is the full ensemble.
Figure I:
EXPERIMENTAL RESULTS
In this section, we study how ensembles of WGAN (Arjovsky et al. 2017) compare with a single WGAN and a conditional WGAN. We use code from the authors' official repository (Arjovsky et al. 2018) to train the baseline model. We modified this code to implement our ensembles of GANs and cGAN. For evaluating performance, we use the FID score (Heusel et al. 2017 2020), average MSE to the training data (Metz et al., 2016 Lipton & Tripathi] (017). and precision/recal (Sajjadi et al.
2018 2019). More details about the experimental setup are discussed in Appendix C
5.1
ENSEMBLES PERFORM BETTER THAN SINGLE NETWORKS
We consider basic ensemble of WGANs where we simply copy over the WGAN architecture 10 times and train each network on the corresponding class CIFAR-I0; we call this the "full ensemble". We compare this ensemble the baseline WGAN trained on CIFAR-10.
Figure shows that the full ensemble of WGANs performs better than the single WGAN. is not immediately clear, however, whether this boost in performance is due to the functional difference of having an ensemble or it is happening because the ensemble has more parameters. The ensemble has 10 times more parameters than the single WGAN, the comparison is hard to make. Thus, consider constraining the ensemble so that it has fewer parameters than the single WGAN.
5.2 ENSEMBLES WITH FEWER TOTAL PARAMETERS STILL OUTPERFORM A SINGLE NETWORK
The "equivalent ensemble" (3.120,040 total generator parameters) in Figure still outperforms the single WGAN (3,476,704 generator parameters) showing that the performance increase comes from using the ensemble rather than just having larger capacity. In other words, considering ensembles of GANs allows for improved performance even though the ensemble is simpler than the original network in terms of number of parameters.
We see performance boost result of increasing the number of parameters, in Figure I Therefore, we perform better because of having a better model (an ensemble) as well by having more parameters. Now, we investigate way that we can further improve performance.
5.3 PARAMETER SHARING AMONG ENSEMBLE COMPONENTS LEADS TO BETTER PERFORMANCE
We study how the regularization penalty À affects performance. As discussed in Section 4.1 can learn a model that is somewhere between an ensemble and a single network by using regularization.
In Figure the performance increases when increase À in the equivalent ensemble from 0 to 0.001, implying that there is some benefit to regularization. Recall that by having O, we force parameter sharing between generator and discriminator networks. This performance increase is likely data dependent and has to do with the structure of the underlying data X. For example, we can have pictures of badgers (X) and zebras (22) in dataset and they disconnected. However, the
7

---

Under review as a conference paper at ICLR 2021
FID
Average MSE
Precision
Recall
0.3
0.6
400
0.3
300
0.2
0.4
0.2
200
0.1
0.2
0.1
100
0
0
500
1,000
500
1,000
500
1,000
0
500
1,000
epoch
epoch
epoch
epoch
Figure 2: Ensembles of WGANs have performance sweet spot when we regularize the optimization problem in expression 4 with different values of X. Each curve is calculated using the equivalent ensemble of WGANs increase À to 0.001, the performance increases but then decreases
discussed in Section[5.2 [5.2 We that when we continue increase to 0.01. This implies that there hyperparameter tuning. The solid blue line the equivalent ensemble with À = 0.01, dotted line is the equivalent ensemble WGAN, and the dashed black line is the equivalent ensemble with À = 0.001.
optimal value for that can be found via
FID
Average MSE
Precision
Recall
0.6
400
0.3
0.3
0.4
300
0.2
0.2
200
0.1
0.2
0.1
100
0
500
1,000
0
500
1,000
0
500
1,000
0
500 1,000
epoch
epoch
epoch
epoch
Regularized ensembles of WGANs using the optimization in outperform cWGANs, even though cGANs type of ensemble. Here, cWGAN actually performs similarly to the baseline WGAN even though it takes into considcration class information. The solid blue line the baseline, the dotted red line is the eWGAN, and the dashed black line the equivalent ensemble with 0.001.
Figure 3:
backgrounds these images are likely similar so that there is some benefit in G₁ and G2 treating these images similarly, if only to remove the background.
As we increase from 0.001 to 0.01 we notice that performance decreases. This means that there is sweet spot and we may be able to find optimal X 0.01 via hyperparameter tuning. We know that the performance is not monotonic with respect to À because it decreases and then increases again; in other words, the performance has minima that is not
or 0c. The optimization problem in expression 4 therefore can be used to find a better ensemble than the equivalent ensemble used Section 5.2 which still has fewer parameters than the baseline WGAN.
5.4 ENSEMBLES OUTPERFORM CGANS
We modify a WGAN to be conditional and call cWGAN. This cWGAN is trained on CIFAR 10. and we compare eWGAN ensembles of WGANs. We do this because we showed in Section that eGANs are an ensemble of GANs with a specific type of parameter sharing.
As can be seen from Figure ensembles perform better than the cWGAN. The baseline WGAN model actually performs similarly to the eWGAN, which implies that the conditioning is not helping in this specific case. We hypothesize that our model (A 0.001) performs better because there are more parameters that are free in the optimization, instead of just the bias in the first layer. Thus, although cGANs are widely used, an ensemble with the regularization described 4.. can outperform them because the ensemble captures the disconnected structure of the data better.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 1oXiv:/701.077 2017.
Wasserstein GAN. arXiv preprint
8

---

Under review as a conference paper at ICLR 2021
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. https: /qithub
2018. Accessed: June 3, 2020.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In International Conference on Machine Learning (ICML), pp. 224-232. JMLR. org, 2017.
Matan Yosef and Daphna Weinshall. Gaussian mixture generative adversarial networks for diverse datasets, and the unsupervised clustering of images. arXiv preprint arXiv: 1808. 10356, 2018.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Tatjana Chavdarova and François Fleuret. SGAN: An alternative training of generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9407-9415, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), pp. 2172-2180, 2016.
Arnab Ghosh, Viveka Kulharia, Vinay P Namboodiri, Philip HS Torr, and Puneet K Dokania. Multi- Agent diverse generative adversarial networks. In Advances in Neural Information Processing 2018.
Systems (NeurIPS), pp.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), pp. 2672-2680, 2014.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume MIT press Cambridge, 2016.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. DeLiGAN: Genera- tive adversarial networks for diverse and limited data. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 166-174, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NIPS), pp. 6626-6637, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhhard Nessler, and Sepp Hochreiter. Two time-scale update rule for training GANs. htttps://githbb.. htw.://hh...ccoomo//o..o.... fuwninnfnjjk/KTTS-- 2020. Accessed: June 3, 2020.
Quan Hoang, Tu Dinh Nguyen, Trung Le, and Dinh Phung. MGAN: Training generative adversarial
nets with multiple generators. In International Conference on Learming Representations (ICLR). OpenReview, 2018
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196. 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401 44104 2019a.
Tero Karras, Samuli Laine, Miika Aittlla, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. arXiv preprint arXiv: 049588 2019b.
Mahyar Khayatkhoei, Maneesh K Singh, and Ahmed Elgammal. Disconnected manifold learning for generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 7343-7353. 2018.

---

Under review as a conference paper at ICLR 2021
Diederik P Kingma and Max Welling. arXiv:13/2.114,222 2013.
Auto-encoding variational bayes. arXiv preprint
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Yixing Lao. Pytorch-reversegan. 2017. Accessed: June 3, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):277-2222, 1998.
Zachary C Lipton and Subarna Tripathi. Precise recovery of latent vectors from generative adversarial networks. arXiv preprint arXiv: 102.00477 2017.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl- Dickstein. Unrolled generative adversarial networks. arXiv preprint arXiv: 1611.0261 2016.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 2014.
Gonçalo Mordido, Haojin Yang. and Christoph Meinel. Dropout- GAN: Learning from dynamic ensemble of discriminators. arXiv preprint arXiv:1807.11344 2018.
Tu Nguyen, Trung Le, Hung Vu, and Dinh Phung. Dual discriminator generative ad adversarial nets. In Advances in Neural Information Processing Systems (NIPS), pp. 2670-2680, 2017.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier GANs. Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 2642-2651. JMLR. org, 2017.
Teodora Pandeva and Matthias Schubert. MMGAN: Generative adversarial networks for multi-modal distributions. arXiv preprint arXiv: 00111116663...11 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint urXiv:1511.066 2015.
Walter Rudin. Principles of mathematical analysis. McGraw-hill New York, third edition, 1964.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015. doi: 1007//// 1263-015-0816-y.
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lučić, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5228-5237, 2018
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lučić, Olivier Bousquet, and Sylvain Gelly. As- com/ jaamsajjacl
sessing generative models via precision and recall.
2019.
Accessed: June 3, 2020.
Ilya O Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabrias and Bernhard Schölkopf. AdaGAN: Boosting generative models. In Advances in Neural Information Processing Systems (NIPS), pp. 5424 5433, 2017.
Yaxing Wang, Lichao Zhang, and Joost Van De Weijer. Ensembles of generative adversarial networks. arXiv preprint arXiv: /6]2.009), 2016.
10

---

Under review as a conference paper at ICLR 2021
A PROOFS
We first show that under Definition 1 our data is disconnected.
Proof of Lemma Since X, 2 22 o Xx Ø. we see that X' disconnected.
Remark 1. Note that if our data is disconnected, it doesn't t necessarily follow Definition This means that Definition i a stronger condition than just having disconnected data. This can be seen by the following counter-example. We denote a truncated Gaussian (u, ) where the distribution is non-zero on S. Let
1 Pxs 2 ()0,)))((..)))) 2 _V(0 1))
Px = Px,
2
be the true distributions. Note that (-00, 0) and (0, 00) are disconnected but do not follow Definition T because they are not compact. Moreover, we can learn this distribution easily by letting G be the identity function and having P3- AO, 1): )) it trivial to show that this results in Pg Px Hence, disconnected data is too weak of an assumption we need there to be non-zero distance between our disconnected sets and that is what Definition captures.
Proof of Theorem 1 Without loss of generality. the two components 21 and 2z We define = G-1(X) and note that G(B) must be disconnected because G can generate from least X1 and X2 and they are disconnected. Since, X' is disconnected and closed in IR?, we have that / is disconnected and closed in R because G' is continuous (Theorem 4.8 and 4.22 in (Rudin 1964)). Since is closed in RR this means that R B an open Moreover, R.th R is not empty because we know that R is connected and B is not. We also know that the Lebesgue measure
assume that G can generate at least from
nonempty, open set positive, hence have that
A(R B) >
Since À is equivalent to Pz, we have that P=(zE E R BB) > 0. Thus,
Po(xER" = Pz(= x)) Pz B)) 0,
as desired.
Since the data lies only X, we know that (x RR,2) 0 for any valid probability measure. However, we have that Pa(x e R > Hence, d(Pa. Px) > 0 for any metric d.
Proof of Corollary [
Proof of Theorem 2. The truncated latent distribution is denoted Pay and is defined as
Pz(Bn
(B) P2(2)
for any set B E F. Hence, we have that
Paiz (A) P2r (G
D2(G-1(A) 2)
Pz(Z)
1
(A)
Pz(Z)
Pz(Z) (A(
1
Pz(Z) Po(Anx)
Px(An
P2(Z)
Pz(Z) Px(A)
11

---

Under review as a conference paper at ICLR 2021
for any Ae F The last equality is true because Px = so that any points outside of X' have zero probability. For the result above, set A 1º to see that c c P= implying that
Paix (A) = Px(A)
AEF,
as desired.
Proof of Theorem 3
First prove i). If max
K [K--1 1l K 1 then the constraints
on are unnecessary so that the problem reduces to
K
K
min
max
v(OG., 0D.)=
min
max ViOG,,
o BGk DD 8PK
w.. 88k
8Dk
K
min max V(OG.: pp.),
0c, 0D.
which is equivalent to solving the optimization problem
min max V(0G, Onw), ke (1,...., K}. dG, 0n,
Thus, i) is shown.
Next, we prove ii). Suppose that O. Note that given distribution Px, we can restrict each component X₂ and normalize to get the restricted distributions
(x(A) Px. (A) Px((
for each A F and each k. Since assume that Px(X,) Px((K)) for all j, k and that Px(x) = '. (xt) = 1, we see that (x(X = for each This implies that for any measurable function f IRP R, we have that
Bx-Pr Pxs [f(x)] = Σ 1=1 Jx
(())Px((d))
K K
Σ
Px
k=1
JoEN
K
f(x) Px (ddz)
Since X'
U Xx k- 1
JeEK
KE [f(x)]
Suppose that V is the standard cross entropy objective function. We will use the notation V Vla. 0p; P) to show that we are evaluating V(Og, Op) with the data distribution P. Then, we see that
V(0a, Opi Px.) = Σ En- Px2 [log D(=)] +
D(G(=)))
KK Px [log D(x) p/og-
= KV 0p: Px).
Similarly, if V is the Wasserstein objective function, then
K
K
v(OG, 0p; =
Σ
Ex- Px2 D(G())
KE P -K=,,
KV(0,, 0p: Px).
12

---

Under review as a conference paper at ICLR 2021
This means that
K
K
K
min 0or
max
V(0Gz, 0Dz: Px.) s.t.
0D, 1 = s.t. Σ
0c,
8G 1o =
O 0pir,,,D
min max
V(0g. 0n; Px.)
1-1
min max KV O Px).
On
which is equivalent to the optimization problem in T as desired.
Theorem 7. Let G and D be the generator and discriminator network in a GAN. Suppose that for k (1....,K] K) we have that G5 and Dy have the same architectures as G and D, respectively. Moreover, assume that Px(X)) Pxx(xx, ( for all j, k. Then,
Suppose that X O. Then for all k {I....,KK K} we have that (0 T.)) is a solution 4 if and only (0G.. Te) is a solution to
ii) Suppose that (1,..., K} if and only if (0%, ()) is a solution to M
Then have that (0g 0%) is solution 4 for each k.
Proof of Theorem 7 First prove i). If X 0 then have that
K
0p, OD
Oc, 0c
on (.. Hence, the problem reduces to the unconstrained problem of
Next, we prove ii). Since λ - x0, any solution where 0D, 0p, tc, * 0g for all j. (1...., K) is suboptimal. Consequently, it means that the optimization problem in reduces to
min Oa. Per..... 0Gg
max
Σ
V V(OG, Op.; Px:)
Op, -Dp 11
A Σ
0a, Ocl
k-1
min max KV (V 0D: Be
0D
which is equivalent the optimization problem in T We mainly just outline the proof here because it is so similar to the proof of Theorem 3
Proof of Theorem 4 Note that Px is the total data distribution and that Px, is the distribution of each disconnected set. This means that
K
Xx
Px.
1
for some mixture coefficients ag > 0 so that -i-1 ag. 1. K.....,K, min max ((lG= PD.) min max Ex- [log Dx(x)] Ez-P, [lgg(1 - - (x(G)()))) Oc, OD
Fix an arbitrary k E (1,..., K}. Since Paş = Px,, we have that
Oc, dp,
has solution of Pas Px, from Theorem of (Goodfellow et al. 2014). Since this is true for
every & and since Px 2k mi Px., we leam the complete data distribution.
13

---

Under review as a conference paper at ICLR 2021
Proof of Theorem 5 First show that ensembles equivalent to cGANs under the right condi- tions.
Fix the architecture of the networks considered and only focus the generator. We want show that cGANs and ensembles are equivalent, so we first focus on the generators in cGANs. We define the set of functions which represent conditional versions of the fixed architecture
G(K) {
Go. R (1,.... Kj Rº B,0 are network parameters
where B is a matrix with K columns and whose rows depend on the width of the first hidden layer discussed in Section 2. The rest of the parameters represented as the vector 0 above. It is clear that for function in G(K), there could be many corresponding networks; some of these are due to activation and weight symmetries (Bishop 2006). This implicitly create an equivalence class of networks; in particular, two networks are equivalent if they are the same mapping, regardless of weights. These symmetries do not affect our argument but it subtlety to keep mind.
Obviously not every ensemble is the same as the generator in cGAN; however, we focus on very specific type ensemble. We define the set of all ensembles that have variable bias in the first layer as
GE(K) = { R -R") b%, are network parameters for each k
which a collection of K-tuples of functions that represent networks with our fixed architecture. In the definition above, for j * k. we see that Go.b; and Gø share the same parameter O, but the biases by and by may be different. Ensembles that are used in the construction of Gg(K) are essentially ensembles that have parameter sharing everywhere except in the bias term of the first layer; these biases are not constrained to be similar at all.
Since network that induces function in ((K) is the conditional version of the networks in the ensembles that are used to construct (K) ) as described in Section 2.1 then parameter equality implies functional equality. In other words, for Go. E G(K) and (G ) 95(() have that [x]
Go (z) for all z RR,, , le (1,..., , K) and all parameters 0, ba-
We will now show that there exists one-to-one correspondence between these two sets. Suppose that T: GE(K) > (K) is defined by
((((G.,,) (2. Go.b. (z)
for each z E TR. and E (1....,K) K). Informally, we map an ensemble to a single network by just picking the k-th network in the ensemble. For fixed 0 and b,.... bx that (G )=1)) is indeed function from R x (1,.... K} to Rº. Moreover, T(Go,,,,) is equal to (as function) to Go. E G(K) so that is well defined.
Let Go.B g(K) be given. Then we just let bi..... by be the columns of B and we see that T(GGo,,, 11) Go,b. by implies that T is surjective. Next suppose that Gø BB Gga _ are functions in G(K) with = = [bf..... . 6&l and B -bP
[b...... ......... l. . Then clearly we have that for
(Gon bg)) K and (Goo,og (bp. k- in (K( that
Gø og (z) (z)= Gg B- (z, k) = Go (z, Go Goo (z),
implying that T is injective.
Thus, is a one-to-one correspondence between GE(K) and G(K). Hence, for every ensemble of networks defined above, can find cGAN which is equivalent to the ensemble. This equivalence defined as the equivalence of the functions induced by these networks. The above result holds for any architecture and all K z Thus, it also holds for the discriminator networks.
Next, we want to show that a modified version of the optimization problem from optimization problem.
3 yields the cGAN
14

---

Under review as a conference paper at ICLR 2021
We begin with the generic optimization problem from 3 and that it can be rewritten
K
K
min
max O 0p ODK
V(OG. DDx) s.t.
10D - ODello Et s.t.
|G; - 0c. lo St
0G1 Pe....... AGK
min 00 0ei Bcx
max 0D
V(OGi: OD,) ) s.t. CD
s.t. Cc
DDK k=1
min
max DD o... DDK
Ev
O' (Bo)
0DD (Bn) 1) s.t. Cp
s.t. Ca.
k=1
where we simply use the name CD for the constraint
D 00D ID and similarly for CG-
This is purely for notational convenience. Likewise, simply denote 6c, [( (Ba)T, and similarly for dD., for each k. Keep in mind that BG and BD matrices such that the k-th column is the the bias the first layer the k-th network in the ensemble. So far, we have only introduced notational changes.
Consider what happens we change the constraints to
K
DD
10,
DD loo
j=k
K
C/2
Σ 16,s 8'c, O.
We have that BG and Bp are unconstrained and that 0,. forced to be equal to 0, for all and j. Similarly 0. = DD for all k and j. Hence, we can that the optimization problem above with the new constraint is
(1
0' (1''
cb) s.t. Co (BD) 1) s.t. %%) (Bo), x
1
[1000
0'D
min
max Og ..... ODx k=1
Oax
([
0g (Bc).a
[
1
0D (Bb),
min
max
v
0p ..... DDx
k=1
which is equivalent the eGAN optimization problem. Here, we just define 0g be shorthand for any one of the Oc,, vectors, since they are all the same.
Hence, a cGAN is equivalent solving the ensemble optimization problem in with a modified constraint.
Proof of Theorem 6. The proof for this is very similar to the proof for Theorem 5
B ESTIMATION OF ENSEMBLE PARAMETERS
In Section we assume that l Pk: is multinomial distribution of degree K parameters: for
Using the maximum likelihood estimator (Bishop) 2006) we obtain
MLE
1( i)
for i {1. Hinton 2009). a uniformly distributed random varialle For others one may have to calculate based class imbalances.
K). For datasets like MNIST (LeCun et 1998) and CIFAR- 10 (Krizhevsky &
15

---

Under review as a conference paper at ICLR 2021
C EXPERIMENTAL DETAILS
In this section we describe the details of our experiments.
C.I PERFORMANCE MEASURES
We use FID (Heusel et al. 2017). average MSE (Metz et al. 2016), precision, and recall (Sajjadi et al. 2018 to evaluate our models.
For FID, precision, and recall we use the official repositories (Heusel et al. 2020 Sajjadi et al. 2019). For each of these methods, we compare a set of generated images to set of images from the training set. For the FID calculation, we use the precalculated statistics for CIFAR-10 10 and compare to 10,000 generated images from trained networks. For precision and recall, we compare 10,000 generated images to 10,000 images in the training set. All other parameters are left the same.
For the average MSE calculation, which was empirically shown to work 100% the time on DCGAN architecture, such as WGAN. We modified the code in (Lao (017) so that it be with multiple restarts if desired. We experiments with 1000 iterations and restarts. We ran the code on 100 training images.
the algorithm introduced in (Lipton & Tripathi (017).
C.2 BASELINE MODEL
For the baseline model, we ran the default WGAN code for 1000 epochs on CIFAR-10. All other parameters are left at their default values.
C.3 FULL ENSEMBLE
To create the full ensemble, we just copied over the baseline model 10 times and trained each network pair (Gk; D&) in the ensemble single class of CIFAR-10. The training also lasted for 1000 epochs. This is equivalent to solving the optimization problem in (
C.4 EQUIVALENT ENSEMBLE
Normally WGAN trained with the following two architecture parameters: ngf 64 and ndf 64. However, to get 10% of the parameters we trained each ensemble component with ngf 15 and ndf = 20. The depth of the generator and discriminator in the equivalent ensemble are the same as in single WGAN, however, we modify the width of each corresponding layer so that the total parameters are fewer in the ensemble than the single WGAN. Specifically, the generator of the WGAN has 3. 576 704 parameters and each generator of the equivalent ensemble has 312, 004 parameters. The discriminator of the WGAN has 2, 765, 568 parameters and each discriminator ensemble has 272, 880 parameters. Reducing the width of each layer is necessarily the optimal way to reducing parameters in a network. We do this because it is easy and effective, not because we trying reduce parameters an optimal way, which equivalent to solving the optimization problem in
568
the equivalent
of the scope of this paper. This is
C.5 REGULARIZED ENSEMBLES
For all the ensembles with À > we use the equivalent ensemble architecture, while solving
C.6 THE CGAN MODEL
For this architecture, modify the baseline architecture and concatenate the class label, represented as one-hot vector, to the input of the generator and discriminator networks.
16


Reviews:
[{"review_id": "gVtlX1VM_g_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of contributions:**\nThis paper proposes to study how to model distributions with disconnected support. They show that a single continuous generator cannot model such distributions correctly. However they show that using an ensemble of generator instead of a single generator can indeed model such distributions. They then show that training an ensemble of GANs can be formulated as a constraint optimization problem, they show that this general formulation encompass several other GAN variants by varying the constraint, such as the standard GAN formulation, conditional GAN and gaussian-mixture GAN. Finally they show experimentally that ensemble GANs outperforms other GANs formulation on CIFAR.\n\n**Pros:**\nI find the paper quite well written and clear. The ideas are interesting and convincing, they rigorously show theoretically and experimentally the advantage of using ensemble methods for training GANs. I believe this work is significant and could lead to breakthrough in generative modeling or other settings where we might need to model distributions with disconnected support.\n\n**Cons:**\n- The theory seems to suggest that we only need to have an ensemble of generator and that we might not need to have an ensemble of discriminator. Might be interesting to see what happens if we use a single discriminator ? do we observe a big drop of performance ? If the theory is correct the performance shouldn't drop too much.\n- The experiments are quite convincing, I think they could be even more convincing with a toy experiment where the distribution clearly has disconnected support. It would also be nice to have a comparison to GM-GAN. The CIFAR dataset is also quite limited and larger scale experiment such as ILSVRC would be very interesting to see, but I understand that this requires a lot of computational ressources and can be hard to do. Finally it's always nice to have some confidence interval for the empirical results.\n\n**Minor comments or Typos:**\n- Typo in theorem 1: $i \\in \\{2, ..., K\\}$\n- In theorem 2: State explicitly that the condition needs to be valid for $\\forall A \\in \\mathcal{F}', A \\subset \\mathcal{X}$. \n- In section 3.3 about truncated distribution, the argument about having a mixture distribution for the prior on the latent Z being problematic is not very clear. While this indeed restrict the shape of the prior, the generative network should be able to compensate this, thus as long as the support of Z is disconnected the generator should be able to learn distribution with disconnected support.\n- In theorem 5 it would be nice to explicitly state how the optimization problem (3) is modified.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dcZsea4YJE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows that ensembles of GANs are better at learning disconnected distributions than a single GAN. They introduce a optimization procedure that allows to optimize the degree of parameter sharing between different GANs models in an ensemble. The theoretical findings are supportede empirically by experiments on the CIFAR-10 dataset.\n\nWhile the problem setting is an important one I fail to see the main novelty in this paper. The fact that continuous functions keep the connectedness of the input space in tact is not a novel finding and explains why a single GAN with an input vector sampled from a Gaussian (or similar) cannot model a disconnected data distribution.\n\nIn this respect, I believe Section 4.1 and especially equations three and four are the more interesting part of the paper. While other papers have already showed that training ensembles of GANs can increase the performance, eq (4) offers a way to control the parameter sharing in GAN ensembles which might help with regularization.However, the experiments for the approach outlined in section 4.1 are very sparse. Only one GAN (relatively \"old\") architecture is evaluated (WGAN) and on only one relatively small dataset (CIFAR-10).\n\nSince this approach introduces a new hyperparameter to control the effect of parameter-sharing more experiments on more datasets and GAN architectures are needed to verify the effectiveness. It would also be helpful to have the numeric results of Figures 1-3 instead of only graphs. E.g. in Fig 1 it often looks like the baseline WGAN and the equivalent ensemble (i.e. similar amount of parameters) have similar performance. All Figures show that the ensemble methods perform much better in the recall metric and it would be interesting to evaluate if this means that ensemble methods also improve mode dropping behavior and also how more modern GANs perform here.\n\nOverall:\n- I think equations (3) and (4) are the main novelty in this paper and the experiments should be expanded to evaluate this more\n- for this I suggest to perform experiments with more modern architectures (at least WGAN-GP) and on more complex datasets (CIFAR 100, ImageNet, subsets of LSUN, etc).\n- the experiments should then evaluate how much improvement we can actually achieve with ensembles and how different values of lambda affect the outcome on different GAN models and datasets\n\nMinor: Section 4.2 has a spelling mistake in its title (should be \"for\" instead of \"fo\" I guess).\n\nSome missing related work:\nKundu, J. N., Gor, M., Agrawal, D., & Babu, R. V. (2019). Gan-tree: An incrementally learned hierarchical generative framework for multi-modal data distributions. In Proceedings of the IEEE International Conference on Computer Vision (pp. 8191-8200).\n\nTanielian, U., Issenhuth, T., Dohmatob, E., & Mary, J. (2020). Learning disconnected manifolds: a no GANs land. International Conference on Machine Learning (pp. 6767-6776).", "extra_scores": {}, "extra_text": {}}, {"review_id": "atEo6HF_cD4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work proposes that most relevant datasets to the machine learning community today have support on a mixture of disconnected components. They argue that popular GAN models cannot fit distributions of this kind and provide a number of proofs to convince the reader of this claim. The authors discuss a number of simple modifications on top of standard GANs which can alleviate this issue. These include replacing the standard unimodal latent distribution of the generator with a mixture model and replacing the standard generator with an ensemble of generators. The authors demonstrate that by replacing the standard GAN with an ensemble (or a pseudo-ensemble) they can achieve improved performance over a standard GAN given a fixed parameter budget with respect to a number of different evaluation metrics. \n\n\nStrong areas:\n\nI found this paper very clear and easy to follow which I very much appreciate.  I feel the theorems were motivated well and the correct amount of detail was shown in the main paper. To the best of my knowledge, all presented theory appears correct, but I am not an expert in that area. I think this paper asks the right kind of questions about the types of data we deal with. Beyond the “manifold hypothesis” there is not much discussion (to my knowledge) of other generic and reasonable assumptions which can be made about the types of data we see in the world. I think it is important to study how those properties of data interact with the properties of our models. Curitally, I think it is important we study how these properties positively reinforce each other (such as the manifold hypothesis and the low-dimensional latent spaces of a GAN) or how they negatively reinforce each other (like discussed in this work). So this paper focuses on what I believe to be an important set of issues.\n\nWeaknesses:\n\nI think it is widely accepted that most of the models used today cannot actually fit the distributions of the data we train them on. There are many ways in which the models we use can be deficient and this paper presents one such way which few people are likely to disagree with. Beyond this observation, there are many, equally valid, reasons why our models may be deficient. For example, the data may lie on a lower dimensional manifold which has a different topological structure than the base distribution of our generator (such as a ring). This is an equally obvious concern and similar theory to that presented in this work could be used to demonstrate this. But, the solutions presented here will not fix that issue. If the authors believe that the disconnected support issue is significantly more important than other topological issues then they should discuss this and provide evidence for why the disconnectivity issue is more important. The disconnectivity issue is a subset of the general topology issue so further evidence is needed to motivate study on this specific issue ignoring the larger problem.\n\nBeyond this, I am left wondering why the authors restricted themselves to studying GANs. It seems to me that alternative classes of generative models will suffer from the same problems such as normalizing flows and VAEs (which are almost identical to GAN generators). It also seems to me that these models might be better candidates to study this effect due to their ability to compute exact likelihoods (or likelihood estimates in the case of VAEs).\n\nThe theory provided in this work was focused on what kind of distributions can be modeled by passing a fixed distribution through a learned, nonlinear mapping. For this reason, I feel the focus on GANs is very limiting since many popular models work like this. I think the paper would be much more beneficial to the community if they addressed these concerns in other classes of model.\n\nSome more nit-picky issues:\n\nFigures 1,2,3 would be more clear with a legend. I found myself going back and forth to the caption to figure out what was going on there. I am also concerned by the reported scores in these figures. It seems like the FIDs reported here are >100 which seems much higher than simple GANs (WGAN-GP gets around 36 I believe). Beyond that, I do not see much value in the way these results are presented. The plots allow us to see how these quantities evolve over training but that is not really discussed in the paper. I feel like a table would be more useful since some of the values do appear quite close. \n\nMy recommendation:\n\nFor the reasons stated above, I would recommend to reject this paper. From my perspective, the main focus of the paper feels too limited to have much impact. It seems to me there are a number of related issues that are just as pernicious which are not addressed at all in this work. If the authors provided more discussion of these issues and provided a convincing explanation as to why they are not as important as the disconnected sets issue, then I would be inclined to review this paper more favorably. Along those lines, I feel the theory developed here could be useful to study many classes of generative models. If the authors applied their theory to these additional models, I feel like this could be a much stronger contribution. This type of problem has been addressed within the flow community (one such reference is https://arxiv.org/pdf/2002.07101.pdf but there are many others along this line). The approach presented here is different and would be interesting to compare with the solutions provided by the flow community. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RECQxTo641l", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem that models like GANS which learn continuous mappings from a connected latent space, are incapable of producing a mapping that contains disconnected components. The authors argue that real-world classes like \"badger\" and \"zebra\" are indeed disconnected and thus the inability of GANs to learn a disconnected mapping poses a problem. The authors formalize the statement and argue instead that one should learn an ensemble of GANs. These aren't really ensembles as in classification but components of a mixture model. They are using each GAN to represent one component of a mixture distribution. They also argue that conditional GANs and GANs with mixture distributions over the latents are similar. The paper closes with interesting but insufficient experiments showing the benfits of ensembles of GANs on one dataset (CIFAR10).\n\nThe paper is clear and nice to read. However the main arguments are actually fairly obvious and well known to all theorists that have worked on these problems. In fact I know multiple people that have discussed these points years ago, not as a paper contribution in themselves but as a jumping off point. The first argument flows straightforwardly from basic facts about continuity and push-forward distributions. \n\nOne key issue is that the result in Theorem 1 is not so striking as it may appear. All it says is that there will be nonzero probability assigned to points outside of either of the data components. But how large must this probability be? It's easy to see with a simple 1D example on the number line that we might be able to make that probability vanishingly small, at the expense of ratcheting up the  Lipschitzness of the neural network to arbitrary extremes.  So disconnected distributions may still be learnable (in a learning theoretic sense) even if they are not exactly representable, with an error that could still approach 0, even if for any particular network the error would be nonzero. \n\nThe most novel result, and the only one that isn't already plainly evident is theorem 3, concerning a proposed objective for learning ensembles of GANs and its equivalence to the original GAN objective when certain conditions on the number of parameters that can differ between GANs is capped. Unfortunately, the paper does little to unpack intuition for why this result is true or why it is interesting. \n\nOverall it's a great start and I think the authors are on the right path. The writing is clear and the problem is important. But the authors need to go deeper, clarify what precisely is the learnability problem, why their method is justified, and/or to contribute far more by way of experimental support for the proposed ensembling techniques. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I am recommending rejection for this paper for the following reasons:

I agree that the main claim is an obvious consequence of the structure of the GAN generator and the prior.
I'm also not sure why the authors restricted their analysis to GANs, but that's not a super important point to me. 
More important is that the experimental validation of the ensembling idea is way below the bar for this conference. 
Moreover, I know the authors touched on this a bit, but all of these modern GAN variants that work on imagenet
are implicitly ensembling anyway through e.g. the conditioning input and the structure of the special batch-norm. 
Finally, the authors didn't respond to the reviews. 
Decision: Reject