Ensembles of Generative Adversarial Networks for Disconnected Data

1 INTRODUCTION . Generative networks , such as generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) and variational autoencoders ( Kingma & Welling , 2013 ) , have shown impressive performance in generating highly realistic images that were not observed in the training set ( Karras et al. , 2017 ; 2019a ; b ) . However , even state of the art generative networks such as BigGAN ( Brock et al. , 2018 ) generate poor quality imagery if conditioned on certain classes of ILSVRC2012 ( Russakovsky et al. , 2015 ) . We argue that this is due to the inherent disconnected structure of the data . In this paper , we theoretically analyze the effects of disconnected data on GAN performance . By disconnected , we mean that the data points are drawn from an underlying topological space that is disconnected ( the rigorous definition is provided below in Section 3.1 ) . As an intuitive example , consider the collection of all images of badgers and all images of zebras . These two sets are disconnected , because images of badgers do not resemble images of zebras , and modeling the space connecting these sets does not represent real images of animals . We rigorously prove that one can not use a single continuous generative network to learn a data distribution perfectly under the disconnected data model . Because generative networks are continuous , they can not map a connected latent space ( R ` ) into the disconnected image space , resulting in the generation of data outside of the true data space . In related work , ( Khayatkhoei et al. , 2018 ) has empirically studied disconnected data but does not formally prove the results in this paper . In addition , the authors use a completely unsupervised approach to attempt to find the disconnected components as a part of learning . In contrast , we use class labels and hence work in the supervised learning regime . Our suggested approach to best deal with disconnected data is to use ensembles of GANs . We study GANs in particular for concreteness and because of their widespread application ; however , our methods can be extended to other generative networks with some modification . Ensembles of GANs are not new , e.g. , see ( Nguyen et al. , 2017 ; Ghosh et al. , 2018 ; Tolstikhin et al. , 2017 ; Arora et al. , 2017 ) , but there has been limited theoretical study of their properties . We prove that ensembles can learn the data distribution under the disconnected data assumption and study their relationship to single GANs . Specifically , we develop a first-of-its-kind theoretic framework that relates single GANs , ensembles of GANs , conditional GANs , and Gaussian mixture GANs . The framework makes it easy to , e.g. , develop regularized GAN ensembles that encourage parameter sharing , which we show outperform cGANs and single GANs . While our primary focus here is on theoretical insight , we also conduct a range of experiments to demonstrate empirically that the performance ( measured in terms of FID ( Heusel et al. , 2017 ) , MSE to the training set ( Metz et al. , 2016 ) , Precision , and Recall ( Sajjadi et al. , 2018 ) ) increases when we use an ensemble of WGANs over a single WGAN on the CIFAR-10 dataset ( Krizhevsky & Hinton , 2009 ) . The performance increase can be explained in terms of three contributing factors : 1 ) the ensemble has more parameters and hence has higher capacity to learn complex distributions , 2 ) the ensemble better captures the disconnected structure of the data , and 3 ) parameter sharing among ensemble networks enables successful joint learning , which we observe can increase performance . We summarize our contributions as follows : • We prove that generative networks , which are continuous functions , can not learn the data distribution if the data is disconnected ( Section 3.2 ) . The disconnected data model is defined in Section 3.1 , where we argue that it is satisfied in many common datasets , such as MNIST , CIFAR10 , and ILSVRC2012 . Restricting the generator to a disconnected subset of the domain is one solution ( Section 3.3 ) , but we study a better solution : using ensembles . • We demonstrate how single GANs and ensembles are related ( Section 4.1 ) . We then prove that ensembles are able to learn the true data distribution under our disconnected data model ( Section 4.2 ) . Finally , we demonstrate that there is an equivalence between ensembles of GANs and common architectures such as cGANs and GM-GANs due to parameter sharing between ensemble components ( Section 4.3 ) . • We empirically show that , in general , an ensemble of GANs outperforms a single GAN ( Section 5.1 ) . This is true even if we reduce the number of parameters used in an ensemble so that it has fewer total parameters than a single GAN ( Section 5.2 ) . Finally , we empirically show that parameter sharing among ensemble networks leads to better performance than a single GAN ( Section 5.3 ) or even a cGAN ( Section 5.4 ) . 2 BACKGROUND AND RELATED WORK . 2.1 GENERATIVE ADVERSARIAL NETWORKS ( GANS ) . GANs are generative neural networks that use an adversarial loss , typically from another neural network ( Goodfellow et al. , 2014 ) . In other words , a GAN consists of two neural networks that compete against each other . The generator G : R ` Ñ Rp is a neural network that generates pdimensional images from an ` -dimensional latent space . The discriminator D : Rp Ñ p0 , 1q is a neural network which is trained to classify between the training set and generated images . As compositions of continuous functions ( Goodfellow et al. , 2016 ) , both G and D are continuous . G has parameters θG P R|θG| , where |θG| is the possibly infinite cardinality of θG . Similarly , D has parameters θD P R|θD| . The latent , generated , and data distributions are Pz , PG , and PX , respectively . We train this network by solving the following optimization problem : min θG max θD V pθG , θDq “ min θG max θD Ex „ PX rlogDpxqs ` Ez „ Pz rlogp1´DpGpzqqqs . ( 1 ) Here we write min and max instead of minimize and maximize for notational compactness , but we are referring to an optimization problem . The objective of this optimization is to learn the true data distribution , i.e. , PG “ PX . Alternatively , we can use the Wasserstein distance instead of the typical cross-entropy loss : V pθG , θDq “ Ex „ PXDpxq ´ Ez „ PzDpGpzqq restricted to those θG , θD which force D to be 1-Lipschitz as done in the WGAN paper ( Arjovsky et al. , 2017 ) . Thus , we will use V to denote either of these two objective functions . 2.2 GANS THAT TREAT SUBSETS OF DATA DIFFERENTLY . Ensembles of GANs . Datasets with many different classes , such as ILSVRC2012 ( Russakovsky et al. , 2015 ) , are harder to learn in part because the relationship between classes is difficult to quantify . Some models , such as AC-GANs ( Odena et al. , 2017 ) , tackle this complexity by training different models on different classes of data in a supervised fashion . In the AC-GAN paper , the authors train 100 GANs on the 1000 classes of ILSVRC2012 . The need for these ensembles is not theoretically studied or justified beyond their intuitive usefulness . Several ensembles of GANs have been studied in the unsupervised setting , where the modes or disconnected subsets of the latent space are typically learned ( Pandeva & Schubert , 2019 ; Hoang et al. , 2018 ; Khayatkhoei et al. , 2018 ) with some information theoretic regularization as done in ( Chen et al. , 2016 ) . These are unsupervised approaches which we do not study in this paper . Models such as SGAN ( Chavdarova & Fleuret , 2018 ) and standard GAN ensembles ( Wang et al. , 2016 ) use several GANs in part to increase the capacity or expressibility of GANs . Other ensembles , such as Dropout-GAN ( Mordido et al. , 2018 ) , help increase robustness of the generative network . Conditional GANs ( cGANs ) . Conditional GANs ( Mirza & Osindero , 2014 ) attempt to solve the optimization problem in ( 1 ) by conditioning on the class y , a one-hot vector . The generator and discriminator both take y as an additional input . This conditioning can be implemented by having the latent variable be part of the input , e.g. , the input to the generator will be rzT yT sT instead of just z . Typically , conventional cGANs have the following architecture modification . The first layer has an additive bias that depends on the class vector y and the rest is the same . For example , consider a multilayer perceptron , with matrix W in the first layer . Converting this network to be conditional would result in the following modification to the matrix in the first layer : Wconditional „ x y “ rW Bs „ x y “ Wx ` By “ Wx ` B¨ , k . Hence , we can think ofB as a matrix with columnsB¨ , k , k P t1 , . . . , Ku being bias vectors andW being the same as before . We pick a bias vector B¨ , k based on what class we are conditioning on but the other parameters of the network are held the same , independent of k. This is done to both the generator and the discriminator . Some cGANs condition on multiple layers , such as BigGAN ( Brock et al. , 2018 ) , or on different types of layers , such as convolutional layers , but our formulation here extends clearly to those other architectures . Gaussian Mixture GANs ( GM-GANs ) . The latent distribution Pz is typically chosen to be either uniform , isotropic Gaussian , or truncated isotropic Gaussian ( Goodfellow et al. , 2014 ; Radford et al. , 2015 ; Brock et al. , 2018 ) . We are not restricted to these distributions ; research has been conducted in extending and studying the affect of using different distributions , such as a mixture of Gaussians ( Ben-Yosef & Weinshall , 2018 ; Gurumurthy et al. , 2017 ) . 3 CONTINUOUS GENERATIVE NETWORKS CAN NOT MODEL DISTRIBUTIONS DRAWN FROM DISCONNECTED DATA . 3.1 DISCONNECTED DATA MODEL . We begin by introducing a new data model that accounts for disconnected data . Typical datasets with class labels satisfy this model ; we provide additional examples below . Definition 1 ( Disconnected data model ) . We assume that the data lies on K disjoint , compact sets Xk Ă Rp , k P t1 , . . . , Ku so that the whole data lies on the disjoint union of each component : Ť ¨ Kk “ 1 Xk “ X . Moreover , we assume that each component Xk is connected ( Rudin , 1964 ) . We then draw data points from these sets in order to construct our finite datasets . In Definition 1 , we let each Xk be compact in order to remove the degenerate case of having two components Xk and Xj that are arbitrarily close to one another , which is possible if we only assume that X is closed and disjoint . If that is the case , there are trivial counter-examples ( see the appendix ) to the theorems proved below . Lemma 1 . X is a disconnected set , and Xj is disconnected from Xk for j ‰ k. Disconnected datasets are ubiquitous in machine learning ( Khayatkhoei et al. , 2018 ; Hoang et al. , 2018 ; Pandeva & Schubert , 2019 ) . For example , datasets with discrete labels ( typical in classification problems ) will often be disconnected . We study this disconnected data property , because generative networks are unable to learn the distribution supported on such a dataset , as we show below .

Summary: This paper addresses the problem that models like GANS which learn continuous mappings from a connected latent space, are incapable of producing a mapping that contains disconnected components. The authors argue that real-world classes like "badger" and "zebra" are indeed disconnected and thus the inability of GANs to learn a disconnected mapping poses a problem. The authors formalize the statement and argue instead that one should learn an ensemble of GANs. These aren't really ensembles as in classification but components of a mixture model. They are using each GAN to represent one component of a mixture distribution. They also argue that conditional GANs and GANs with mixture distributions over the latents are similar. The paper closes with interesting but insufficient experiments showing the benfits of ensembles of GANs on one dataset (CIFAR10).