Ensembles of Generative Adversarial Networks for Disconnected Data

1 INTRODUCTION . Generative networks , such as generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) and variational autoencoders ( Kingma & Welling , 2013 ) , have shown impressive performance in generating highly realistic images that were not observed in the training set ( Karras et al. , 2017 ; 2019a ; b ) . However , even state of the art generative networks such as BigGAN ( Brock et al. , 2018 ) generate poor quality imagery if conditioned on certain classes of ILSVRC2012 ( Russakovsky et al. , 2015 ) . We argue that this is due to the inherent disconnected structure of the data . In this paper , we theoretically analyze the effects of disconnected data on GAN performance . By disconnected , we mean that the data points are drawn from an underlying topological space that is disconnected ( the rigorous definition is provided below in Section 3.1 ) . As an intuitive example , consider the collection of all images of badgers and all images of zebras . These two sets are disconnected , because images of badgers do not resemble images of zebras , and modeling the space connecting these sets does not represent real images of animals . We rigorously prove that one can not use a single continuous generative network to learn a data distribution perfectly under the disconnected data model . Because generative networks are continuous , they can not map a connected latent space ( R ` ) into the disconnected image space , resulting in the generation of data outside of the true data space . In related work , ( Khayatkhoei et al. , 2018 ) has empirically studied disconnected data but does not formally prove the results in this paper . In addition , the authors use a completely unsupervised approach to attempt to find the disconnected components as a part of learning . In contrast , we use class labels and hence work in the supervised learning regime . Our suggested approach to best deal with disconnected data is to use ensembles of GANs . We study GANs in particular for concreteness and because of their widespread application ; however , our methods can be extended to other generative networks with some modification . Ensembles of GANs are not new , e.g. , see ( Nguyen et al. , 2017 ; Ghosh et al. , 2018 ; Tolstikhin et al. , 2017 ; Arora et al. , 2017 ) , but there has been limited theoretical study of their properties . We prove that ensembles can learn the data distribution under the disconnected data assumption and study their relationship to single GANs . Specifically , we develop a first-of-its-kind theoretic framework that relates single GANs , ensembles of GANs , conditional GANs , and Gaussian mixture GANs . The framework makes it easy to , e.g. , develop regularized GAN ensembles that encourage parameter sharing , which we show outperform cGANs and single GANs . While our primary focus here is on theoretical insight , we also conduct a range of experiments to demonstrate empirically that the performance ( measured in terms of FID ( Heusel et al. , 2017 ) , MSE to the training set ( Metz et al. , 2016 ) , Precision , and Recall ( Sajjadi et al. , 2018 ) ) increases when we use an ensemble of WGANs over a single WGAN on the CIFAR-10 dataset ( Krizhevsky & Hinton , 2009 ) . The performance increase can be explained in terms of three contributing factors : 1 ) the ensemble has more parameters and hence has higher capacity to learn complex distributions , 2 ) the ensemble better captures the disconnected structure of the data , and 3 ) parameter sharing among ensemble networks enables successful joint learning , which we observe can increase performance . We summarize our contributions as follows : • We prove that generative networks , which are continuous functions , can not learn the data distribution if the data is disconnected ( Section 3.2 ) . The disconnected data model is defined in Section 3.1 , where we argue that it is satisfied in many common datasets , such as MNIST , CIFAR10 , and ILSVRC2012 . Restricting the generator to a disconnected subset of the domain is one solution ( Section 3.3 ) , but we study a better solution : using ensembles . • We demonstrate how single GANs and ensembles are related ( Section 4.1 ) . We then prove that ensembles are able to learn the true data distribution under our disconnected data model ( Section 4.2 ) . Finally , we demonstrate that there is an equivalence between ensembles of GANs and common architectures such as cGANs and GM-GANs due to parameter sharing between ensemble components ( Section 4.3 ) . • We empirically show that , in general , an ensemble of GANs outperforms a single GAN ( Section 5.1 ) . This is true even if we reduce the number of parameters used in an ensemble so that it has fewer total parameters than a single GAN ( Section 5.2 ) . Finally , we empirically show that parameter sharing among ensemble networks leads to better performance than a single GAN ( Section 5.3 ) or even a cGAN ( Section 5.4 ) . 2 BACKGROUND AND RELATED WORK . 2.1 GENERATIVE ADVERSARIAL NETWORKS ( GANS ) . GANs are generative neural networks that use an adversarial loss , typically from another neural network ( Goodfellow et al. , 2014 ) . In other words , a GAN consists of two neural networks that compete against each other . The generator G : R ` Ñ Rp is a neural network that generates pdimensional images from an ` -dimensional latent space . The discriminator D : Rp Ñ p0 , 1q is a neural network which is trained to classify between the training set and generated images . As compositions of continuous functions ( Goodfellow et al. , 2016 ) , both G and D are continuous . G has parameters θG P R|θG| , where |θG| is the possibly infinite cardinality of θG . Similarly , D has parameters θD P R|θD| . The latent , generated , and data distributions are Pz , PG , and PX , respectively . We train this network by solving the following optimization problem : min θG max θD V pθG , θDq “ min θG max θD Ex „ PX rlogDpxqs ` Ez „ Pz rlogp1´DpGpzqqqs . ( 1 ) Here we write min and max instead of minimize and maximize for notational compactness , but we are referring to an optimization problem . The objective of this optimization is to learn the true data distribution , i.e. , PG “ PX . Alternatively , we can use the Wasserstein distance instead of the typical cross-entropy loss : V pθG , θDq “ Ex „ PXDpxq ´ Ez „ PzDpGpzqq restricted to those θG , θD which force D to be 1-Lipschitz as done in the WGAN paper ( Arjovsky et al. , 2017 ) . Thus , we will use V to denote either of these two objective functions . 2.2 GANS THAT TREAT SUBSETS OF DATA DIFFERENTLY . Ensembles of GANs . Datasets with many different classes , such as ILSVRC2012 ( Russakovsky et al. , 2015 ) , are harder to learn in part because the relationship between classes is difficult to quantify . Some models , such as AC-GANs ( Odena et al. , 2017 ) , tackle this complexity by training different models on different classes of data in a supervised fashion . In the AC-GAN paper , the authors train 100 GANs on the 1000 classes of ILSVRC2012 . The need for these ensembles is not theoretically studied or justified beyond their intuitive usefulness . Several ensembles of GANs have been studied in the unsupervised setting , where the modes or disconnected subsets of the latent space are typically learned ( Pandeva & Schubert , 2019 ; Hoang et al. , 2018 ; Khayatkhoei et al. , 2018 ) with some information theoretic regularization as done in ( Chen et al. , 2016 ) . These are unsupervised approaches which we do not study in this paper . Models such as SGAN ( Chavdarova & Fleuret , 2018 ) and standard GAN ensembles ( Wang et al. , 2016 ) use several GANs in part to increase the capacity or expressibility of GANs . Other ensembles , such as Dropout-GAN ( Mordido et al. , 2018 ) , help increase robustness of the generative network . Conditional GANs ( cGANs ) . Conditional GANs ( Mirza & Osindero , 2014 ) attempt to solve the optimization problem in ( 1 ) by conditioning on the class y , a one-hot vector . The generator and discriminator both take y as an additional input . This conditioning can be implemented by having the latent variable be part of the input , e.g. , the input to the generator will be rzT yT sT instead of just z . Typically , conventional cGANs have the following architecture modification . The first layer has an additive bias that depends on the class vector y and the rest is the same . For example , consider a multilayer perceptron , with matrix W in the first layer . Converting this network to be conditional would result in the following modification to the matrix in the first layer : Wconditional „ x y “ rW Bs „ x y “ Wx ` By “ Wx ` B¨ , k . Hence , we can think ofB as a matrix with columnsB¨ , k , k P t1 , . . . , Ku being bias vectors andW being the same as before . We pick a bias vector B¨ , k based on what class we are conditioning on but the other parameters of the network are held the same , independent of k. This is done to both the generator and the discriminator . Some cGANs condition on multiple layers , such as BigGAN ( Brock et al. , 2018 ) , or on different types of layers , such as convolutional layers , but our formulation here extends clearly to those other architectures . Gaussian Mixture GANs ( GM-GANs ) . The latent distribution Pz is typically chosen to be either uniform , isotropic Gaussian , or truncated isotropic Gaussian ( Goodfellow et al. , 2014 ; Radford et al. , 2015 ; Brock et al. , 2018 ) . We are not restricted to these distributions ; research has been conducted in extending and studying the affect of using different distributions , such as a mixture of Gaussians ( Ben-Yosef & Weinshall , 2018 ; Gurumurthy et al. , 2017 ) . 3 CONTINUOUS GENERATIVE NETWORKS CAN NOT MODEL DISTRIBUTIONS DRAWN FROM DISCONNECTED DATA . 3.1 DISCONNECTED DATA MODEL . We begin by introducing a new data model that accounts for disconnected data . Typical datasets with class labels satisfy this model ; we provide additional examples below . Definition 1 ( Disconnected data model ) . We assume that the data lies on K disjoint , compact sets Xk Ă Rp , k P t1 , . . . , Ku so that the whole data lies on the disjoint union of each component : Ť ¨ Kk “ 1 Xk “ X . Moreover , we assume that each component Xk is connected ( Rudin , 1964 ) . We then draw data points from these sets in order to construct our finite datasets . In Definition 1 , we let each Xk be compact in order to remove the degenerate case of having two components Xk and Xj that are arbitrarily close to one another , which is possible if we only assume that X is closed and disjoint . If that is the case , there are trivial counter-examples ( see the appendix ) to the theorems proved below . Lemma 1 . X is a disconnected set , and Xj is disconnected from Xk for j ‰ k. Disconnected datasets are ubiquitous in machine learning ( Khayatkhoei et al. , 2018 ; Hoang et al. , 2018 ; Pandeva & Schubert , 2019 ) . For example , datasets with discrete labels ( typical in classification problems ) will often be disconnected . We study this disconnected data property , because generative networks are unable to learn the distribution supported on such a dataset , as we show below .

Summary: This work proposes that most relevant datasets to the machine learning community today have support on a mixture of disconnected components. They argue that popular GAN models cannot fit distributions of this kind and provide a number of proofs to convince the reader of this claim. The authors discuss a number of simple modifications on top of standard GANs which can alleviate this issue. These include replacing the standard unimodal latent distribution of the generator with a mixture model and replacing the standard generator with an ensemble of generators. The authors demonstrate that by replacing the standard GAN with an ensemble (or a pseudo-ensemble) they can achieve improved performance over a standard GAN given a fixed parameter budget with respect to a number of different evaluation metrics.