microGAN: Promoting Variety through Microbatch Discrimination

1 INTRODUCTION . Generative adversarial networks ( Goodfellow et al . ( 2014 ) ) , or GANs , consist of a framework describing the interaction between two different models - one generator ( G ) and one discriminator ( D ) - that are trained together . While G tries to learn the real data distribution by generating realistic looking samples that are able to fool D , D tries to do a better job at distinguishing between real and the fake samples produced by G. Although showing very promising results across various domains ( Edwards and Storkey ( 2015 ) ; Ho and Ermon ( 2016 ) ; Yu et al . ( 2017 ) ; Kim et al . ( 2017a ) ; Yang et al . ( 2017 ) ; Donahue et al . ( 2018 ) ) , GANs have also been continually associated with instability in training , more specifically mode collapse ( Kim et al . ( 2017b ) ; Arjovsky and Bottou ( 2017 ) ; Mescheder et al . ( 2017 ) ; Che et al . ( 2016 ) ; Arjovsky et al . ( 2017 ) ) . This behavior is observed when G is able to fool D by only generating samples from the same data mode , leading to very similar looking generated samples . This suggests thatG did not succeed in learning the full data distribution but , instead , only a small part of it . This is the main problem we are trying to solve with this work . The proposed solution is to use multiple discriminators and assign each D a different portion of the real and fake minibatches , i.e. , microbatch . Then , we update each D ’ s task to discriminate between samples coming from its assigned fake microbatch and samples from the microbatches assigned to the other discriminators , together with the real samples . We call this microbatch discrimination . Throughout training , we gradually change from the originally proposed real and fake discrimination by Goodfellow et al . ( 2014 ) to the introduced microbatch discrimination by the use of an additional diversity parameter α that ultimately controls the diversity in the overall minibatch . The main idea of this work is to force G to reduce its loss by inducing variety in the generated set , complicating each D ’ s task on separating the samples in its microbatch from the rest . Even though only producing very similar images would also complicate the desired discrimination , it would not benefit any of the models . This is due to the attribution of distinct probabilities by eachD to samples from and outside its microbatch being required to minimize G and D ’ s losses . Hence , all models in the proposed framework , called microGAN , benefit directly from diversity in the generated set . Our main contributions can be stated as follows : ( i ) proposal of a novel multi-adversarial GANs framework1 ( Section 3 ) that mitigates the inherent mode collapse problem in GANs ; ( ii ) empirical evidence on multiple datasets showing the success of our approach in promoting sample variety since early stages of training ( Section 4 ) ( iii ) Competitiveness against other previously proposed methods on multiple datasets and evaluation metrics ( Section 5 ) . 1Code will be available upon publication . 1.1 RELATED WORK . Previous works have optimized GANs training by changing the overall models ’ objectives , either by using discrepancy measurements ( Li et al . ( 2017 ) ; Sutherland et al . ( 2016 ) ) or different divergence functions ( Nowozin et al . ( 2016 ) ; Uehara et al . ( 2016 ) ) to approximate the real data distribution . Moreover , Zhao et al . ( 2016 ) ; Berthelot et al . ( 2017 ) ; Unterthiner et al . ( 2017 ) proposed to use energy-driven objective functions to encourage sample variety , Mroueh et al . ( 2017 ) tried to match the mean and covariance of the real data , and Metz et al . ( 2016 ) used an unrolled optimization of D to train G. Che et al . ( 2016 ) ; Warde-Farley and Bengio ( 2016 ) ; Wang et al . ( 2017 ) ; Berthelot et al . ( 2017 ) penalized missing modes by using an extra autoenconder in the framework . Salimans et al . ( 2016 ) performed minibatch discrimination by forcing D to condition its output on the similarity between the samples in the minibatch . Springenberg ( 2015 ) increasedD ’ s robustness by maximizing the mutual information between inputs and corresponding labels , while Lin et al . ( 2017 ) forced D to make decisions over multiple samples of the same class , instead of independently . Regarding using multiple discriminators , Neyshabur et al . ( 2017 ) extended the framework to several discriminators with each focusing in a low-dimensional projection of the data , set a priori . Durugkar et al . ( 2016 ) proposed GMAN , consisting of an ensemble of discriminators that could be accessed by the single generator according to different levels of difficulty . Nguyen et al . ( 2017 ) introduced D2GAN , introducing a single generator dual discriminator architecture where one discriminator rewards samples coming from the true data distribution whilst the other rewards samples coming from the generator , forcing the generator to continuously change its output . Mordido et al . ( 2018 ) applied adversarial dropout by omitting the feedback of a given D at the end of each batch . 2 GENERATIVE ADVERSARIAL NETWORKS . The original GANs framework ( Goodfellow et al . ( 2014 ) ) consists of two models : a generator ( G ) and a discriminator ( D ) . Both models are assigned different tasks : whilst G tries to capture the real data distribution pr , D learns how to distinguish real from fake samples . G maps a noise vector z , retrieved from a noise distribution pz , to a realistic looking sample belonging to the data space . D maps a sample to a probability p , representing the likeliness of that given sample coming from pr rather than from pg . The two models are trained together and play the following minimax game : min G max D V ( D , G ) = Ex∼pr ( x ) [ logD ( x ) ] + Ez∼pz ( z ) [ log ( 1−D ( G ( z ) ) ) ] , ( 1 ) whereD maximizes the probability of assigning samples to the correct distribution andGminimizes the probability of its samples being considered from the fake data distribution . Alternatively , one can also train G to maximize the probability of its output being considered from the real data distribution , i.e. , logD ( G ( z ) ) . Even though this changes the type of the game , by being no longer minimax , it avoids the saturation of the gradient signals at the beginning of training ( Goodfellow et al . ( 2014 ) ) , where G only receives continuously negative feedback , making training more stable in practice . However , since we employ multiple discriminators in the proposed framework , it is less likely that G does not receive any positive feedback from the whole adversarial ensemble ( Durugkar et al . ( 2016 ) ) . Therefore , we make use of the original value function in this work . 3 MICROGAN . In this work , we propose a novel generative multi-adversarial framework named microGAN , where we start by splitting each minibatch into several microbatches and assigning a unique one to eachD . The key aspect of this work is the usage of microbatch discrimination , where we change the original discrimination task of distinguishing between real and fake samples , as proposed in Goodfellow et al . ( 2014 ) , to each D distinguishing between samples coming or not from its fake microbatch . This change is performed in a gradual fashion , using an additional diversity parameter α . Thus , each D ’ s output gradually changes from the probability of a given sample being real to the probability of a given sample not belonging to its fake microbatch . Moreover , since each D is trained with different fake and real samples , we encourage them to focus on different data properties . Figure 1 illustrates the proposed framework . The proposed microbatch-level discrimination task leads to G making such discrimination harder for each D to lower its loss . Hence , G is forced to induce variety on the overall minibatch , making it a substantially harder task for each D to be able to separate its subset of fake samples in the diverse minibatch . Note that producing very similar samples across the whole minibatch would also make such discrimination difficult by making the whole minibatch the same . However , G also benefits from each D assigning distinct probabilities to samples from inside and outside its designed microbatch to lower its loss , making the generation of different samples in the minibatch a necessary requirement to obtain different outputs from D. Hence , all models in our framework benefit directly from sample variety in the generated set . In the microGAN scenario with a positive diversity parameter α , each D assigns low probabilities to fake samples from its microbatch and high probabilities to fake samples from the rest of the microbatches as well as samples from the real data distribution . Hence , fake samples in the rest of the minibatch , i.e. , not coming from its assigned microbatch , shall be given distinct output probabilities by each D. On the other hand , G minimizes the probability given by eachD to the samples outside its microbatch and maximizes the probability given to the fake samples assigned to that specificD . The value function of our minimax game is as follows : min G max { Dk } K∑ k=1 V ( Dk , G ) = K∑ k=1 Ex∼prDk ( x ) [ logDk ( x ) ] + Ez∼pzGDk ( z ) [ log ( 1−Dk ( G ( z ) ) ) ] +α× Ez′∼pzGD \ { zGDk } ( z ′ ) [ logDk ( G ( z ′ ) ) ] , ( 2 ) where K represents the number of total discriminators in the set . prDk represents real samples from Dk ’ s real microbatch , pzGDk indicates fake samples from Dk ’ s fake microbatch , and pzGD\ { zGDk } relates to the rest of the fake samples in the minibatch but not in pzGDk . α represents the diversity parameter responsible for penalizing the incorrect discrimination of fake samples coming from pzGD\ { zGDk } by each Dk . Note that α = 0 would represent the original GANs objective for each D in the set . Appropriate α values and its effects on the whole minibatch diversity are discussed next . 3.1 DIVERSITY PARAMETER α We control the weight of the microbatch discrimination in the models ’ losses by introducing an additional diversity parameter α . Lower α values lead to G significantly lowering its loss by generating realistic looking samples on each microbatch without taking much consideration on the variety of the overall minibatch . On the other hand , higher α values induce a stronger effect onG ’ s loss if each D is able to discriminate between samples inside and outside its microbatch . However , high values of α might compromise the realistic properties of the produced samples , since too much weight is given to the last part of Eq . 2 , being sufficient to effectively minimize G ’ s loss . Thus , using α > 0 represents an additional way of ensuring data variety within the minibatch produced by G at each iteration . An overview of different possible α settings follows below . 3.1.1 STATIC α First , we statically set α to values between 0 and 1 throughout the whole training . For the evaluation of the effects of each α value , we used a toy experiment of a 2D mixture of 8 Gaussian distributions ( representing 8 data modes ) firstly presented by Metz et al . ( 2016 ) , and further adopted by Nguyen et al . ( 2017 ) . We used 8 discriminators for all the experiments . Results are shown in Figure 2 . When setting α = 0 , G mode collapses on a specific mode , showing the importance of using positive α values to mitigate mode collapse . When setting 0.1 ≤ α ≤ 0.5 , G is able to capture all data modes during training . However , learning problems in the early stages are observed , with G only focusing on promoting variety in the generated samples . For higher α values ( α ≥ 0.6 ) , G was unable to produce any realistic looking samples throughout the whole training , focusing solely on sample diversity to lower its loss , suggesting the dominance of the last part of Eq . 2 . Hence , a mild , dynamic , manipulation of α values seems to be necessary for a successful training of G , ultimately meaning both realistic and diverse samples from an early training stage . 3.1.2 SELF-LEARNED α We dynamically set α over time by adding it as a parameter of G and letting it self-learn its values to lower its loss . However , we observed that G takes advantage of being able to reduce its loss by increasing α at a large rate , focusing simply on promoting diversity in the generated samples without much realism , similarly to what was observed when using α = 0.6 in the toy experiment ( Figure 2 ) . Hence , we suggest several properties that α should have so that diversity does not compromise the veracity of the generated samples . First , α should be upper bounded so that the last part of Eq . 2 ( responsible for sample diversity ) does not overpower the first part ( responsible for sample realism ) , ultimately not compromising the feedback given to G to also be able to generate realistic samples . Second , α ’ s growth should saturate over time , meaning that continuously increasing at large rates α is no longer an option to substantially decrease G ’ s loss over time . Lastly , to tackle the problem in learning of early to mid stages , we suggest that α should grow in a controlled fashion , so focus can also be given in the realistic aspect of the samples since the beginning of training . Thus , we propose to make α a function of β , where α ( β ) ∈ [ 0 , 1 [ , and let G regulate β instead of directly learning α . We evaluated regulating α over three different functions that have the desired properties : α ( β ) = αsigm ( β ) = Sigmoid ( β ) , β ≥ βsigm αsoft ( β ) = Softsign ( β ) , β ≥ βsoft αtanh ( β ) = Tanh ( β ) , β ≥ βtanh ( 3 ) with βsigm , βsoft , and βtanh representing the initial values of β when training begins for the respective functions . For all the experiments of this paper , we set βtanh = βsoft = 0 , to obtain a positive codomain , and βsigm = −1.8 , since we achieved better empirical results by starting β with this value ( for further discussion about the effects of using different βsigm on αsigm ( β ) ’ s growth please see the Appendix ) . Note that learning α without any constraints can be characterized as using the identity function ( α ( β ) = αident ( β ) = β ) . Thus , each used function promotes a different α growth over time . To ease presentation , we neglect to write β ’ s dependence for the rest of the manuscript and use only the function names to described each α setting : αsigm , αsoft , αtanh , and αident . Results on the toy dataset using the different proposed α functions are shown in Figure 3 . The benefits of increasing α in a milder fashion , as performed when using αsigm , are observed especially early on training , with G being concerned with the realism of the generated samples . On the other hand , when using αtanh and αsoft , the network takes longer to focus on the data realism ( 10K steps ) since it is able to reduce its loss significantly by simply promoting variety due to the steeper growth of α in the earlier stages on both functions . Nevertheless , as the functions gradually saturate , all α settings manage to eventually capture the real data distribution while still keeping the diversity in the generated samples . In conclusion , one can summarize microGAN ’ s training using these variations of selflearned α as the following : in the first iterations , G increases α to reduce its loss , expanding its output . As α starts to saturate and each D learns how to distinguish between real and fake samples , G is forced to lower its loss by creating both realistic and diverse samples .

Summary: The submission proposes to increase the variety of generated samples from GANs by a) using an ensemble of discriminators, and b) tasking them with distinguishing not only fake from real samples, but also their fake samples from the fake samples given to the respective other discriminators. The cost function of the minimax GAN optimization problem is changed accordingly. Experimental results suggest that this approach leads to improved results, both visually and w.r.t. FID metric.