Small-GAN: Speeding up GAN Training using Core-Sets

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have become a popular research topic . Arguably the most impressive results have been in image synthesis ( Brock et al. , 2018 ; Salimans et al. , 2018 ; Miyato et al. , 2018 ; Zhang et al. , 2018 ; 2017 ) , but they have also been applied fruitfully to text generation ( Fedus et al. , 2018 ; Guo et al. , 2018 ) , domain transfer learning ( Zhu et al. , 2017 ; Zhang et al. , 2017 ; Isola et al. , 2017 ) , and various other tasks ( Xian et al. , 2018 ; Ledig et al. , 2017 ; Zhu & Bento , 2017 ) . Recently , Brock et al . ( 2018 ) substantially improved the results of Zhang et al . ( 2018 ) by using very large mini-batches during training . The effect of large mini-batches in the context of deep learning is well-studied ( Smith et al. , 2017 ; Goyal et al. , 2017 ; Keskar et al. , 2016 ; Shallue et al. , 2018 ) and general consensus is that they can be helpful in many circumstances , but the results of Brock et al . ( 2018 ) suggest that GANs benefit disproportionately from large batches . In fact , Table 1 of Brock et al . ( 2018 ) shows that for the Frechet Inception Distance ( FID ) metric ( Heusel et al. , 2017 ) on the ImageNet dataset , scores can be improved from 18.65 to 12.39 simply by making the batch eight times larger . Unfortunately , increasing the batch size in this manner is not always possible since it increases the computational resources required to train these models – often beyond the reach of conventional hardware . The experiments from the BigGAN paper require a full ‘ TPU Pod ’ . The ‘ unofficial ’ open source release of BigGAN works around this by accumulating gradients across 8 different V100 GPUs and only taking an optimizer step every 8 gradient accumulation steps . Future research on GANs would be much easier if we could have the gains from large batches without these pain points . In this paper , we take steps toward accomplishing that goal by proposing a technique that allows for mimicking large batches without the computational costs of actually using large batches . In this work , we use Core-set selection ( Agarwal et al. , 2005 ) to sub-sample a large batch to produce a smaller batch . The large batches are then discarded , and the sub-sampled , smaller , batches are used to train the GAN . Informally , this procedure yields small batches with ‘ coverage ’ similar to that of the large batch – in particular the small batch tries to ‘ cover ’ all the same modes as are covered in the large batch . This technique yields many of the benefits of having large batches with much less computational overhead . Moreover , it is generic , and so can be applied to nearly all GAN variants . Our contributions can be summarized as follows : • We introduce a simple , computationally cheap method to increase the ‘ effective batch size ’ of GANs , which can be applied to any GAN variant . • We conduct experiments on the CIFAR and LSUN datasets showing that our method can substantially improve FID across different GAN architectures given a fixed batch size . • We use our method to improve the performance of the technique from Kumar et al . ( 2019 ) , resulting in state-of-the-art performance at GAN-based anomaly detection . 2 BACKGROUND AND NOTATION . Generative Adversarial Networks A Generative Adversarial Network ( or GAN ) is a system of two neural networks trained ‘ adversarially ’ . The generator , G , takes as input samples from a prior z ∼ p ( z ) and outputs the learned distribution , G ( z ) . The discriminator , D , receives as input both the training examples , X , and the synthesized samples , G ( z ) , and outputs a distribution D ( . ) over the possible sample source . The discriminator is then trained to maximize the following objective : LD = −Ex∼pdata [ logD ( x ) ] − Ez∼p ( z ) [ log ( 1−D ( G ( z ) ) ) ] ( 1 ) while the generator is trained to minimize1 : LG = −Ez∼p ( z ) [ logD ( G ( z ) ) ] ( 2 ) Informally , the generator is trained to trick the discriminator into believing that the generated samples G ( z ) actually come from the target distribution , p ( x ) , while the discriminator is trained to be able to distinguish the samples from each other . Inception Score and Frechet Inception Distance : We will refer frequently to the Frechet Inception Distance ( FID ) ( Heusel et al. , 2017 ) , to measure the effectiveness of an image synthesis model . To compute this distance , one assumes that we have a pre-trained Inception classifier . One further assumes that the activations in the penultimate layer of this classifier come from a multivariate Gaussian . If the activations on the real data are N ( m , C ) and the activations on the fake data are N ( mw , Cw ) , the FID is defined as : ‖m−mw‖22 +Tr ( C + Cw − 2 ( CCw ) 1/2 ) ( 3 ) Core-set selection : In computational geometry , a Core-set , Q , of a set P is a subset Q ⊂ P that approximates the ‘ shape ’ of P ( Agarwal et al. , 2005 ) . The problem of approximating the shape of a larger set using a smaller subset is formally called shape fitting Lempitsky & Boykov ( 2007 ) . Core-sets perform sequential selection of points for shape-fitting ( Agarwal et al. , 2005 ) . Core-sets are used to quickly generate approximate solutions to problems whose full solution on the original set would be burdensome to compute . Given such a problem2 , one computes Q , then computes the solution to the problem for Q and converts that into an approximate solution for the original set P . The general Core-set selection problem can be formulated several ways , here we consider the the minimax facility location formulation ( Farahani & Hekmatfar , 2009 ) : min Q : |Q|=k max xi∈P min xj∈Q d ( xi , xj ) ( 4 ) where k is the desired size of Q , and d ( . , . ) is a metric on P . Informally , the formula above encodes the following objective : find some set , Q , of points of size k such that the maximum distance between a point in P and its nearest point in Q is minimized . Since finding the exact solution to the minimax facility location problem is NP-Hard ( Wolsey & Nemhauser , 2014 ) , we will have to make do with a greedy approximation , detailed in Section 3.3 . Using the minimax facility location formulation is an appropriate heuristic since Agarwal et al . ( 2005 ) demonstrated that minimizing the maximum distance between Q and P is an effective approximate solution for shape fitting . 1 This is the commonly used “ Non-Saturating Cost ” . There are many others , but for brevity and since our technique we describe is agnostic to the loss function , we will omit them . 2 As an example , consider computing the diameter of a point-set ( Agarwal et al. , 2005 ) . Algorithm 1 GreedyCoreset Input : batch size ( k ) , data points ( x where |x| > k ) Output : subset of x of size k s← { } . Initialize the sampled set while |s| < k do . Iteratively add points to sampled set p← argmaxxi /∈s minxj∈s d ( xi , xj ) s← s ∪ { p } end whilereturn s 3 USING CORE-SET SAMPLING FOR GANS ( OR SMALL-GAN ) . We aim to use Core-set sampling to increase the effective batch size during GAN training . This involves replacing the basic sampling operation that is done implicitly when minibatches are created . This implicit sampling operation happens in two places : First , when we create a minibatch of samples drawn from the prior distribution p ( z ) . Second , when we create a minibatch of samples from the target distribution pdata ( x ) to update the parameters of the discriminator . The first of these replacements is relatively simple , while the second presents challenges . In both cases , we have to work around the fact that actually doing Core-set sampling is computationally hard . 3.1 SAMPLING FROM THE PRIOR . We need to sample from the prior when we update the discriminator and generator parameters . Our Core-set sampling algorithm doesn ’ t take into account the geometry of the space we sample from , so sampling from a complicated density may cause trouble . This problem is not intractable , but it ’ s nicer not to have to deal with it , so in the absence of any evidence that the choice of prior affects the training , we define the prior in our experiments to be a uniform distribution over a hypercube . To add Core-set sampling to the prior distribution , we randomly sample n points from the prior , where n is greater than the desired batch size , k. We then perform Core-set selection on the large batch of size n to create a batch of size k. By applying Core-set sampling on the randomly over-sampled prior , we obtain a small sparse batch that approximates the shape of the hypercube . The smaller batch is what ’ s actually used to perform an SGD step . 3.2 SAMPLING FROM THE TARGET DISTRIBUTION . Sampling from the target distribution is more challenging . The elements drawn from the distribution are high dimensional images , so taking pairwise distances between them will tend to work poorly due to concentration of distances ( Donoho et al. , 2000 ; Sinha et al. , 2019 ) , and the fact that Euclidean distances are semantically meaningless in image space ( Girod , 1993 ; Eskicioglu & Fisher , 1995 ) . To avoid these issues , we instead pre-process our data set by computing the ‘ Inception Embeddings ’ of each image using a pre-trained classifier ( Szegedy et al. , 2017 ) . This is commonly done in the transfer-learning literature , where it is generally accepted that these embeddings have nontrivial semantics ( Yosinski et al. , 2014 ) . Since this pre-processing happens only once at the beginning of training , it doesn ’ t affect the per-training-step performance . In order to further reduce the time taken by the Core-set selection procedure , and inspired by the Johnson-Lindenstrauss Lemma ( Dasgupta & Gupta , 2003 ) , we take random low dimensional projections of the Inception Embeddings . In practice , we multiply the Inception Embeddings by a fixed and random matrix , to find low-dimensional embeddings where the distances are nearly preserved ( Dasgupta & Gupta , 2003 ) . Combined with Core-set selection , this gives us low-dimensional representations of the training set images in which pairwise Euclidean distances have meaningful semantics . We can then use Core-set sampling on those representations to select images at training time , analogous to how we select images from the prior .

Summary: This paper addresses the challenging problem of how to speed up the training of GANs without using large mini-batch sizes and causing significant performance drop. To achieve this, the authors propose to use the method of core-sets, mainly inspired by recent use of core-set selection in active learning. The proposed method allows us to generate effectively large mini-batches though actually small during the training process, or more concretely, drawing a large batch of samples from the prior and then compress that batch using core-set selection. To address the curse of dimensionality issue for high-dimensional data like images, the authors suggest using a low-dimensional embedding based on Inception activations of each training image. Regarding the experimental evaluation, it is clearly shown that the proposed core-set selection greatly improves GAN training in terms of timing and memory usage, and allows significantly reducing mode collapse on a synthetic dataset. As a by-product, it is successfully applied to anomaly detection and achieves state-of-the-art results.