Small-GAN: Speeding up GAN Training using Core-Sets
Authors: Samarth Sinha
Han Zhang
Anirudh Goyal
Yoshua Bengio
Hugo Larochelle
Augustus Odena
Venue:  (2020)
Keywords: GANs
Coreset

Abstract: BigGAN suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large minibatch sizes. This finding is interesting but also discouraging -- large batch sizes  are slow and expensive to emulate on conventional hardware. Thus, it would be nice if there were some trick by which we could generate batches that were effectively big though small in practice. In this work, we propose such a trick, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of real images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected embeddings at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it helps us use GANs to reach a new state of the art in anomaly detection.

Under review as a conference paper at ICLR 2020
SMALL-GAN: SPEEDING UP GAN TRAINING USING CORE-SETS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work by by/Brock et.l (0188) suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actually small. In this work, we propose method do this, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of 'real' images, we create a cached dataset Inception activations cach training image, randomly project them down to a those projected activations
smaller dimension, and then use Coresct-selection at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it allows GANs to reach new state of the art in anomaly detection.
INTRODUCTION
Generative Adversarial Networks (GANs) GGoodfellow et al. 2014) have become a popular research topic. Arguably the most impressive results have been in image synthesis (Brock et al. 2018
Sal-
imans et al. 2018 Miyato et al.] 2018 Zhang et al. 20118] 2017). but they have also been applied fruitfully to text generation (Fedus et al. 2018 Guo et al., 2018). domain transfer learning (Zhu
2017 Zhang et al. 2017: Isola et al. 2017). 7) and various other tasks (Xian et al. 2018 8 cet al.
et al.
Ledig
2017 Zhu & Bento 2017
Recently, Brock et al. (2018) substantially improved the results of Zhang et al. (2018) by using very
large mini-batches during training. The effect of large mini-batches in the context of deep learning is well-studied (Smith et al. 2017 Goyal et al. 2017; Keskar et al. 2016; SSaalue al. (2018) and general consensus is that they can be helpful in many circumstancess but the results of Brock et al. (2018) suggest that GANs benefit disproportionately from large batches. In fact, Table I of Brock et al. (2018) shows that for the Frechet Inception Distance (FID) metric (Heusel et al.) (017) on the ImageNNet dataset, scores be improved from 18.65 to 12.39 simply by making the batch eight times larger.
Unfortunately, increasing the batch size in this manner is not always possible since it increases the computational resources required to train these models - often beyond the reach of conventional hardware. The experiments from the BigGAN paper require full 'TPU Pod', The 'unofficial" open source release of BigGAN works around this by accumulating gradients across 8 different V100 GPUs and only taking an optimizer step every 8 gradient accumulation steps. Future research on GANs would be much easier if we could have the gains from large batches without these pain points. In this paper, we take steps toward accomplishing that goal by proposing a technique that allows for mimicking large batches without the computational costs of actually using large batches.
In this work, we use Core-set selection (Agarwal et al., 2005) to sub-sample large batch to produce a smaller batch. The large batches are then discarded, and the sub-sampled, smaller, batches are used to train the GAN. Informally, this procedure yields small batches with 'coverage' similar to that of the large batch in particular the small batch tries to "cover' all the same modes are covered in the large batch. This technique yields many of the benefits of having large batches with much less computational overhead. Moreover, it is generic, and can be applied nearly all GAN variants.

---

Under review as a conference paper at ICLR 2020
Our contributions can be summarized as follows:
We introduce a simple, computationally cheap method to increase the 'effective batch size' of GANs, which can be applied any GAN variant.
We conduct experiments on the CIFAR and LSUN datasets showing that our method can substantially improve FID across different GAN architectures given a fixed batch size,
We use our method to improve the performance of the technique from Kumar et al. (2019) resulting in state-of-the-ad performance at GAN-basee -bseed anomaly detection.
BACKGROUND AND NOTATION
Generative Adversarial Networks A Generative Adversarial Network (or GAN) is a system of two neural networks trained 'adversarially'. The generator, G, takes as input samples from prior 2 p(z) and outputs the learned distribution, G(z). The discriminator, D, receives as input both the training examples, X, and the synthesized samples, G(2), and outputs distribution D(.) over the possible sample source. The discriminator then trained to maximize the following objective:
LD Ear- (log ((x)] Egsp(s)) [log(1 D(G(=)))
while the generator is trained to minimize
LG = - -===p(p (log:llooit D(G(=))
(2)
Informally, the generator trained to rrick the discriminator into believing that the generated sam- ples Gzz) actually come from the target distribution, p(x), while the discriminator is trained to be able to distinguish the samples from each other.
Inception Score and Frechet Inception Distance: We will refer frequently to the Frechet Incep- tion Distance (FID) (Heusel et al. 2017). to measure the effectiveness of an image synthesis model. To compute this distance, one assumes that have pre-trained Inception classifier. One fur- ther assumes that the activations in the penultimate layer of this classifier come from a multivariate Gaussian. If the activations on the real data are N(m, C)) and the activations on the fake data N (m,,,, C the FID is defined as:
Im- mw Tr C
Core-set selection: In computational geometry, a Core-set, Q, of a set is a subset Q C that approximates the 'shape' of P (Agarwal al. 2005). The problem of approximating the shape of a larger set using a smaller subset is formally called shape fitting Lempitsky & Boykov (2007). Core-sets perform sequential selection of points for shape-fitting (Agarwal al. 2005). Core-sets are used to quickly generate approximate solutions to problems whose full solution on the original set would be burdensome compute. Given such problem one computes Q, then computes the solution to the problem for Q and converts that into an approximate solution for the original set P.
The general Core-set selection problem can be formulated several ways, here we consider the the minimax facility location formulation (Farahani & Hekmatfar 2009):
min Q k P 2,EQ
min d(z;, 2z))
(4)
where & is the desired size of Q, and d(.,.) a metric on P. Informally, the formula above encodes the following objective: find some set, Q, of points of size k such that the maximum distance between a point in P and its nearest point in Q is minimized. Since finding the exact solution to the minimax facility location problem is NP-Hard (Wolsey & Nemhauser 2014), we will have to make do with a greedy approximation, detailed in Section 3.3 Using the minimax facility location formulation is appropriate heuristic since Agarwal et al. (2005) demonstrated that minimizing the maximum distance between Q and P is an effective approximate solution for shape fitting.
This the commonly used Moon-Saturating Cost". There are many others, but for brevity and since our technique we descrihe is agnostic to the loss function, we will omit them.
As example, consider computing the diameter of point-set (Agarwal
2005
2

---

Under review as a conference paper at ICLR 2020
Algorithm 1 GreedyCoreset
Input: batch size (k),, data points (z where |2| >
Output: subset of a of size k O
Initialize the sampled set Iteratively add points to sampled set
while < k do
p - arg max,, n, ming,e 25)
5 J {p}
end whilereturn
USING CORE-SET SAMPLING FOR GANS (OR SMALL-GAN)
We aim to use Core-set sampling to increase the effective batch size during GAN training. This involves replacing the basic sampling operation that is done implicitly when minibatches cre- ated. This implicit sampling operation happens in two places: First, when we create a minibatch of samples drawn from the prior distribution p(z). Second, when we create minibatch of samples from the target distribution Paata(x) (x) to update the parameters of the discriminator. The first of these replacements relatively simple, while the second presents challenges. In both cases, we have to work around the fact that actually doing Core-set sampling is computationally hard.
3.1 SAMPLING FROM THE PRIOR
We need to sample from the prior when we update the discriminator and generator parameters. Our Core-set sampling algorithm doesn't take into account the geometry of the space sample from, so sampling from a complicated density may cause trouble. This problem not intractable, but it's nicer not have to deal with it, so in the absence of any evidence that the choice of prior affects the training, we define the prior in our experiments to be uniform distribution over hypercube. To add Core-set sampling to the prior distribution, we randomly sample points from the prior, where is greater than the desired batch size, k. We then perform Core-set selection on the large batch of size n to create batch of size k. By applying Core-set sampling on the randomly over-sampled prior, we obtain a small sparse batch that approximates the shape of the hypercube. The smaller batch what's actually used to perform an SGD step.
3.2 SAMPLING FROM THE TARGET DISTRIBUTION
Sampling from the target distribution more challenging. The elements drawn from the distribution are high dimensional images, so taking pairwise distances between them will tend to work poorly due to concentration of distances (Donoho et al. 2000, Sinha al. 2019), and the fact that Fuclidean distances are semantically meaningless in image space (Girod) 1993 Eskicioglu & Fisher 1995).
To avoid these issues, we instead pre-process our data set by computing the 'Inception Embeddings' of each image using pre-trained classifier (Szegedy et al.) (017). This is commonly done in the transfer-learning literature, where it is generally accepted that these embeddings have nontrivial semantics (Yosinski et al., 2014). Since this pre-processing happens only once at the beginning of training. it doesn't affect the per-training-
performance.
In order to further reduce the time taken by the Core-set selection procedure, and inspired by the Lindess--sndssstruuss Lemma (Dasgupta & Gupta 2003), we take random low dimensional pro- jections of the Inception Embeddings. In practice, fixed and random matrix, to find low-dimensional embeddings where the distances are nearly pre- served (Dasgupta & Gupta 2003). Combined with Core-set selection, this gives low-dimensional representations of the training set images in which pairwise Euclidean distances have meaningful semantics. We can then use Core-set sampling on those representations to select images training time, analogous to how we select images from the prior.
multiply the Inception Embeddings by
3.3 GREEDY CORE-SET SELECTION
In the above sections, we have invoked Core-set selection while glossing over the detail that exactly solving the k-center problem is NP-hard. This is important, because propose Core-set
3

---

Under review as a conference paper at ICLR 2020
selection at every training s Fortunately, make do with approximate solution, which faster to compute: we use the greedy k-center algorithm (similar to Sener & Savarese (2017) summarized in Alg.
3.4 SMALL-GAN
Our full proposed algorithm for GAN training presented in Alg. 2 the underlying GAN framework and therefore can replace random sampling of mini-batches for all GAN variants. More implementation details and design choices are presented Section
Our technique is agnostic to
Algorithm 2 Small 1-GAN
Input: target batch size (k), starting batch size (n > k), Inception embeddings (Φ1)
Output: trained GAN
Initialize networks G and D
for step 1 to do
~ p(z)
D Sample n points from the prior
(())
Sample points from the data distribution >> Get cached embeddings for Get Core-set of Get Core-set of embeddings D Get æ corresponding to sampled embeddings
<(x) t 0r(x)
GreedyCoreset(s)
<(z) + GreedyCoreset((fc)
- 07 (w(x))
Update GAN parameters usual
end for
4 EXPERIMENTS
In this section we look at the performance of our proposed sampling method on various tasks: In the first experiment, we train GAN on a Gaussian mixture dataset with large number of modes and confirm our method substantially mitigates 'mode- mnode-rrppping. In the second, we apply our technique GAN-based anomaly detection (Kumar al.] (019) and significantly improve on prior results. Finally, we test our method on standard image synthesis benchmarks and confirm that our technique seriously reduces the need for large mini-batches in GAN training. The variety settings these experiments testifies to the generality of our proposed technique.
4.1 IMPLEMENTATION DETAILS
For our Core-set algorithm, the distance function, d(-,)) the f--norm for both the prior and target distributions. Since we use the current state-of-the-art GAN architectures, each GAN model is trained with the same hyper-parameters and optimizers as proposed in the corresponding paper, to ensure fair comparison. The only pyper-parameter altered is the batch-size, which is stated for all experiments. For over-sampling, use factor of for the prior p(z) and factor of 8 for the target, p(z), unless otherwise stated. We investigate the effects of different over-sampling factors in the ablation study in Section 4.6
4.2 MIXTURE OF GAUSSIANS
We first investigate problem mode dropping (Arora et al. 2018 in GANs, where the GAN generator is unable to recover some modes from the target data set. We investigate the performance of training a GAN to recover different number of modes of isotropic Gaussian distributions, with standard deviation of 0.05. We similar experimental setup as Azadi al. (2018). where our generator and discriminator are parameterized using 4 ReLU-fully connected networks, and use the standard GAN loss in Eq. and 2 To evaluate the performance of the models, generate
3 Though the Core-set sampling does happens on CPU and so could be done in parallel to the GPU opera- tions used to train the model, as long as the Corc-set sampling time doesn't exceed the time of forward and backward pass- which it doesn't.
4

---

Under review as a conference paper at ICLR 2020
Number of Recovered Modes
% Recovered Modes (Ours)
% High-Quality Samples (GAN)
% High-Quality Samples (Ours)
Modes (GAN)
25
100
100
95.76
98.9
36
100
100
92.73
95.34
49
98. 12
99.85
84.28
88.1
64
96.13
99.01
68.81
82.11
81
92.59
98.84
49.74
71.75
100)
90.67
97.33
23.31
49.87
Table 1: Experiments with large number of modes
Held-out Digit
Bi-GAN
MEG
Core-set+MEG
0.287
0.281
0.351
4
0.443
0.401
0.501
5
0.514
0.402
0.518
7
0.347
0.29
0.387
9
0.307
0.342
0.39
Table 2: Experiments with Anomaly Detection on MNIST dataset. The Held-out digit represents the digit that was held out the training set during training and treated as the anomaly class. The numbers reported is the area under the precision-recal curve.
10, 000 samples and assign them to their closest mode. As in Azadi et al. (2018). the metrics we to evaluate performance are: i) 'high quality samples', which are samples within standard deviations of the assigned mode and ii) "recovered modes' which are mixture components with at least assigned sample.
Our results are present in table[] where we experiment with an increasing number of modes. We see that the number of modes increases, normal GAN suffers from increased mode dropping and lower sample quality compared to Core-set selection. With 100 modes, Core-set selection recovers 97.33% of the modes compared 90.67% for the vanilla GAN. Core-set selection also generates 49.87% 'high quality' samples compared to 23.31 % for the vanilla GAN.
4.3 ANOMALY DETECTION
To whether method can be useful for more than just GANs, also apply it to the Maximum Entropy Generator (MEG) from Kumar et al. (2019). MEG an energy- based model whose training procedure requires maximizing the entropy of the samples generated from the model. Since MEG gives density estimates for arbitrary data points, it can be used for anomaly detection a fundamental goal of machine learning research (Chandola et al. (2009] Kwon al. (017) in which one aims to find samples that are 'atypical' given source data set. Kumar et al. (2019) do use MEG successfully for this purpose, achieving results close to the state-ol-the-art technique for GAN-based anomaly detection (Zenati et al. 2018). We hypothesized that since energy estimates can in theory be improved by larger batch sizes these results could be further improved by using Core-set selection, and we ran an experiment to confirm this hypothesis.
We follow the experimental set-up from Kumar et (019) by training the MEG with samples from a chosen MNIST digit left-out during training. Those samples then serve as the 'anomaly class' during evaluation. We report the area under the precision-recall curve and average the score over the last 10 epochs. The results are reported in Table 2 which provides clear evidence in favor of our above hypothesis: for all digits tested, adding Core-set selection to MEG substantially improves the results. By performing these experiments, selection, not to suggest that MEG superior to BiGANs (Zenati et al.) 2018) on the task. We think it's likely that similar improvements could be achieved by using Core-set selection with BiGANs.
aim to show the general applicability of Core-set
5

---

Under review as a conference paper at ICLR 2020
Small-GAN (batch-size 128)
Small-GAN (batch-size
Small-GAN
GAN (batch- size 128)
GAN (batch- size 256)
GAN (batch-
(batch-size
256)
size = 512)
-12)
18.75 + 0.2
16.73 + 0.1
17.9 = 0.1 16.22 + 0.3
15.68 + 0.2
15.08 + 0.1
Table 3: FID scores for CIFAR using SN-GAN as the batch-size is progressively doubled. The FID score is calculated using 50, 000 generated samples from the generator.
Small-GAN (batch-
GAN (batch-
GAN (batch-
GAN (batch-
size 64)
size = 64)
size = 128)
size 256)
13.08
14.82
13.02
12.63
Table 4: FID scores for LSUN using SAGAN the batch-size is progressively doubled. The FID score is calculated using 50, 000 generated samples from the generator. All experiments were run on the 'outdoor church' subset of the dataset.
4.4 IMAGE SYNTHEESS
We also conduct experiments on standard image synthesis benchmarks. To further show the generality of our method, experiment with two different GAN architectures and
CIFAR and LSUN:
two image datasets. We use Spectral Normalization-Gii GAN (Miyato et al.] 2018) and Self Attention- GAN (Zhang al., 2018) on the CIFAR (Krizhevsky al. (209)) and LSUN (Yu al., 2015) datasets, respectively. For the LSUN dataset, which consists of 10 different categories, we train the model using the 'outdoor church' subset of the data.
For evaluation, we measured the FID scores (Heusel et al. 2017) of 50, 000 generated samples from the trained model We compare the performance using SN-GANs with and without Core-set se- lection across progressively doubling batch sizes. We observe a similar effect to Brock et al. (2018): just by increasing the mini-batch size by a factor of from 128 512, we are able improve the FID scores from 18.75 to 15.68 for SN-GANs. This further demonstrates the importance of large mini-batches for GAN training. Adding Core-set selection significantly improves the perfor- mance of the underlying GAN for all batch-sizes. For a batch size of 128, our model using Core-set sampling significantly outperforms the normal SN-GAN trained with a batch size of 256, and is comparable to SN-GAN trained with batch size of 512. The results suggest that the models perform significantly better for any given batch size when Coreset-sampling is used.
However, Core-set sampling does become less helpful as the underlying batch size increases: for SN-GAN, the performance improvement at batch size of 128 is much larger than the improvement batch size of 512. This supports the hypothesis that Core-set selection works by approximating the coverage of a larger batch; a larger batch can already recover more modes of the data so under this hypothesis, we would expect Core-sct selection to help less.
We see similar results when experimenting with Self Attention GANs (SAGAN) (Zhang et al. 2018) on the LSUN dataset (Yu al. 2015)). Compared to our results with SN-GAN, increasing the batch size results in a smaller difference in the performance for the SAGAN model, but we still see the FID improve from 14.82 12.63 as the batch-size is increased by factor of Using Core-set sampling with batch size of 64, we are able to achieve comparable score to when the model is trained with batch size of 128. We believe that one reason for a comparably smaller advantage of using Core-set sampling on LSUN is the nature of the data itself: using the 'outdoor church' subset of LSUN reduces the total number of 'modes' possible in the target distribution, since images of churches have fewer differences than the images in CIFAR-10 data set. We see similar effects in the mixture of Gaussians experiment 4.2) where the relative difference between a GAN trained with and without Core-set sampling increases the number of modes increased.
Finally. in order to test that our method would work 'at scale, we ran an exper- iment on the ImageNet data set. Using the code at hhttps: //tp::///tthbbb com heykeet Self-Actention--o
ImageNet:
we trained two GANs: The first is trained exactly as described in the
'Note that we measure the performance of all the models using the PyTorch version of FID scores, and not the official Tensorflow one. We all experiments with the same code for accurate comparison.

---

Under review as a conference paper at ICLR 2020
Small-GAN (batch
SN-GAN (batch
SN-GAN (batch
SN-GAN (batch size = 512)
size 128)
size 128)
size = 256)
14.51
13.31
26.46
51.64
Table 5: Timing to perform 50 gradient updates for SN-GAN with and without Core-sets. The time is measured in seconds. All the experiments The sampling factor was 4 for the prior and 8 for the target distribution.
performed single NVIDIA Titan-XP GPU.
Small-GAN
B
C
D
E 17.45
16.73
18.75
18.09
17.03
17.88
Table 6: FID scores for CIFAR using SN-GAN. The experiment list is: A = Training an SN-GAN, B
Core-set selection directly on the images, Core-set applied directly on Inception embeddings without a random projection, D = Core-set applied only on the prior distribution, E = Core-set applied only on target distribution.
open-source code. The second is trained using Coreset selection, with all other dyper-paramen unchanged. Simply adding Coreset selection the existing SAGAN code materially improved the FID (which compute using 50000 samples): the baseline model had FID of 19.40 and the Core-set model had an FID of 17.33.
4.5 TIMING ANALYSIS
Since random sampling can be done very quickly, it is important to investigate the amount of time it takes to train GANs with and without Core-set sampling. We measured the time for SN-GAN to do 50 gradient steps the CIFAR dataset with various mini-batch sizes: the results are in Table On average, for each gradient step, the time added by performing Core-Set sampling only 0.024 seconds.
4.6 ABLATION STUDY
We conduct an ablation study to investigate the reasons the effectiveness of Core-set selection. We also investigate the effect of different sampling factors and other hyper-parameters. We run all ablation experiments on the task of image synthesis using SN-GAN (Miyato et al., 2018) with the CIFAR- 10 dataset (Krizhevsky et al. 2009). We use the same hyperparameters as in main image synthesis experiments and a batch size of 128, unless otherwise stated.
4.7 EXAMINATION OF MAIN PAREME-RRSMESERS
We examine i) the importance of the chosen target distribution for Core-set selection and ii) the importance of performing Core-set on that target distribution. The FID scores are reported in Table 6 The importance of the target distribution clear, since performing Core-set selection directly on the images (experiment B) performs similar to random-sampling. Experiment C supports our hypothesis that performing random projection on the Inception embeddings preserve semantic information while reducing the dimensionality of the features. This increases the effectiveness of Core-set sampling and reduces sampling time. Our ablation study also shows the importance of performing Core-set selection on both the prior and target distribution. The FID scores the models considerably worse when Core-set sampling is used either distribution alone.
4.8 EXAMINATION OF SAMPLING FACTORS
Another important hyper-paramete for training GANs using Core-set selection is the sampling fac- tor. In Table 7 we varied the factors by which both the prior and the target distributions were overesampled. We see that using for the sampling factor for the prior and for the sampling factor for the target distribution results in the best performance. One interesting observation: that the per- formance eventually starts degrading as the sampling factor is increased. Since the greedy algorithm in Alg. sequentially selects the point that is the furthest away from the the already sampled set,
7

---

Under review as a conference paper at ICLR 2020
B
C
G
H
18.01
17.8
17.59
17.12
16.83
16.73
16.9
17.95
20.79
Table 7: FID scores for CIFAR using SN-GAN. Each of the experiment shows a different pair of over- sampling factors for the prior and target distributions. The factors are listed as: sampling factor for prior distribution x sampling factor for target distribution. A 2 x 2: B = x 4: C=
x 4; R = x 4; F = x 8; G 8 x 8; = 16 x 16; 32 x 32
believe that when the sampling factors are set too high, the algorithm becomes sensitive to outliers. See the appendix for more discussion of this phenomenon.
RELATED WORK
Variance Reduction in GANs: Researchers have proposed reducing variance in GAN training from an optimization perspective, by directly changing the way each of the networks are optimized. Some have proposed applying the extragradient method (Chavdarova et al. 2019), and others have proposed casting the minimax two-player game as a Auriational-iequallle 2018 Brock et al. (018)) recently proposed reducing variance by using large mini-batches.
problem (Gidel et al.)
Stability in GAN Training: Stabilizing GANs has been extensively studied theoretically. Re- searchers have worked on improving the dynamics the two player minimax game a variety of ways (Nagarajan & Kolter 2017: Mescheder al., 2018 Mescheder 2018 Li et al. 2017b Arora
ct al. 2017). Training instability has been linked to the architectural properties of GANs: especially the discriminator (Miyato et al. 2018). Proposed architectural stabilization techniques include using Convolutional Neural Networks (CNNs) (Radford et 2015), using very large batch sizes (Brock al., 2018), using an ensemble of the discriminators (Durugkar al.) (016)). using spectral normalization for the discriminator (Miyato et al., 2018 8). adding self- elffaattenioon layers for the genera- tor and discriminator networks (Vaswani to global generator and discriminator using an DecembbleChavdarov & Fleuret 2018). Different objectives have also been proposed to stabilize GAN training (Arjovsky et al. 2017 Gulrajani al.) 2017
[017]
al. | (208)) and using iterative updates
Li al. 2017a Mao al. 2017
Mroueh & Sercu 2017 Bellemare et 2017).
Core-set sampling has been widely studied from an algorithmic perspective in attempts to find better approximate solutions to the original NP-Hard problem (Agarwal et al. 2005 Clarkson] 2010: Pratap & Sen 2018). The optimality of the sub-sampled solutions have also been studied theoretically (Barahona & Chudak 2005; Goldman| 197I). See Phillips (2016) for recent survey on Core-set selection algorithms. Core-sets have been applied to many machine learning problems such as k-means and approximate clustering (Har-Peled
Core-set Selection:
Mazumdar 2004
Har-Peled & Kushal, [0077] Bādoiu et al. (0022), active learning for SVMs (T'sng et al., 2005
2007], unsupervised subset selection for hidden Markov models (Wei et al. 2013) scalable Bayesian inference. (Huggins et al.) 2016) and mixture models (Feldman et al. 201 l). We are not aware of Core-set selection being applied to GANs.
Core-set Selection in Deep Learning: Core-set selection is largely underexplored in the Deep Learning literature, but interest has recently increased. [Sener & Savarese (2017) proposed to use Core-set sampling as batch-mode active learning sampler for CNNs. Their method used the em- beddings of trained network to sample from. MMssay et al. (2019) proposed using Core-set se- lection on the activations of a neural network for network compression. Core-set selection has also been used in continual learning to sample points for episodic memory (Nguyen et al.) 2017).
6 CONCLUSION
In this work we present a general way to mimic using a large batch-size in GANs while minimizing computational overhead. This technique uses Core-set selection and improves performance wide variety of contexts. This work also suggets further research: a similar method could be applied to other learning tasks where large mini-batches may be useful.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pankaj K Agarwal, Sariel Har-Peled, and Kasturi R Varadarajan. Geometric approximation via coresets. Combinatorial and computational geometry, 52:1-30, 2005.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875. 2017.
Wasserstein gan. arXiv preprint
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In Proceedings of the 34th International Conference on Machine Learning- Searning-Voluuee Volume 70, pp. 224-232. JMLR. org. 2017.
Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do gans learn the distribution? some theory and empirics. 2018.
Samanch Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discrim- inator rejection sampling. arXiv preprint arXiv:1810.06756 2018.
Mihai Bădoiu, Sariel Har-Peled, and Piotr Indyk. Approximate clustering via core-sets. Proceed- ings of the thiry-fourth annual ACM symposium on Theory of computing, pp. 250-257. ACM, 2002.
Francisco Barahona and FabiáN Chudak. Near-optimal solutions to large-scale facility location problems. Discrete Optimization, 2(1):35-50, 2005.
Mare G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance as a solution to biased wasserstein gradi- ents. arXiv preprint arXiv:/705. 10743, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint /80i 1096, 2018.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM com- puting surveys (CSUR), 41(3::15, 2009.
Tatjana Chavdarova and François Fleuret. Sgan: An alternative training of generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9407-9415, 2018.
Tatjana Chavdarova, Gauthier Gidel, François Fleuret, and Simon Lacoste-Julien. Reducing noise in gan training with variance reduced extragradient. arXiv preprint arXiv:1904.088: 2019.
Kenneth L Clarkson. Coresets, sparse greedy approximation, and the frank-wolfe algorithm. ACM Transactions on Algorithms (TALG), 6(4):63, 2010.
Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and linden- strauss. Random Structures & Algorithms, 22(1))60 2003.
David L Donoho et High-dimensional data analysis: The curses and blessings of dimensionality. AMS math challenges lecture, 1(2000):32, 2000.
Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative multi-adversarial networks. arXiv preprint 1r 1.016111.11677 16166 2016.
Ahmet M Eskicioglu and Paul S Fisher. Image quality measures and their performance. IEEE Transactions on communications, 43(12):2959-299 2965, 1995.
Reza Zanjirani Farahani and Masoud Hekmatfar, Facility location: concepts, models, algorithms and case studies. Springer, 2009.
William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the.. arXiv preprint 1801.077300073 2018.
Dan Feldman, Matthew Faulkner, and Andreas Krause. Scalable training of mixture models via coresets. In Advances in neural information processing systems, pp. 2142-2150, 2011.

---

Under review as a conference paper at ICLR 2020
Gauthier Gidel, Hugo Berard, Gaëtan Vignoud, Pascal Vincent, and Simon Lacoste-Julien A variational inequality perspective on generative adversarial networks. arXiv preprint 10X2051802.10551, 2018.
Bernd Girod. What's wrong with mean-squared error? 207-220, 1993.
Digital images and human vision, pp.
AJ Goldman. Optimal center location in simple networks. Transportation science, 5(2):212-221, 1971.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint (xXi::7706.02677 2017.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp- 5767-5777, 2017.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Sariel Har-Peled and Akash Kushal. Smaller coresets for k-median and k-means clustering. Discrete & Computational Geometry, 37(1):3-19, 2007.
Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing, pp. 291-300. ACM, 2004.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. In Advances in Neural Information Processing Systems, pp. 4080-4088, 2016.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609 1609.043666 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Rithesh Kumar, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv: 1901.08508, 2019.
Donghwoon Kwon, Hyunjoo Kim, Jinoh Kim, Sang Suh, Ikkyun Kim, and Kuinam J Kim. A survey of deep learning-basse based network anomaly detection. Cluster Computing, pp. 1-13, 2017.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Victor Lempitsky and Yuri Boykov. Global optimization for shape fitting. In 2007 IEEE Conference Computer Vision and Pattern Recognition, pp. 1-8. IEFE, 2007.
10

---

Under review as a conference paper at ICLR 2020
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pp. 2203-2213, 2017a.
Jerry Li, Aleksander Madry, John Peebles, and Ludwig Schmidt. Towards understanding the dy- namics of generative adversarial networks. arXiv preprint arXiv:1706.0988 2017b.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Confer- ence on Computer Vision, pp. 2794-2802, 2017.
Lars Mescheder. On the convergence properties of gan training. arXiv preprint 1.Xiv:8801.04400 1:16, 2018.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv:1801.00 04406, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprini arXiv: 2018.
Youssef Mroueh and Tom Sercu. Fisher gan. In Advances in Neural Information Processing Systems, pp. 2513-2523, 2017.
Ben Mussay, Margarita Osadchy, Vladimir Braverman, Samson Zhou, and Dan Feldman. Data- independent neural pruning via coresets, 2019.
Vaishnavh Nagarajan and Zico Kolter. Gradient descent gan optimization is locally stable. In Advances in Neural Information Processing Systems, pp. 5585-5595, 2011
2017.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard Turner. Variational continual learning. arXiv preprint a70iv: 10628, 2017.
Jeff M Phillips. Coresets and sketches. arXiv preprint arXiv:1601.00617, 2016.
Rameshwar Pratap and Sandeep Sen. Faster coreset construction for projective clustering via low- rank approximation. In International Workshop Combinatorial Algorithms, pp. 336-348.
Springer, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 2015.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport. arXiv preprint 1803.05573 2018.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint 1r..iv:170..004440 2017.
Christopher J Shallue, Jachoon Lee, Joe Antognini. Jascha Sohl-Dickstein, Roy Frostig, and George Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv: /.03600, 2018.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell, Variational adversarial active learning. arXiv preprint 1w0..v:9044.0030 2019.
Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V Le. Don't decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00488, 2017.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI Confer- ence on Artificial Intelligence, 2017.
Ivor W Tsang. James Kwok, and Pak-Ming Mak-Ming Cheung. Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research, 6(Apr):363-99) 2005.

---

Under review as a conference paper at ICLR 2020
Ivor W Tsang, Andras Kocsor, and James T Kwok. Simpler vector machines with enclosing balls. In Proceedings of the 24th international conference on Machine learning, pp. 911-9191111111111111 918. ACM, 2007.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-60090 2017.
Kai Wei, Yuzong Liu, Katrin Kirchhoff, and Jeff Bilmes. Using document summarization techniques for speech data subset selection. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 721-726, 2013.
Laurence A Wolsey and George L Nemhauser. Integer and combinatorial optimization. John Wiley & Sons, 2014.
Yongqin Xian, Tobias Lorenz, Bernt Schicle, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, pp. 3320-3328, 2014.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.033:
2015.
Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chan- drasekhar. Efficient gan-based anomaly detection. arXiv preprint arXiv: (00::8802.0000 1802.06222, 2018.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to reallstc image synthesis with stacked generative adver- sarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp- 5907-5915, 2017.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint (rXiv:1805.08311 2018.
Jia-Jie Zhu and José Bento. arXiv:1702.07956, 2017.
Generative adversarial active learning.
arXiv preprint
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2223323 2017.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 POTENTIAL DISTORTING AFFECTS OF CORE-SET SAMPLING
Theoretically, Core-set sampling of the training distribution can cause some distortion of the distri- bution learned by the generator. However, there are good ways to mitigate this, we will discuss. Moreover, in cases where Core-set sampling causes more distortion, a baseline GAN will tend to distort things in the opposite direction.
In more detail: We train GANs using 4 different algorithms on a mixture of two Gaussians with mixing coefficients of 0.01 and 0.99. For all 4 experiments, we use a batch size of 10. The results are as follows:
Vanilla GAN: Training 'vanilla' GAN on this mixture of two Gaussians results in the 0.01-mode being completely dropped. This is consistent with existing results on GANs. It's also important because it suggests that, even if Core-set sampling unavoidably distorting (which it's not below), there might not be a non-distorting baseline that works in the same situation.
Core-set Sampling Training a GAN with Core-set sampling (sampling factor of 20) results in con- siderable distortion: 90.9% of samples are from the 0.99-mode and 8.9% of samples are from the 0.01-mode (the rest of the samples were over standard deviations from either mean)
the 0.01 mode is over-represented by about a factor of 10. This makes a lot of sense: the final batch size is 10, and we take 200 samples to start, so the expected number of 0.01-mode samples in our original batch is 2. Core-set sampling will ensure we keep at least 1 of those 2 in our final batch of 10, yield- ing 10% representation for mode that should only have % representation. It's worth pointing out that you have to make the skew in mixture components quite high relative to the batch size for this to matter, but there likely exist interesting data-sets that have this characteristic.
Thus, we test two methods for reducing the distortion:
Anneal Sampling Factors: Training GAN with Core-set sampling but annealing the sampling factor from 20 to 2 during training results in 98.3% and 1.6% of samples from the 0.99 and 0.01 modes, respectively. This is substantial reduction in distortion.
Fine-Tune on Randomly Sampled Data: Training a GAN with Core-set sampling to start and then random sampling near the end results in 99.5% and 0.5% of samples from the 0.99 and 0.01 modes, respectively, for our best try. This method resulted some instability the longer the GAN is training randomly, the more likely the GAN was to drop the 0.01 mode. For this reason would probably recommend the annealing method in practice.
Final Observations: We have two observations to make about this: First, the FID should in principle be sensitive to such distortions, and so the fact Core-set sampling was able to improve the FID in some sense suggests that on CIFAR, LSUN, and ImageNet the distortion was not that significant. Second, Azadi et al. (2018) seems like another promising way to reduce distortion, but we did not test it since the annealing method described above seems satisfactory.
13


Reviews:
[{"review_id": "rkloqqAycS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies core-set selection to the training of GANs. The motivation is to limit the minibatch size with suitably sampled sets of datapoints. The proposed technique is relatively reasonable: e.g. extract features from an image, reduce dimensionality by the taking random projections, then run Core-Set selection. The Core-Set selection part of the method is modular from the rest of the GAN training, and can be applied easily. \n\nGenerally, I think this is reasonabl work. While the idea itself is not extremely novel, it is interesting to see CoreSets applied to GANs. The paper can be made stronger if there is more discussion about the theory of CoreSets and how good are the heuristics used in this paper. \n\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJlFubv3FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Training with large batches provides a disproportionate improvement for GANs (e.g. FID drops from 18.65 to 12.39 by simply increasing the batch size by a factor of 8 for BigGAN). The authors point out that not everybody has access to the computing power which is required to run large batches. Therefore, this paper proposes a method to select the image of the batch and thereby obtaining the benefits of large batches while running only small batches.\n\nThe authors perform coreset selection of a large set of images (actually a greedy variant from Sener & Savarese). The selected images are then used for training with small batch size. The coreset selection is applied to the inception activations of the images (or a randomly downsampled version of them). Experiments show that the trained GANs obtain better mode coverage, improve anomaly detection,  and obtain GANs with higher FID scores on image synthesis. \n\nThe paper is well motivated and solutions that prevent having to use large batches will have a significant impact on the field. \n\nConclusion: At the moment I recommend a weak reject. The technical contribution of the paper is rather small, and also the depth of analysis could be improved. Moreover, I think experimental results could have been more complete. However, given the importance of the problem addressed in the paper (and the little existing work) I could be open to increasing my score if my concerns are addressed. \n\nMain points:\n1. GANs aim to generate high-quality images. In addition, they aim to generate these high-quality images according to the distribution of the train dataset. The main danger of not randomly sampling from the train distribution is that the trained GAN does no longer generate images according to the train distribution. I think this issue should be more clearly discussed and evaluated in the paper (for some applications this might not be a big problem for others yes). For example, if you would introduce mixing coefficients for the Gaussian mixture and Gaussians with different variances (section 4.2), it could be possible that the GAN would generate according to these mixing coefficients with more accuracy than small-GAN (because the coreset selection would introduce a bias). The proposed metrics do not measure this. \nMaybe this could simply be solved by adding some final epochs which just sample randomly again? \n2. A naive approach to creating large batches is by only updating the network every N batches, and summing the gradients of the N batches. I would like to see this option discussed and compared to. I can see how the BN is different from a large batch but it would still be interesting to see. \n3. I think the examination of sampling factors should be explained in more detail. The importance of the sampling factor of the prior is harder to understand (and might be necessary to counter the effect I discuss in 1?) Is it not expected that both these sampling factors should be equal? \n\n\nMinor points (do not need to be addressed in rebuttal):\n-Since FID several other GAN evaluation metrics have been proposed. I think the authors could also consider ‘Assessing Generative Models via Precision and Recall’ or ‘Improved precision and recall metric for assessing generative models’ for a more complete insight insight \n-I am not convinced this GAN needs a name (small-GAN), especially since it can be applied to other GAN architectures. \n-The claim in the abstract for state of the art in anomaly detection should be removed (no extensive study nor comparison is performed)\n-I prefer to see the venue of publication when possible (for example Sener & Savarese is ICLR 2018)\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJlnHqQhFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper addresses the challenging problem of how to speed up the training of GANs without using large mini-batch sizes and causing significant performance drop. To achieve this, the authors propose to use the method of core-sets, mainly inspired by recent use of core-set selection in active learning. The proposed method allows us to generate effectively large mini-batches though actually small during the training process, or more concretely, drawing a large batch of samples from the prior and then compress that batch using core-set selection. To address the curse of dimensionality issue for high-dimensional data like images, the authors suggest using a low-dimensional embedding based on Inception activations of each training image. Regarding the experimental evaluation, it is clearly shown that the proposed core-set selection greatly improves GAN training in terms of timing and memory usage, and allows significantly reducing mode collapse on a synthetic dataset. As a by-product, it is successfully applied to anomaly detection and achieves state-of-the-art results.\n\nStrengths:\nThe paper is generally well written and clearly presented.  As mentioned in the text, the use of core-sets is not novel in machine learning, but unfortunately not yet sufficiently explored in deep learning, and there are still few useful tools available in the literature. I believe this work will have a positive impact on the community and especially help establishing more efficient methods for training GANs.\n\nWeaknesses:\n- Experimental results are indeed very promising, however, GAN implementation details and hyperparameters used for training, such as optimizer and learning rate, do not seem to be mentioned in the text. I think this would be helpful for readers to better understand how this all works.\n- There does not seem to be any discussion on the convergence and stability of GAN training, which should be clarified in the experimental section.\n- On page 3, in Sect. 3.2,  I find “random low dimensional projections of the Inception Embeddings” is not clear, more technical details should be provided.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use greedy core set sampling to improve GAN training with large batches. Although the problem is clear and the solution works, reviewers have raised several concerns. One concern is that the technical novelty is limited; another (in the first version) that even a simpler version of gradient accumulation can solve the  main task (rather that computing core-sets). In the end, some discussion was done, with quite a few additions and experiments done by the authors. The final concern that seemingly was not addressed: the gradient accumulation seems to give the same numbers as large batches, thus you can 'mimic' large batch sizes with smaller ones and gradient accumulation, making the main claim of the paper questionable. The achievement of SOTA is good, but it is not clear wether it is due to the proposed technique, or rather smart tuning of a larger set of hyperparameters. Thus, I would agree with the concern of Reviewer1.
Decision: Reject