Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper applies core-set selection to the training of GANs. The motivation is to limit the minibatch size with suitably sampled sets of datapoints. The proposed technique is relatively reasonable: e.g. extract features from an image, reduce dimensionality by the taking random projections, then run Core-Set selection. The Core-Set selection part of the method is modular from the rest of the GAN training, and can be applied easily. 

Generally, I think this is reasonabl work. While the idea itself is not extremely novel, it is interesting to see CoreSets applied to GANs. The paper can be made stronger if there is more discussion about the theory of CoreSets and how good are the heuristics used in this paper. 




review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Training with large batches provides a disproportionate improvement for GANs (e.g. FID drops from 18.65 to 12.39 by simply increasing the batch size by a factor of 8 for BigGAN). The authors point out that not everybody has access to the computing power which is required to run large batches. Therefore, this paper proposes a method to select the image of the batch and thereby obtaining the benefits of large batches while running only small batches.

The authors perform coreset selection of a large set of images (actually a greedy variant from Sener & Savarese). The selected images are then used for training with small batch size. The coreset selection is applied to the inception activations of the images (or a randomly downsampled version of them). Experiments show that the trained GANs obtain better mode coverage, improve anomaly detection,  and obtain GANs with higher FID scores on image synthesis. 

The paper is well motivated and solutions that prevent having to use large batches will have a significant impact on the field. 

Conclusion: At the moment I recommend a weak reject. The technical contribution of the paper is rather small, and also the depth of analysis could be improved. Moreover, I think experimental results could have been more complete. However, given the importance of the problem addressed in the paper (and the little existing work) I could be open to increasing my score if my concerns are addressed. 

Main points:
1. GANs aim to generate high-quality images. In addition, they aim to generate these high-quality images according to the distribution of the train dataset. The main danger of not randomly sampling from the train distribution is that the trained GAN does no longer generate images according to the train distribution. I think this issue should be more clearly discussed and evaluated in the paper (for some applications this might not be a big problem for others yes). For example, if you would introduce mixing coefficients for the Gaussian mixture and Gaussians with different variances (section 4.2), it could be possible that the GAN would generate according to these mixing coefficients with more accuracy than small-GAN (because the coreset selection would introduce a bias). The proposed metrics do not measure this. 
Maybe this could simply be solved by adding some final epochs which just sample randomly again? 
2. A naive approach to creating large batches is by only updating the network every N batches, and summing the gradients of the N batches. I would like to see this option discussed and compared to. I can see how the BN is different from a large batch but it would still be interesting to see. 
3. I think the examination of sampling factors should be explained in more detail. The importance of the sampling factor of the prior is harder to understand (and might be necessary to counter the effect I discuss in 1?) Is it not expected that both these sampling factors should be equal? 


Minor points (do not need to be addressed in rebuttal):
-Since FID several other GAN evaluation metrics have been proposed. I think the authors could also consider ‘Assessing Generative Models via Precision and Recall’ or ‘Improved precision and recall metric for assessing generative models’ for a more complete insight insight 
-I am not convinced this GAN needs a name (small-GAN), especially since it can be applied to other GAN architectures. 
-The claim in the abstract for state of the art in anomaly detection should be removed (no extensive study nor comparison is performed)
-I prefer to see the venue of publication when possible (for example Sener & Savarese is ICLR 2018)

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper addresses the challenging problem of how to speed up the training of GANs without using large mini-batch sizes and causing significant performance drop. To achieve this, the authors propose to use the method of core-sets, mainly inspired by recent use of core-set selection in active learning. The proposed method allows us to generate effectively large mini-batches though actually small during the training process, or more concretely, drawing a large batch of samples from the prior and then compress that batch using core-set selection. To address the curse of dimensionality issue for high-dimensional data like images, the authors suggest using a low-dimensional embedding based on Inception activations of each training image. Regarding the experimental evaluation, it is clearly shown that the proposed core-set selection greatly improves GAN training in terms of timing and memory usage, and allows significantly reducing mode collapse on a synthetic dataset. As a by-product, it is successfully applied to anomaly detection and achieves state-of-the-art results.

Strengths:
The paper is generally well written and clearly presented.  As mentioned in the text, the use of core-sets is not novel in machine learning, but unfortunately not yet sufficiently explored in deep learning, and there are still few useful tools available in the literature. I believe this work will have a positive impact on the community and especially help establishing more efficient methods for training GANs.

Weaknesses:
- Experimental results are indeed very promising, however, GAN implementation details and hyperparameters used for training, such as optimizer and learning rate, do not seem to be mentioned in the text. I think this would be helpful for readers to better understand how this all works.
- There does not seem to be any discussion on the convergence and stability of GAN training, which should be clarified in the experimental section.
- On page 3, in Sect. 3.2,  I find “random low dimensional projections of the Inception Embeddings” is not clear, more technical details should be provided.
review_title: Official Blind Review #3

Meta-review: The paper proposes to use greedy core set sampling to improve GAN training with large batches. Although the problem is clear and the solution works, reviewers have raised several concerns. One concern is that the technical novelty is limited; another (in the first version) that even a simpler version of gradient accumulation can solve the  main task (rather that computing core-sets). In the end, some discussion was done, with quite a few additions and experiments done by the authors. The final concern that seemingly was not addressed: the gradient accumulation seems to give the same numbers as large batches, thus you can 'mimic' large batch sizes with smaller ones and gradient accumulation, making the main claim of the paper questionable. The achievement of SOTA is good, but it is not clear wether it is due to the proposed technique, or rather smart tuning of a larger set of hyperparameters. Thus, I would agree with the concern of Reviewer1.
Decision: Reject