Adaptive Data Augmentation with Deep Parallel Generative Models
Authors: Boli Fang
Miao Jiang
Abhirag Nagpure
Jerry Shen
Venue:  (2020)

Abstract: Data augmentation(DA) is a useful technique to enlarge the size of the training set and prevent overfitting for different machine learning tasks when training data is scarce. However, current data augmentation techniques rely heavily on human design and domain knowledge, and existing automated approaches are yet to fully exploit the latent features in the training dataset. In this paper we propose an adaptive DA strategy based on generative models, where the training set adaptively enriches itself with sample images automatically constructed from deep generative models trained in parallel. We demonstrate by experiments that our data augmentation strategy, with little model-specific considerations, can be easily adapted to cross-domain deep learning/machine learning tasks such as image classification and image inpainting, while significantly improving model performance in both tasks. 

Under review as a conference paper at ICLR 2020
ADAPTIVE DATA AUGMENTATION WITH DEEP PARAL- LEL GENERATIVE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Data augmentation((AA) useful technique to enlarge the size of the training set and prevent overfitting for different machine learning tasks when training data is scarce. However, current data augmentation techniques rely heavily on human design and domain knowledge, and existing automated approaches are yet to fully exploit the latent features in the training dataset. In this paper we propose an adap- tive DA strategy based on generative models, where the training set adaptively en- riches itself with sample images automatically constructed from deep generative models trained in parallel. We demonstrate by experiments that our data augmen- tation strategy, with little model-specific considerations, can be easily adapted to cross-domain deep learning/machine learning tasks such as image classification and image inpainting, while significantly improving model performance in both tasks.
INTRODUCTION
Deep learning and machine learning models produce highly successful results when given sufficient training data. However, when training data is scarce, overfitting will occur and the resulting model will be generalized poorly. It is also unavoidable that some data very hard to retrieve. Data aug- mentation(DA) ameliorates such issues by enlarging the original data set and making more effective use of the information in existing data. Much prior work has centered on data augmentation strate- gies based on human design, including heuristic data augmentation strategies such as crop, mirror, rotation and distortion Krizhevsky et al. (2012); (Simard al. (200)), interpolating through labeled data points in feature spaces De Vries & Taylor (201)), and adversarial data augmentation strategies based on Teo et al. 20088; Fawzi et al. (2016). These methods have greatly aided many deep learn- ing tasks across several domains such classification Krizhevsky et al. (2012). image segmentation Yang et al. (017) and image reconstruction/inpainnng Alvarez-Gila et al. (2017).
Despite their success, these DA methods generally require Nomain-speciffe expert knowledge, man- ual operations and extensive amount of tuning depending on actual contexts Ciresan et al.] Dosovit- skiy et al. 2016). In particular, the need to directly operate on existing data with domain knowledge prevents many previous data augmentation strategies from being applicable to more general settings. To circumvent the need for specific domain knowledge in data augmentation, more recent work An- toniou et al. (0117) utilizes generative adversarial Seworks(GANss Goodfellow et al. (2014) to produce images that better encode features in the latent space of training data. By alternatively optimizing the generator G' and the discriminator D in the GAN, the GAN is able to produce im- ages similar to the original data and effectively complement the training set. has been shown in experiments Antoniou et al. (2017) that GAN-based methods have indeed significantly boosted the performance of classifiers under limited data through automatic augmentation, but applications into other tasks yet to be explored. Furthermore, given the computational complexity and difficulty in GAN training, a natural way to promote scalability is to consider parallelism Intrator et al. (2018);
Durugkar et al. (2016).
In view of these considerations, we propose in this paper a new adaptive DA strategy based on deep generative models. Under such a framework, the original training set adaptively enriches itself with sample images automatically constructed from Generative Adversarial Networks (GANs) trained in parallel. Our contributions in this paper be summarized as follows:

---

Under review as a conference paper at ICLR 2020
We propose a general adaptive black-box data augmentation strategy diversify enhance training data, with no task-specific considerations.
We also include in our model a novel K-fold parallel framework, which helps make the most use of the existing data.
Experiments over various datasets and tasks demonstrate the effectiveness of our method in different contexts and tasks.
RELATED WORK
Augmenting data from limited amounts of training data has been a persistent challenge within the machine learning community. Here we summarize the related work in different categories.
Data AugmentationnDA) Previous work data augmentation can be classified into several groups. Traditional Heuristic DA strategies, such as crop, mirror, rotation and distortion Krizhevsky et al. 2012 Simmard et al. (2003), have found their way many deep classification tasks, but these meth- ods generally require domain-specific expert knowledge, manual operations and extensive amount of tuning depending on actual contexts Ciresan al., Dosovitskiy et al. (2016). Altemnttve DA meth- ods consider interpolation through labeled data points in feature spaces De Vries & Taylor (2017), but their dependence on class labels makes them inapplicable for tasks with weak or no supervision. Adversarial Data Augmentation strategies Teo al. (2008); et al. (2016) choose from select number of transformation operations to maximize the loss function of the end classification model involved in the task. While good motivations for methods, these methods make strong assump- tions over the types of augmentation and are difficult to generalize. Ratner et al. (2017); Cubuk et al. (2188) transform the problem of choosing data augmentation strategies into reinforcement learning policy search problems, but the choice of augmentation methods are still limited and the reinforcement learning algorithms have non-trivial computation overhead in addition the main task.
ML problems with limited data For classfication with limited samples, Salamon & Bello (2017) proposed a convolutional neural network (CNN) to classify environmental sounds with limited sam- ples. Other algorithms have been proposed in Frir-Adarr et al. (2018) ; Zhu et al. (2017). yet many of them have assumptiins/connsraintt that hurts their capacity for generalization. For unsuper- vised learning models, recent research on sample complexity reduction in GAN training seeks to reparametrize the input noise using variational inference Gurumurthy et al. (2017): Nowozin et al. (2016), but this method has severe mathematical limitation that prevents further generalization. Wang et al.) (2018) adopts transfer learning techniques train new GAN for limited data from a pre-trained GAN network. While effective, this approach requires a pre-trained network in the first place and doesn't 't apply to the cases when data is scarce.
ddaptive and Distributed Learning In recent years, adaptive learning has achieved good results in many training tasks. Yu et al.) (2018), Clavera et al. (2018) have proposed adaptive learning in reinforcement learning and meta-learning, and the strategies described within motivate our methods in this paper. On the side of deep learning, distributed multi-discriminator models Intrator et al. 20188;; Durugkar et al. (2016) also enhance the performances of generative algorithms.yet these models require large datasets to train and have great computational complexity. Moreover, these models are trained on fixed given datasets, meaning they are still susceptible to the inherent biases of the training data.
Deep learning with Limited Data Salamon & Bello 2017) proposed an algorithm to classify en- vironmental sounds with limited samples, through implementing Convolutional Neural Network with augmented samples. Training GANs on limited amounts of training data has been persistent challenge within deep learning community. Previous efforts to address the issue of labeled data scarcity generally fall into two groups: optimizing the structures of GANs to allow for better feature representation of data, and augmenting the training data through techniques.
Along the first line of research, prior research optimized the GAN in Radford et al. (2015) by con- sidering stronger mathematical objectives for more powerful latent space representation in general Nowozin et al. (2016); Arjovsky et al.) (2017) 7 Gulrajani al. (2017;; Mrouch & Sercu (2017). In addition, recent research on GANs Gurumurthy et al. (2017); Nowozin et al. (2016) reparametrized the input noise using variational inference Kingma & Welling (2014) by assuming that the latent
2

---

Under review as a conference paper at ICLR 2020
space could be modeled by tractable prior, but noise reparametrization has severe mathematical limitation that prevents applicability to more general models.
PAGDA: PARALLEL ADAPTIVE GENERATIVE DATA AUGMENTATION
In this section we deseribe the details of Parallel Adaptive Generative Data Ingmentation(PAGDD. Our method consists of three interrelated components: generative data augmentation, parallel image generation with fold division, and adaptive weight adjustment.
3.1 GENERATIVE DATA AUGMENTATION
To ensure that make full use of the information contained in the existing images, the first part of our method involves generative data augmentation, which constructs varied images given the training set by repeatedly generating samples from and adding samples to the training set using generative adversarial net.
We start off with limited training set, and consecutively run the generative adversarial net using the set. After running fixed number of regular training epochs, we proceed to the augmentation epoch where the augmentation is conducted. During the augmentation epoch, we extract a number of sample images from the generator G' using standard procedures of sample image generation as described in Radford et al. (2015); (ulraajjn et al. (2017). For this batch of samples, we calculate the Inception Score(IS) as defined by Salimans et al. (2016) to measure the authenticity of the images generated, which we denote as w and is described as follows:
w = (Dxxp((((KK(((//))))
Here the Inception Score (IS) measures the entropy of generated images, with higher scores indi- cating greater diversity. It thus provides metric that evaluates the power of generator to produce realistic images y from the distribution of original real images x: the higher the value of w, the more power the corresponding generator G. This batch of images are then added back into the original training set for subsequent augmentation epochs. We alternate running regular training epochs and the augmentation epoch for fixed number of times until convergence. Figure 1 is flow-chart of our procedure.
Notice that our procedure is agnostic to the specific architecture of generative adversarial net used to augment the training data. Since GANs capture the information the latent feature space of the images and translate such information into generated images, our method has the capacity to reveal the potential features that are possibly not visually evident in the original training images. Moreover, compared with many other data augmentation strategies which require one to pre-define the operations to be carried on the images, our method automatically enriches the training set and does not require human intervention.
3.2 PARALLEL FOLD DIVISION
The second part of method consists of a parallel data generation strategy, inspired by K-fold cross validation in machine learning Bishop (2006). Dividing the training data into K folds at the beginning, we run in parallel independent generators [G, Each generator G, is trained on one of data groups, and each data group consists of K folds of the training set, except for the i-th fold. After images are generated in each generator G, in the training epochs, the sample images produced by each generator during the augmentation epoch are fed back into the respective training data groups. To allow for maximal usage of each generated image, we insert the images in way such that the images generated by one generator G; are sent to the training data groups corresponding to all other - 1 generators except for that corresponding to G, This is to insure that the different generators in parallel have access to many varied data pieces possible in subsequent steps of training, so as to prevent overfitting and bolster the robustness of our strategy. Figure demonstrates our algorithm.
3

---

Under review as a conference paper at ICLR 2020
Real Samples
Training
K-fold
Yes, put in images by the weights of Gi
Gi
No
Di
Is Correct?
augmn epoch?
Discriminators
Generators
Fine Tune Training
Figure Generative Data Augmentation
Data Group 1
Generator
Image Set
Send all the Images
Data Group
K-Fold Images
Image Set
Image set
all folds except the i-th
Data Group k
Generator
Image Set
Data Group: all folds except
Figure 2: Parallel Image Generation
3.3 ADAPTIVE GENERATOR WEIGHTING
Furthermore, determine which generators the most effective in generating authentic images, we introduce adaptive generator weighting at each augmentation epoch. At the initial stage, all the generators treated equally. Before the batch of sample images generated by generator G,

---

Under review as a conference paper at ICLR 2020
Ni= # images from G
Image Set
Weighting
W P W
Wi=S((Image Set i)
NI=NP
Inception Score (IS)
N= total # of images for augmn
Figure 3: Adaptive Generator Weighting
are sent the data group corresponding other generators, collect the inception scores [wi] computed in section 3.1. Since higher inception scores imply better performance of the generator, we define the generator weight p; of generator G; as
exp (ws)
p: Σラ= exp (wr)²
and use this weight to determine how many images should be sampled from generator G; to be sent to other data groups for subsequent training in the very next augmentation epoch. When the total number of samples to be collected from generators are fixed, this method enables generators with better realistic image generation power to contribute more to the future training data groups. More realistic training sets thus augmented, in turn, exert more positive influence on the images to be generated. Algorithm illustrates the whole 3-step process, which denote PAGDA(Parallel Adaptive Generative Data Augmentation).
Algorithm Parallel Adaptive Generative Data Augmentation
Input: A limited dataset 2 {(
y:)
Output: An augmented dataset D = {(1 1:)
1: procedure PAGDA(D)
2: Divide into K equal size folds {D,)
Parallel Fold Division 3: Initialize a Generative Adversarial Network G; on each D; and all GAN weights = 1;
4: Fix size of augmentation batch no:
5:
while Training do
6: Train G; on D; for certain number of epochs;
7:
Extract S; = nop: images from discriminator of G;; Generative Data Augmentation
8:
W, = Inception Score of Imageset(i);
9:
pi < exp (105)
exp (w:)
Adaptive Generator Weighting
10:
D: D; U (k):
return D' = DUK D.)
Note that all three strategies introduced go hand in hand, with no need for model specific consid- erations. As demonstrated by experiments Section 4, training different GANs in parallel from different folds of data substantially boosts the quality of the training set and that the generated images.
EXPERIMENTS
4.1
MULTITASK EXPERIMENTAL SETTINGS
To illustrate the effectiveness of our adaptive model, applied model for multiple machine learning tasks, we have applied our data augmentation method to two tasks: image classification and image inpainting.
For image classification we constructed our dataset from Imagenet and Cifar- 10 by randomly draw- ing 5000 images from each dataset respectively and applied adaptive generative models on these

---

Under review as a conference paper at ICLR 2020
reduced datasets. The augmented datasets are then used to train an AlexNet CNN classifier, and the classification results are compared with the results obtained from an AlexNet trained on the corresponding original unaugmented datasets or augmented datasets with different strategies.
For image inpainting, we constructed our reduced sub-datasets from open dataset named Places. The whole Places dataset consists of different sub datasets for images with different scenarios (e.g. ocean sub datsets contains images with all beautiful scenes from the sea). We randomly pickup images of 3 scenarios (ocean, orchard, piers) to obtain reduced sub dataset. Each sub dataset contains 5000 images.
To ensure the parallelism of the experiments, trained our model in multi-gpu environment make simultaneously training. Under such setting, all the data groups are trained at the same time, and each GAN model corresponding experiments are conducted on 2 servers with 8 Tesla-V GPU (32GB RAM, 7.8 TeraFLOPS) and Intel Xeon() Processor E5 (2.00 GHz).
each data group trained in separate thread. All of our
4.2 EVALUATION ON CLASSIFICATION FROM IMAGE DATASETS
For the task of image classifidation, have conducted extensive experiments different image datasets to illustrate the effect of our data augmentation strategy. For our experiments on classifi- cation, we first augment the reduced Cifar-10, reduced Imagenet datasets, and then train the CNN classifier with the augmented dataset. classifier accuracies with and without augmentation are listed in Table below with the experimental setting described in the next paragraph.
Table 1: Classification Accuracy with different augment strategy
Datasets
w/o augment
image flip image crop
with augment
Reduced Cifar-10
74.7
75.3
75.6
81.3
Reduced Imagenet
78.7
78.9
78.8
88.1
In our experiments, we augment the training set with 1000 noised images every 100 training epoches, and repeat the procedure times. By comparison, the unaugmented GAN is run the same initial training data, with the number of epochs the same as the product of 100 and augmentation times. For optimal choice of fold number K, we varied K from using different values of K, and choose 5 to produce the yield the best classification results as listed table 1. Our observation is largely consistent with the heuristics of K-fold cross validation in machine learning, where K-values between 4-5 are often the optimal Bishop (006).
10, examined the level of accuracy
88
86
84
82
80
reduced cifar-10 reduced imagenet
78
Figure 4: Accuracy with different number of folders
4.3 EVALUATION ON NEW IMAGE GENERATION
Along with the task of classification, we considered the task of new image generation from given training images. We have applied similar settings as in image classification, and the goal to

---

Under review as a conference paper at ICLR 2020
produce images with higher resolution and more authentic semantics with generative adversarial networks. In our setting, we first train the sfute-of-the-aee WGAN to produce images from an unaug- mented dataset, and repeat the procedure
image data set that has been augmented.
Figures 5,6,7,8,9 are some sample images that our method has produced compared with those pro- duced without data augmentation. We observe that GANs with parallel recurrent image augmenta- tion produce semantically coherent and visually diverse images earlier than the unaugmented GANs, while able to avoid fluctuations seen in unaugmented GANs during training.
Figure 5: Augmented GAN
Figure 6: Unaugmented GAN at Epoch Jol.RRducdd-IFA.
Figure 7: Unaugmented GAN at Epoch 342, Reduced-CIFAR
Epoch 271, [.ReducedCIIAAR
Figure Unaugmented GAN, Figure 9: CelebA
Augmented GAN,
CelebA
4.4 EVALUATION ON INPAINTING
For the task of inpainting, augment the reduced Places dataset constructed in the experiment. As we mentioned 4.1, we apply 3 subset named with Ocean, Orchard and Piers, each subset contains 5000 images. Without loss of generality, inpainting from the augmented dataset. We then select testing images that are not selected in the training set, and add to them gray masks covering the center part of these images. We then applied our trained WGAN-GP to generate patches that cover the masked portion of the inpainting image.
train WGAN-GP model generative model for
For qualitative analysis, figure 10 lists couple of generated images with and without augmentation. The semantic coherence within visual comparisons demonstrate the effectiveness of our method.
For quantitative evaluation of quality for the images generated by our augmentation method as compared with the conventional methods such as image flipping and image cropping, we use the Inception Score(IS) [Salimans et al. (2016) described previously as well Frechet Inception Dis- tance(FII) Heusel et al. (2017) for metrics. IS measures the entropy of generated images, with higher scores indicating greater diversity. On the other hand, FID measures the distance between the generated data and real data with two respective means and variances. Thus, the larger the IS and the smaller the FID, the better the performances of the model. An alternative measure involving the exponent of inception score has been proposed in Gurumurthy et al. (2017); given the diversity and semantic complexity of images involved in our experiments, we will stick to the original formulation as proposed in Salimans al. (0016).
7

---

Under review as a conference paper at ICLR 2020
original image
masked image
with PAGANDA
w/o PAGANDA
Figure 10: Inpainting results on reduced datasets
The inception score(IS) and Frechet Inception Distance (FID) we used in all experi- ments are calculated by Python scripts available respectively at _core and htttps://githn
Table 2: IS/FID scores of on Places with different Data Augmentation strategies
Subsets
Noise
Image Flip
Image Crop
Our Strategy
Ocean 4.31/102.7
4.33/107.8
3.99/114.2
5.75/79.3
Orchard
4.39/105.3
4.28/106.4
3.97/113.2
5,83/82.4
Pier
4.34/107.1
4.30/109.8
4.00/109.3
5.79/90.2
Table 2 lists the results of data augmentation based on different subsets of Places dataset. We have conducted the set of experiments with varying numbers of parallel folders and have discovered that the value K=5 has helped achieve the best image quality. Notice that the images generated with our method are produced with the state-of-the-ard WGAN-GP as each parallel generative model. Here in addition to considerations K-fold Cross SulidationBishopp (2006), our choice of K-values are mo- tivated by distributed MD-GAN as described in Hardy et al. (2018). where K-values ranging from 3 to 10 are shown to achieve good performances. Table 2 list and FID of results experimented on different sub datasets with conventional data augmentation. Clearly, adaptive sample augmentation produces better images as measured quantitatively and some other simple augmentation strategies.
5 CONCLUSION AND FUTURE WORK
We have proposed this paper a parallel adaptive based data augmentation strategy, and demon- strate through experiments and analysis that strategy significantly improves the performances of existing models on a variety of tasks, outcompeting many traditional methods aiming towards similar goals. In addition, our paper shows that our adaptive data augmentation strategy effectively requires little task-specific considerations, since the proposed framework can easily be adapted to variety of generative models and ML paradigms. Our strategy is not only simple implement, but also demonstrates capability on multiple tasks in machine learning and computer vision, since it does not require specific information about the task being analyzed.
As a further step, we are investigating the relationship between our proposed approach and other established methods. One possible pathway, for instance, lies in the DA strategy based on reinforce- ment learning as described in Cubuk et al.) (2018) that gives more control to image generation via reward designation. We believe such motivation would inspire better design of the adaptive gener- ator weight functions in our paper. We also hope to apply our idea to other generative models such as VAE in Kingma & Welling (2014), and further optimize our strategy using recent theoretical ad- vances, and wish to investigate the scenarios where the tasks involved are interrelated. Application wise, we are aiming to extend our parallel GAN model to medical image synthesis/generratoon since the amount of data available for machine learning is often highly limited. Another goal is to extend the scope application from CV/DL to more general data mining, where alternative evaluation metrics can be used for designing the generator weight function in model.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Aitor Alvarez-Gila, Joost Van De Weijer, and Estibaliz Garrote. Adversarial networks for spatial context-aware spectral image reconstruction from rgb. In Proceedings of the IEEE International Conference on Computer Vision, pp. 480-490, 2017.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arXi:: 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research. PMLR, 2017.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
DC Ciresan, U Meier, LM Gambardella, and J Schmidhuber. Deep big simple neural nets excel handwritten digit recognition. соrr 1st/1003.0358 (2010).
Ignasi Clavera, Anusha Nagabandi, Pieter Fearing, Ronald Sland Abbeel, Levine, and Chelsea Finn.
Learning adapt: Meta-learning for model-based control. CoRR, abs/1803. 1347, 2018. URL
http:///rxiv> wwtp://gg/bbbbbbbbgboee 11347
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. CoRR, abs//8055.00 005.09501, 2018.
Terrance DeVries and Graham W Taylor. Dataset augmentation in feature space. arXiv preprint
arXiv:1702.05538, 2017.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE transactions on pattern analysis and machine intelligence, 38(9):17341177,, 2016.
Ishan P. Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative multi-adversarial networks. ICLR, 2016.
Alhussein Fawzi, Horst Samulowitz, Deepak Turaga, and Pascal Frossard. Adaptive data augmenta- tion for image classification. In 2016 IEEE International Conference on Image Processing (ICIP), pp. 3688-3692. Ieee, 2016.
Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan. Synthetic data augmentation using gan for improved liver lesion classification. In 2018 IEEE 15th Interna- tional Symposium on Biomedical Imaging (ISBI 2018), pp. 189-293. IEEE, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems 30. 2017.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R. Venkatesh Babu. Deligan : Gen- erative adversarial networks for diverse and limited data. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Corentin Hardy, Erwan Le Merrer, and Bruno Sericola. MD-GAN: multi-discriminator generative adversarial networks for distributed datasets. CoRR, abs/181 1.03850,, 2018. URL http://
baxivv.or//a hbs//bbs//1111 03850
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems 30. 2017.
Yotam Intrator, Gilad Katz, and Asaf Shabtai. MDGAN: boosting anomaly detection using multi- discriminator generative adversarial networks.
2018.

---

Under review as a conference paper at ICLR 2020
Diederik P. Kingma and Max Welling. abs/1312.6114, 2014.
Auto-encoding variational bayes. In ICLR, volume
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems 25. 2012.
Youssef Mroueh and Tom Sercu. Fisher gan. Advances in Neural Information Processing Systems 30. 2017.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan:
Training generative neural sam- plers using variational divergence minimization. In Advances in Neural Information Processing Systems 29. 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/1511.06434 2015.
Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learn- ing to compose domain-specific transformations for data augmentation. Advances neural information processing systems, pp. 3236-3246, 2017.
Justin Salamon and Juan Pablo Bello. Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Processing Letters, 24(3):279- 223,229,,,, 2017.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems 29. 2016.
Patrice Y. Simard, David Steinkraus, and John C. Platt. Best practices for convolutional neural networks applied to visual document analysis. ICDAR, 2003.
Choon H Teo, Amir Globerson, Sam Roweis, and Alex J Smola. Convex learning with invariances. In Advances neural information processing systems, pp. 1489-1496, 2008
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost de Weijer, Abel Gonzalez-Gareia, and Bog- dan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 218-234, 2018.
Dong Yang, Daguang Xu, Kevin Zhou, Bogdan Georgescu, Mingqing Chen, Sasa Grbie, Dim- itris Metaxas, and Dorin Comaniciu. Automatic liver segmentation using an adversarial image- to-image network. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 507-515. Springer, 2017.
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain- meta-learning. CoRR, abs/1802.01557,, 2018. http:///: wwwpwwwwwwwwwoo.ooooooeoeeoeoeee ort absg///b//// 802. 01557
Xinyue Zhu, Yifan Liu, Zengchang Qin, and Jiahong Li. Data augmentation in emotion classification using generative adversarial networks. arXiv preprint arXiv 2017.
10


Reviews:
[{"review_id": "rJg7jWjCFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an method based on multiple GANs trained on different splits of the training data in order to generate additional samples to include to the training data. The number of added samples from each GAN depends on the quality of the generation based on their inception scores. This approach helps to improve the performance of a convolutional neural network on different tasks such as image classification, image generation and image inpainting. \n\nI rated this paper as weak reject because this paper is weak on several aspects\n- In related work many similar approaches are missing (see below).\n- In the methodology the actual formulation for the GAN is not presented.\n- The main novelty of the paper seems to be the use of multiple GAN on different splits of the data, which seems a bit limited.\n- The experimental evaluation is limited (see below).\n\nRelated work:\nSeveral works with very similar intentions have been overlooked:\n(\"A Bayesian Data Augmentation Approach for Learning Deep Models, Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, Ian Reid) uses GAN during training for generating samples for data augmentation.\n(\"Dada:  Deep adversarial data augmentation for extremely low data regime classification\", Xiaofeng Zhang,  Zhangyang Wang,  Dong Liu,  and Qing Ling) uses another GAN model for generating samples on low data regime for data augmentation\n(\"Adversarial Learning of General Transformations for Data Augmentation\", Saypraseuth Mounsaveng, David Vazquez, Ismail Ben Ayed, Marco Pedersoli), uses GAN for generating samples for data augmentation on reduced datasets.\n(Triple Generative Adversarial Nets, Chongxuan Li, Kun Xu, Jun Zhu, Bo Zhang) uses a gan model for generating samples for semi-supervised learning with few labelled samples.\n(Triangle Generative Adversarial NetworksZhe Gan∗, Liqun Chen∗, Weiyao Wang, Yunchen Pu, Yizhe Zhang,Hao Liu, Chunyuan Li, Lawrence Carin) in which again GAN is used to improve in semi-supervised settings.\n\nExperimental evaluation:\nIn the experimental evaluation the importance of the use of k-fold is shown only on Fig.4. Additionally, in table 1 the method is compared with weak baselines (no DA, flip, crop, but not the combination of flip and crop which is standard on CIFAR10) and it is not compared with any other approach (and there are several as shown in related work).\nFor generating images, no quantitative values are provided, just some generation examples.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgZ2HfPtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose a heuristic for making data augmentation more \"end to end\" rather than an ad hoc technique. The idea is to train multiple GANs on subsets of the data, and once in a while use these to generate new data aid the training of the final model.\n\nI found that the justification for the proposed heuristic was largely missing. Why is the proposed method good?Is it optimal in any sense? Does it correspond to a specific model assumption? Is there an intuition why it's a good idea?\n\nIntuitively, I would expect that it is hard to train a GAN (or another generative model) than it is to train a classifier, so why is it s a good idea to augment the dataset using GANs?\n\nOne of the key tricks of the paper is to split the data in multiple folds and train one GAN per fold. I would expect this to be fairly unstable is building generative models tend to be very \"data demanding\" (Wuch that working with only 1/5 of the total data could be problematic).\n\nIn recent years there have been quite some work on learning of data augmentation which isn't cited in the paper. I'd recommend looking at\n\n  \"Dreaming More Data: Class-dependent Distributions over Diffeomorphisms for Learned Data Augmentation\", Hauberg et al., AISTATS 2016.\n  \"A Bayesian Data Augmentation Approach for Learning Deep Models\", Tran et al. NeurIPS 2017\n\nand the references therein to get a better coverage of previous work.\n\nI generally find it difficult to assess the presented experiments. Since the approach is deemed general, why isn't it applied to general tasks with an established ground truth? E.g. why isn't the technique applied to classification or regression? That would make it much easier to assess if the approach does something sensible. I am also missing elementary baselines, e.g. the usual hand-crafted data augmentation should also appear in the baselines.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "H1gwNWBaYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new approach to data augmentation using GANs, called Parallel Adaptive Generative Data Augmentation (PAGDA), in which the following procedure is used:\n- (1) First, GANs are used to generate different batches of data points based on the starting training set, and then these batches are added back to the training set\n- (2) This is done using an ensemble of K GANs where each GAN is trained on K-1/K portion of the training set, and the resulting samples are sent to the training sets of the other K-1 GANs\n- (3) The generators are sampled from proportional to their (exponentially normalized) inception scores\n\nOverall, this paper unfortunately should be rejected due primarily to inadequate experimental evaluation.  Specifically, the paper proposes a new method (and a very complex and seemingly arbitrary one at that), in a space where there are many pre-existing approaches, and then neither (a) compares to any pre-existing baselines or approaches (other than the most trivial of manual data augmentation strategies), nor (b) experimentally justifies or explores any of the design choices in the extremely convoluted method design.  Therefore, there is no real way for the reader to render any conclusion about the merit of the proposed approach: how does it compare to many (seemingly similar) existing approaches?  Why were the various design decisions made, which ones are more or less important?  How well does it work?  No real way to answer these given the extremely minimal experimental section.\n\nAdditionally, there is relatively little detail in the main description of the method (e.g. how is the GAN trained on labeled data?  How exactly is the sampling done?).  Also, worth noting that \"Multi-task\" is used incorrectly in the experiments section.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a data augmentation method based on Generative Adversarial Networks by training several GANs on subsets of the data which are then used to synthesise new training examples in proportion to their estimated quality as measured by the Inception Score. The reviewers have raised several critical issues with the work, including motivation (it can be harder to train a generative model than a discriminative one), novelty, complexity of the proposed method, and lack of comparison to existing methods. Perhaps the most important one is the inadequate empirical evaluation. The authors didn’t address any of the raised concerns in the rebuttal. I will hence recommend the rejection of this paper.
Decision: Reject