Adaptive Data Augmentation with Deep Parallel Generative Models

Data augmentation ( DA ) is a useful technique to enlarge the size of the training set and prevent overfitting for different machine learning tasks when training data is scarce . However , current data augmentation techniques rely heavily on human design and domain knowledge , and existing automated approaches are yet to fully exploit the latent features in the training dataset . In this paper we propose an adaptive DA strategy based on generative models , where the training set adaptively enriches itself with sample images automatically constructed from deep generative models trained in parallel . We demonstrate by experiments that our data augmentation strategy , with little model-specific considerations , can be easily adapted to cross-domain deep learning/machine learning tasks such as image classification and image inpainting , while significantly improving model performance in both tasks . 1 INTRODUCTION . Deep learning and machine learning models produce highly successful results when given sufficient training data . However , when training data is scarce , overfitting will occur and the resulting model will be generalized poorly . It is also unavoidable that some data is very hard to retrieve . Data augmentation ( DA ) ameliorates such issues by enlarging the original data set and making more effective use of the information in existing data . Much prior work has centered on data augmentation strategies based on human design , including heuristic data augmentation strategies such as crop , mirror , rotation and distortion Krizhevsky et al . ( 2012 ) ; Simard et al . ( 2003 ) , interpolating through labeled data points in feature spaces DeVries & Taylor ( 2017 ) , and adversarial data augmentation strategies based on Teo et al . ( 2008 ) ; Fawzi et al . ( 2016 ) . These methods have greatly aided many deep learning tasks across several domains such as classification Krizhevsky et al . ( 2012 ) , image segmentation Yang et al . ( 2017 ) and image reconstruction/inpainting Alvarez-Gila et al . ( 2017 ) . Despite their success , these DA methods generally require domain-specific expert knowledge , manual operations and extensive amount of tuning depending on actual contexts Ciresan et al . ; Dosovitskiy et al . ( 2016 ) . In particular , the need to directly operate on existing data with domain knowledge prevents many previous data augmentation strategies from being applicable to more general settings . To circumvent the need for specific domain knowledge in data augmentation , more recent work Antoniou et al . ( 2017 ) utilizes generative adversarial networks ( GANs ) Goodfellow et al . ( 2014 ) to produce images that better encode features in the latent space of training data . By alternatively optimizing the generator G and the discriminator D in the GAN , the GAN is able to produce images similar to the original data and effectively complement the training set . It has been shown in experiments Antoniou et al . ( 2017 ) that GAN-based methods have indeed significantly boosted the performance of classifiers under limited data through automatic augmentation , but applications into other tasks are yet to be explored . Furthermore , given the computational complexity and difficulty in GAN training , a natural way to promote scalability is to consider parallelism Intrator et al . ( 2018 ) ; Durugkar et al . ( 2016 ) . In view of these considerations , we propose in this paper a new adaptive DA strategy based on deep generative models . Under such a framework , the original training set adaptively enriches itself with sample images automatically constructed from Generative Adversarial Networks ( GANs ) trained in parallel . Our contributions in this paper can be summarized as follows : • We propose a general adaptive black-box data augmentation strategy to diversify enhance training data , with no task-specific considerations . • We also include in our model a novel K-fold parallel framework , which helps make the most use of the existing data . • Experiments over various datasets and tasks demonstrate the effectiveness of our method in different contexts and tasks . 2 RELATED WORK . Augmenting data from limited amounts of training data has been a persistent challenge within the machine learning community . Here we summarize the related work in different categories . Data Augmentation ( DA ) Previous work on data augmentation can be classified into several groups . Traditional Heuristic DA strategies , such as crop , mirror , rotation and distortion Krizhevsky et al . ( 2012 ) ; Simard et al . ( 2003 ) , have found their way in many deep classification tasks , but these methods generally require domain-specific expert knowledge , manual operations and extensive amount of tuning depending on actual contexts Ciresan et al . ; Dosovitskiy et al . ( 2016 ) . Alternative DA methods consider interpolation through labeled data points in feature spaces DeVries & Taylor ( 2017 ) , but their dependence on class labels makes them inapplicable for tasks with weak or no supervision . Adversarial Data Augmentation strategies Teo et al . ( 2008 ) ; Fawzi et al . ( 2016 ) choose from a select number of transformation operations to maximize the loss function of the end classification model involved in the task . While good motivations for our methods , these methods make strong assumptions over the types of augmentation and are difficult to generalize . Ratner et al . ( 2017 ) ; Cubuk et al . ( 2018 ) transform the problem of choosing data augmentation strategies into a reinforcement learning policy search problems , but the choice of augmentation methods are still limited and the reinforcement learning algorithms have non-trivial computation overhead in addition to the main task . ML problems with limited data For classfication with limited samples , Salamon & Bello ( 2017 ) proposed a convolutional neural network ( CNN ) to classify environmental sounds with limited samples . Other algorithms have been proposed in Frid-Adar et al . ( 2018 ) ; Zhu et al . ( 2017 ) , yet many of them have assumptions/constraints that hurts their capacity for generalization . For unsupervised learning models , recent research on sample complexity reduction in GAN training seeks to reparametrize the input noise using variational inference Gurumurthy et al . ( 2017 ) ; Nowozin et al . ( 2016 ) , but this method has severe mathematical limitation that prevents further generalization . Wang et al . ( 2018 ) adopts transfer learning techniques to train a new GAN for limited data from a pre-trained GAN network . While effective , this approach requires a pre-trained network in the first place and doesn ’ t apply to the cases when data is scarce . Adaptive and Distributed Learning In recent years , adaptive learning has achieved good results in many training tasks . Yu et al . ( 2018 ) , Clavera et al . ( 2018 ) have proposed adaptive learning in reinforcement learning and meta-learning , and the strategies described within motivate our methods in this paper . On the side of deep learning , distributed multi-discriminator models Intrator et al . ( 2018 ) ; Durugkar et al . ( 2016 ) also enhance the performances of generative algorithms , yet these models require large datasets to train and have great computational complexity . Moreover , these models are trained on fixed given datasets , meaning they are still susceptible to the inherent biases of the training data . Deep learning with Limited Data Salamon & Bello ( 2017 ) proposed an algorithm to classify environmental sounds with limited samples , through implementing a Convolutional Neural Network with augmented samples . Training GANs on limited amounts of training data has been a persistent challenge within deep learning community . Previous efforts to address the issue of labeled data scarcity generally fall into two groups : optimizing the structures of GANs to allow for better feature representation of data , and augmenting the training data through techniques . Along the first line of research , prior research optimized the GAN in Radford et al . ( 2015 ) by considering stronger mathematical objectives for more powerful latent space representation in general Nowozin et al . ( 2016 ) ; Arjovsky et al . ( 2017 ) ; Gulrajani et al . ( 2017 ) ; Mroueh & Sercu ( 2017 ) . In addition , recent research on GANs Gurumurthy et al . ( 2017 ) ; Nowozin et al . ( 2016 ) reparametrized the input noise using variational inference Kingma & Welling ( 2014 ) by assuming that the latent space could be modeled by a tractable prior , but noise reparametrization has severe mathematical limitation that prevents applicability to more general models . 3 PAGDA : PARALLEL ADAPTIVE GENERATIVE DATA AUGMENTATION . In this section we describe the details of Parallel Adaptive Generative Data Augmentation ( PAGDA ) . Our method consists of three interrelated components : generative data augmentation , parallel image generation with fold division , and adaptive weight adjustment . 3.1 GENERATIVE DATA AUGMENTATION . To ensure that make full use of the information contained in the existing images , the first part of our method involves generative data augmentation , which constructs varied images given the training set by repeatedly generating samples from and adding samples to the training set using a generative adversarial net . We start off with a limited training set , and consecutively run the generative adversarial net using the set . After running a fixed number t of regular training epochs , we proceed to the augmentation epoch where the augmentation is conducted . During the augmentation epoch , we extract a number of sample images from the generator G using standard procedures of sample image generation as described in Radford et al . ( 2015 ) ; Gulrajani et al . ( 2017 ) . For this batch of samples , we calculate the Inception Score ( IS ) as defined by Salimans et al . ( 2016 ) to measure the authenticity of the images generated , which we denote as w and is described as follows : w = exp ( Ex ( DKL ( p ( y|x ) ‖p ( y ) ) ) , Here the Inception Score ( IS ) measures the entropy of generated images , with higher scores indicating greater diversity . It thus provides a metric that evaluates the power of generator to produce realistic images y from the distribution of original real images x : the higher the value of w , the more power the corresponding generator G. This batch of images are then added back into the original training set for subsequent augmentation epochs . We alternate running t regular training epochs and the augmentation epoch for a fixed number of times or until convergence . Figure 1 is a flow-chart of our procedure . Notice that our procedure is agnostic to the specific architecture of generative adversarial net used to augment the training data . Since GANs capture the information in the latent feature space of the images and translate such information into generated images , our method has the capacity to reveal the potential features that are possibly not visually evident in the original training images . Moreover , compared with many other data augmentation strategies which require one to pre-define the operations to be carried on the images , our method automatically enriches the training set and does not require human intervention . 3.2 PARALLEL FOLD DIVISION . The second part of our method consists of a parallel data generation strategy , inspired by K-fold cross validation in machine learning Bishop ( 2006 ) . Dividing the training data into K folds at the beginning , we run in parallel K independent generators { Gi } Ki=1 . Each generator Gi is trained on one of data groups , and each data group i consists of K − 1 folds of the training set , except for the i-th fold . After images are generated in each generator Gi in the training epochs , the sample images produced by each generator during the augmentation epoch are fed back into the respective training data groups . To allow for maximal usage of each generated image , we insert the images in a way such that the images generated by one generator Gi are sent to the training data groups corresponding to all other K − 1 generators except for that corresponding to Gi . This is to insure that the different generators in parallel have access to as many varied data pieces as possible in subsequent steps of training , so as to prevent overfitting and bolster the robustness of our strategy . Figure 2 demonstrates our algorithm .

Summary: This paper proposes an method based on multiple GANs trained on different splits of the training data in order to generate additional samples to include to the training data. The number of added samples from each GAN depends on the quality of the generation based on their inception scores. This approach helps to improve the performance of a convolutional neural network on different tasks such as image classification, image generation and image inpainting.