NOSE Augment: Fast and Effective Data Augmentation Without Searching

1 INTRODUCTION . Data is an essential and dominant factor for learning AI models , especially in deep learning era where deep neural networks normally require large data volume for training . Data augmentation techniques artificially create new samples to increase the diversity of training data and in turn the generalization of AI models . For example , different image transformation operations , such as rotation , flip , shear etc. , have been used to generate variations on original image samples in image classification and other computer vision tasks . More intricate augmentation operations have also been implemented , such as Cutout ( Devries & Taylor , 2017 ) , Mixup ( Zhang et al. , 2018 ) , Cutmix ( Yun et al. , 2019 ) , Sample Pairing ( Inoue , 2018 ) , and so on . How to formulate effective augmentation strategies with these basic augmentation methods becomes the crucial factor to the success of data augmentation . Recent works ( Cubuk et al. , 2019 ; Lim et al. , 2019 ; Ho et al. , 2019 ) introduced automated searching or optimization techniques in augmentation policy search . The common assumption of these methods is : a selected subset of better-fit augmentation policies will produce more relevant augmented data which will in turn result in a better trained model . Here the augmentation policy is defined by an ordered sequence of augmentation operations , such as image transformations , parameterized with probability and magnitude . Though these methods achieved state-of-the-art accuracies on image classification tasks , they lead to high computational cost in general , due to large search space and extra training steps . More importantly , it is worth exploring whether it is really necessary to find the best-fit subset of policies with specific parameter values of probability and magnitude . RandAugment ( Cubuk et al. , 2020 ) has started to simplify the parameters and scale down the search space defined by AutoAugment Cubuk et al . ( 2019 ) , but their method still relied on grid search for iterative optimization of the simplified parameters . Our method aims to fully avoid policy search and cost , meanwhile to maintain or improve model performance in terms of both accuracy and training efficiency . Our work showed that by applying simple stochastic augmentation policies with the same sampling space and other settings of training , we could obtain equal or very close performance with search-based augmentation methods . Another advantage of stochastic policies is that adding more operations in the pool does not bring additional cost ; while in search-base methods , more operations in the pool causes exponential increase of the search space . Therefore , the second part of our method is to add more operations to the pool to bring more data diversity . In practice , we introduced a new category of operations such as mixup and cutmix into the operation pool . Furthermore , we tackled automated augmentation from overall data usage point of view , in contrast to data creation point of view accentuated by policy-search based methods . Inspired by the idea of Curriculum Learning ( CL ) ( Bengio et al. , 2009 ) , which presents training samples in an increasing order of difficulties , our method defines various complexity levels of augmentation strategies and applies them with orders on phased training stages . To avoid the confounding overfitting problem of original Curriculum Learning in practice , our method applies the inverted order of Curriculum Learning , which presents the hardest augmentation strategies from the beginning and gradually decreases the complexity levels . In general , our augmentation method replaces policy search with stochastic policy generation , upon which it introduces more operations for better diversity and phased augmentation strategy with decreasing complexities for a smooth learning convergence , and as an integral solution it achieves better results . Figure 1 describes our method and the difference compared to search-based methods . The main contributions of this paper can be summarized as follows : 1 . We present a no-search ( NOSE ) augmentation method as an alternative of computation-intensive search-based auto-augment methods . By jointly applying phased augmentation strategy and introducing more augmentation operations on top of a simple stochastic augmentation mechanism , NOSE augment achieves state-of-the-art ( SOTA ) accuracies on CIFAR 10 , CIFAR 100 ( Krizhevsky , 2009 ) and close-to-SOTA results on other benchmark datasets . Our ablation study demonstrates that all the components of our methods should be combined together to achieve the best performance . 2 . We demonstrate that a stochastic-based augmentation approach can obtain accuracies comparable to those of search-based methods while achieving overwhelming advantage on overall augmentation and training efficiency as the searching phase is completely avoided . 3 . Besides image classification , we also applied NOSE augment on face recognition and text detection ( OCR ) tasks , and obtained competitive or better results in comparison with search-based methods . This further proves the advantage and generality of NOSE augment . 2 RELATED WORKS . Our work is most closely related to RandAugment ( Cubuk et al. , 2020 ) . A key idea in RandAugment is to avoid a separate search phase on proxy datasets but chose instead to reduce the search space by decreasing the number of parameters . For instance , the probability for each operator is set to a constant uniform probability . The two sets of data augmentation parameters to be searched are N , the number of transformations for a given image and M , the discrete operator magnitude . These two parameters are regarded as hyperparameters and tuned using grid search . Our method is similar to RandAugment in terms of the operators used and the setting of equal probability for each operator . The most contrasting difference between RandAugment and our method is that RandAugment is still a search-based method with search cost while our method skips policy search completely and tackles the problem by introducing new augmentation operations and further new strategies built upon stochastic policies . Our work stands diametrically opposite to the AutoAugment series of methods ( Cubuk et al. , 2019 ; Lim et al. , 2019 ; Zhang et al. , 2020 ) , in which the best data augmentation policy is derived through a search . AutoAugment ( Cubuk et al. , 2019 ) uses a RNN controller to propose an augmentation strategy . Based on this proposed augmentation strategy , the model is trained and use the resulting validation accuracy to update the RNN controller . Although AutoAugment achieves good results on the standard benchmark datasets , the search cost is prohibitively huge , for instance , 5000 GPU ( NVIDIA Tesla P100 ) hours on the CIFAR-10 dataset and Pyramid-Net+ShakeDrop model . Fast AutoAugment ( Fast AA ) ( Lim et al. , 2019 ) ameliorates AutoAugment ’ s huge search cost with a three pronged approach . Smaller datasets are used for policy search . The idea of density matching is proposed to avoid having to re-train the model for assessing the validation accuracy for each proposed policy . The use of Bayesian Optimization based method ( Bergstra et al. , 2011 ) also helps to converge to an effective augmentation policy quickly . Adversarial Augment ( Adv AA ) ( Zhang et al. , 2020 ) reduces the huge computational cost of AutoAugment through the adversarial policy framework , which generates data samples that maximizes the training loss of the target network . PBA ( Ho et al. , 2019 ) introduces the idea of non-stationary policy schedules instead of the fixed augementation policy proposed in AutoAugment . The non-stationary policy schedules refers to how the policy evolves with the training epochs . Our main differences with aforementioned state-of-the-art methods lie in our stochastic policy described in section 3.1 and the multi-stage complexity driven augmentation policy outlined in section 3.3 . We use the same set of operators and settings as these related works . 3 METHOD . We describe the three key components of our proposed method in this section . In section 3.1 , we put forward a stochastic-based method as opposed to the search-based paradigm advocated by many of the state-of-the-art works described in section 2 . Due to the counter-intuitiveness ( but effective ) nature of this stochastic-based method , we first provide insights and motivations into stochastic augmentation policies in section 3.1 . In section 3.2 , we capitalize on our proposed stochastic-based method by proposing additional operators that will further enrich the data augmentation diversity , since there is no extra search cost for adding new operators with a no-search method . In 3.3 , we propose a multi-stage complexity driven policy that helps to resolve the tension between augmented data diversity and data distribution fidelity . Our ablation study demonstrates that the performance of our method is not a simple incremental benefit with each individual component , but rather , the three components need to work together to achieve a competitive performance with no policy search cost . 3.1 STOCHASTIC POLICY - SKIPPING POLICY SEARCH . The first component of our method removes policy searching completely and applies a stochastic augmentation policy with randomly selected operations and magnitudes . Unlike the search-based methods , once an operation is selected , it is used with 100 % probability . Our stochastic method follows the same policy definition as existing search-based methods . Specifically , one augmentation policy has 5 sub-policies ; each sub-policy consists of 2 augmentation operations . The base operation pool includes the following 15 operations : ShearX/Y , TranslateX/Y , Rotate , AutoContrast , Invert , Equalize , Solarize , Posterize , Contrast , Color , Brightness , Sharpness and Cutout . Each operation has 11 uniformly discretized magnitudes which is randomly selected upon each use . We provide an abstract analysis for stochastic policy through a data deficiency complement point of view , in which a relatively slower accuracy increase is expected for stochastic method in early stage of training . We refer to the time period required by stochastic approach to accumulate enough amount of data in the deficient dimensions as the Stochastic Accumulation Stage ( SAS ) . However , as training carries on , the amount of data of deficient dimension in the random approach may gradually get close to or even go beyond the one in search-based methods . When enough number of epochs is reached , the performance of random policy may match or even overtake search-based methods . More detailed intuitions are provided in section A.1 of the appendix . Note that we are not claiming that this stochastic policy outperforms the search-based augmentation policies , but rather , this stochastic policy provides a good foundation for us to build upon ; it allows us to further incorporate additional operators and a multi-stage complexity augmentation strategy with very little or no extra cost compared to search-based methods . 3.2 INTRODUCING ADDITIONAL AUGMENTATION OPERATIONS . In our method , we introduce mix-based operations such as mixup , cutmix , and augmix ( Hendrycks et al. , 2020 ) in addition to the randomly generated augmentation policies . RandAugment showed certain operation ( e.g . posterize ) brought consistent negative effect to augmentation result regardless of the number of operations in the pool ; while some ( e.g . rotate ) had consistent positive effect . We also observed some operations might harm the augmentation performance when directly applied on stochastic method ; however , an interesting finding is that the negative influence could be weakened or even turned back to positive when these operations were used together with our complexity driven strategy , which is explained in next section . Note that these mix-based operators can also be introduced to the search-based methods . The main difference is that there is no extra overhead or cost as far as our method is concern ; whereas these additional operators will incur even larger cost for an already expensive search process .

Summary: This paper aims to provide an effective augmentation strategy without the need for a separate search. The resulting method is called NOSE Augment, which is presented as a substitute for the previous AutoAugment type methods (e.g. Fast AutoAugment, Population Based Augmentation, RandAugment, Adversarial AutoAugment etc.) In parallel to this goal, the authors propose adding the mixing-based augmentation operations Mixup, Cutmix, and Augmix into the list of operations used in AutoAugment. Finally, authors also employ a curriculum of augmentation strength during training.