NOSE Augment: Fast and Effective Data Augmentation Without Searching

1 INTRODUCTION . Data is an essential and dominant factor for learning AI models , especially in deep learning era where deep neural networks normally require large data volume for training . Data augmentation techniques artificially create new samples to increase the diversity of training data and in turn the generalization of AI models . For example , different image transformation operations , such as rotation , flip , shear etc. , have been used to generate variations on original image samples in image classification and other computer vision tasks . More intricate augmentation operations have also been implemented , such as Cutout ( Devries & Taylor , 2017 ) , Mixup ( Zhang et al. , 2018 ) , Cutmix ( Yun et al. , 2019 ) , Sample Pairing ( Inoue , 2018 ) , and so on . How to formulate effective augmentation strategies with these basic augmentation methods becomes the crucial factor to the success of data augmentation . Recent works ( Cubuk et al. , 2019 ; Lim et al. , 2019 ; Ho et al. , 2019 ) introduced automated searching or optimization techniques in augmentation policy search . The common assumption of these methods is : a selected subset of better-fit augmentation policies will produce more relevant augmented data which will in turn result in a better trained model . Here the augmentation policy is defined by an ordered sequence of augmentation operations , such as image transformations , parameterized with probability and magnitude . Though these methods achieved state-of-the-art accuracies on image classification tasks , they lead to high computational cost in general , due to large search space and extra training steps . More importantly , it is worth exploring whether it is really necessary to find the best-fit subset of policies with specific parameter values of probability and magnitude . RandAugment ( Cubuk et al. , 2020 ) has started to simplify the parameters and scale down the search space defined by AutoAugment Cubuk et al . ( 2019 ) , but their method still relied on grid search for iterative optimization of the simplified parameters . Our method aims to fully avoid policy search and cost , meanwhile to maintain or improve model performance in terms of both accuracy and training efficiency . Our work showed that by applying simple stochastic augmentation policies with the same sampling space and other settings of training , we could obtain equal or very close performance with search-based augmentation methods . Another advantage of stochastic policies is that adding more operations in the pool does not bring additional cost ; while in search-base methods , more operations in the pool causes exponential increase of the search space . Therefore , the second part of our method is to add more operations to the pool to bring more data diversity . In practice , we introduced a new category of operations such as mixup and cutmix into the operation pool . Furthermore , we tackled automated augmentation from overall data usage point of view , in contrast to data creation point of view accentuated by policy-search based methods . Inspired by the idea of Curriculum Learning ( CL ) ( Bengio et al. , 2009 ) , which presents training samples in an increasing order of difficulties , our method defines various complexity levels of augmentation strategies and applies them with orders on phased training stages . To avoid the confounding overfitting problem of original Curriculum Learning in practice , our method applies the inverted order of Curriculum Learning , which presents the hardest augmentation strategies from the beginning and gradually decreases the complexity levels . In general , our augmentation method replaces policy search with stochastic policy generation , upon which it introduces more operations for better diversity and phased augmentation strategy with decreasing complexities for a smooth learning convergence , and as an integral solution it achieves better results . Figure 1 describes our method and the difference compared to search-based methods . The main contributions of this paper can be summarized as follows : 1 . We present a no-search ( NOSE ) augmentation method as an alternative of computation-intensive search-based auto-augment methods . By jointly applying phased augmentation strategy and introducing more augmentation operations on top of a simple stochastic augmentation mechanism , NOSE augment achieves state-of-the-art ( SOTA ) accuracies on CIFAR 10 , CIFAR 100 ( Krizhevsky , 2009 ) and close-to-SOTA results on other benchmark datasets . Our ablation study demonstrates that all the components of our methods should be combined together to achieve the best performance . 2 . We demonstrate that a stochastic-based augmentation approach can obtain accuracies comparable to those of search-based methods while achieving overwhelming advantage on overall augmentation and training efficiency as the searching phase is completely avoided . 3 . Besides image classification , we also applied NOSE augment on face recognition and text detection ( OCR ) tasks , and obtained competitive or better results in comparison with search-based methods . This further proves the advantage and generality of NOSE augment . 2 RELATED WORKS . Our work is most closely related to RandAugment ( Cubuk et al. , 2020 ) . A key idea in RandAugment is to avoid a separate search phase on proxy datasets but chose instead to reduce the search space by decreasing the number of parameters . For instance , the probability for each operator is set to a constant uniform probability . The two sets of data augmentation parameters to be searched are N , the number of transformations for a given image and M , the discrete operator magnitude . These two parameters are regarded as hyperparameters and tuned using grid search . Our method is similar to RandAugment in terms of the operators used and the setting of equal probability for each operator . The most contrasting difference between RandAugment and our method is that RandAugment is still a search-based method with search cost while our method skips policy search completely and tackles the problem by introducing new augmentation operations and further new strategies built upon stochastic policies . Our work stands diametrically opposite to the AutoAugment series of methods ( Cubuk et al. , 2019 ; Lim et al. , 2019 ; Zhang et al. , 2020 ) , in which the best data augmentation policy is derived through a search . AutoAugment ( Cubuk et al. , 2019 ) uses a RNN controller to propose an augmentation strategy . Based on this proposed augmentation strategy , the model is trained and use the resulting validation accuracy to update the RNN controller . Although AutoAugment achieves good results on the standard benchmark datasets , the search cost is prohibitively huge , for instance , 5000 GPU ( NVIDIA Tesla P100 ) hours on the CIFAR-10 dataset and Pyramid-Net+ShakeDrop model . Fast AutoAugment ( Fast AA ) ( Lim et al. , 2019 ) ameliorates AutoAugment ’ s huge search cost with a three pronged approach . Smaller datasets are used for policy search . The idea of density matching is proposed to avoid having to re-train the model for assessing the validation accuracy for each proposed policy . The use of Bayesian Optimization based method ( Bergstra et al. , 2011 ) also helps to converge to an effective augmentation policy quickly . Adversarial Augment ( Adv AA ) ( Zhang et al. , 2020 ) reduces the huge computational cost of AutoAugment through the adversarial policy framework , which generates data samples that maximizes the training loss of the target network . PBA ( Ho et al. , 2019 ) introduces the idea of non-stationary policy schedules instead of the fixed augementation policy proposed in AutoAugment . The non-stationary policy schedules refers to how the policy evolves with the training epochs . Our main differences with aforementioned state-of-the-art methods lie in our stochastic policy described in section 3.1 and the multi-stage complexity driven augmentation policy outlined in section 3.3 . We use the same set of operators and settings as these related works . 3 METHOD . We describe the three key components of our proposed method in this section . In section 3.1 , we put forward a stochastic-based method as opposed to the search-based paradigm advocated by many of the state-of-the-art works described in section 2 . Due to the counter-intuitiveness ( but effective ) nature of this stochastic-based method , we first provide insights and motivations into stochastic augmentation policies in section 3.1 . In section 3.2 , we capitalize on our proposed stochastic-based method by proposing additional operators that will further enrich the data augmentation diversity , since there is no extra search cost for adding new operators with a no-search method . In 3.3 , we propose a multi-stage complexity driven policy that helps to resolve the tension between augmented data diversity and data distribution fidelity . Our ablation study demonstrates that the performance of our method is not a simple incremental benefit with each individual component , but rather , the three components need to work together to achieve a competitive performance with no policy search cost . 3.1 STOCHASTIC POLICY - SKIPPING POLICY SEARCH . The first component of our method removes policy searching completely and applies a stochastic augmentation policy with randomly selected operations and magnitudes . Unlike the search-based methods , once an operation is selected , it is used with 100 % probability . Our stochastic method follows the same policy definition as existing search-based methods . Specifically , one augmentation policy has 5 sub-policies ; each sub-policy consists of 2 augmentation operations . The base operation pool includes the following 15 operations : ShearX/Y , TranslateX/Y , Rotate , AutoContrast , Invert , Equalize , Solarize , Posterize , Contrast , Color , Brightness , Sharpness and Cutout . Each operation has 11 uniformly discretized magnitudes which is randomly selected upon each use . We provide an abstract analysis for stochastic policy through a data deficiency complement point of view , in which a relatively slower accuracy increase is expected for stochastic method in early stage of training . We refer to the time period required by stochastic approach to accumulate enough amount of data in the deficient dimensions as the Stochastic Accumulation Stage ( SAS ) . However , as training carries on , the amount of data of deficient dimension in the random approach may gradually get close to or even go beyond the one in search-based methods . When enough number of epochs is reached , the performance of random policy may match or even overtake search-based methods . More detailed intuitions are provided in section A.1 of the appendix . Note that we are not claiming that this stochastic policy outperforms the search-based augmentation policies , but rather , this stochastic policy provides a good foundation for us to build upon ; it allows us to further incorporate additional operators and a multi-stage complexity augmentation strategy with very little or no extra cost compared to search-based methods . 3.2 INTRODUCING ADDITIONAL AUGMENTATION OPERATIONS . In our method , we introduce mix-based operations such as mixup , cutmix , and augmix ( Hendrycks et al. , 2020 ) in addition to the randomly generated augmentation policies . RandAugment showed certain operation ( e.g . posterize ) brought consistent negative effect to augmentation result regardless of the number of operations in the pool ; while some ( e.g . rotate ) had consistent positive effect . We also observed some operations might harm the augmentation performance when directly applied on stochastic method ; however , an interesting finding is that the negative influence could be weakened or even turned back to positive when these operations were used together with our complexity driven strategy , which is explained in next section . Note that these mix-based operators can also be introduced to the search-based methods . The main difference is that there is no extra overhead or cost as far as our method is concern ; whereas these additional operators will incur even larger cost for an already expensive search process .

Summary: The authors propose a method for learning an augmentation pipeline for image recognition. As opposed to recent existing approaches such as AutoAugment or RandAugment, the authors do not seek for the augmentation pipeline iteratively. Instead they use a stochastic approach, where augmenters are split to three categories based on their complexity to be used by curriculum learning.