Augment your batch: better training with larger batches

1 INTRODUCTION . 1.1 LARGE BATCH TRAINING OF NEURAL NETWORKS . Neural network training is known to be highly compute intensive , often requiring large amounts of resources , preferably utilizing as much parallelism as possible . Training is usually done using stochastic-gradient-descent ( SGD ) or one of its variants , where the gradient for each sample is computed separately and accumulated for each weight update . Using this property , the most straightforward way to scale training is by parallel computation of gradients over many samples , also known as ” data parallelism ” ( Krizhevsky , 2014 ) . While data parallelism by batch computation is common , very large batches ( > 1000 ) were avoided by practitioners , as they were claimed to cause an inherent generalization issue , later described as a tendency to reach ” sharp minima ” ( Keskar et al. , 2017 ) . This so-called ” generalization gap ” was lately challenged by Hoffer et al . ( 2017 ) which identified it to be mostly caused by the reduction in optimization updates in large batch training rather than some inherent property . Recent approaches by Hoffer et al . ( 2017 ) , Goyal et al . ( 2017 ) , You et al . ( 2017 ) and others showed that equally good and even better generalization could be reached in some cases using large batches , by adapting the optimization regime used . These methods often use higher learning rates than previously used , to account for the lower gradient variance in large batch updates . Hoffer et al . ( 2017 ) argued that the quality of the optimized model stems from the number of SGD iterations , rather than the number of cycles through the training data ( epochs ) . Naturally , using large batches reduces the number of iterations in each epoch , and so additional epochs may be needed to reach the same quality of a model trained with a smaller batch . Ott et al . ( 2018 ) demonstrated that large batch training could be beneficial even when batch size does not fit into the device and its update is performed by multiple accumulations . Using this method , they report added stability due to reduced gradient noise , as well as lower communication overhead . In contrast , Masters & Luschi ( 2018 ) suggested that small batch updates may still provide benefits over large batch ones , showing better results over several tasks , with higher robustness to hyperparameter selection such as the learning rate used . With more available computation power , larger batch sizes can allow shorter training time . So far , this motivation is what primarily made large batch training appealing . Here , instead , we propose a novel way of utilizing large batch such that the use of large batch improves generalization performance of the final model . As modern hardware scales up by increasing the computational resources rather than the bandwidth available , small batch updates are causing a noticeable under-utilization of these devices . Because of that , for a given hardware specification it is possible to train with larger batch sizes without a significant time penalty ( You et al. , 2017 ; Grave et al. , 2016 ) . With this notion , we show that it is possible to improve model accuracy by utilizing available resources to the maximum using larger batch sizes . 1.2 ROLE AND SIGNIFICANCE OF DATA AUGMENTATIONS . A common practice in training modern neural network is to use data augmentations – multiple instances of input samples , each with a different transformation applied to it . For example , on image classification tasks , for any input image , a random crop of varying size and scale is applied to it , together with potentially rotation , mirroring and even color jittering ( Krizhevsky et al. , 2012 ) . Data augmentations were repeatedly found to provide efficient and useful regularization , often accounting for significant portion of the final generalization performance ( Zagoruyko , 2016 ; DeVries & Taylor , 2017 ) . Several works even tried to learn how to generate good data augmentations . For example , Bayesian approaches based on the training set distribution ( Tran et al. , 2017 ) , generative approaches based on generative adversarial networks ( Antoniou et al. , 2017 ; Sixt et al. , 2018 ) and search methods aimed to find the best data augmentation policy ( Cubuk et al. , 2018 ) . Our approach is orthogonal to those methods ; thus we believe we can expect even better results by combining them . Other regularization methods , such as Dropout ( Srivastava et al. , 2014 ) or ZoneOut ( Krueger et al. , 2016 ) , although not explicitly considered as data augmentation techniques , can be considered as such by viewing them as random transforms over inputs for intermediate layers . These methods were also shown to benefit models in various tasks . 2 BATCH AUGMENTATION . In this work , we suggest leveraging the merits of data augmentation together with large batch training , by using multiple instances of a sample in the same batch . We consider a model with a loss function ` ( w , xn , yn ) where { xn , yn } Nn=1 is a dataset of N data sample-target pairs . T ( x ) is some data augmentation transform applied to each example – e.g , a random crop from an image . The common training procedure for each batch consists of the following update rule ( here using vanilla SGD with a learning-rate η and mini-batch size of B , for simplicity ) : wt+1 = wt − η 1 B ∑ n∈B ( k ( t ) ) ∇w ` ( wt , T ( xn ) , yn ) where k ( t ) is sampled from { 1 , . . . , N/B } and we assume for simplicity that B divides N . We suggest to introduce M multiple instances of the same input sample by applying the transform Ti , here denoted by subscript i ∈ { 1 .. M } to highlight the fact that they are different from one another . We now use the slightly modified learning rule : wt+1 = wt − η 1 M ·B M∑ i=1 ∑ n∈B ( k ( t ) ) ∇w ` ( wt , Ti ( xn ) , yn ) effectively using a larger M ·B batch at each steps , that is composed of B samples augmented with M different transforms each . We note that this updated rule can be computed either by evaluating on the whole M ·B batch or by accumulating M instances of the original gradient computation . Using large batch updates as part of batch augmentations makes no change to the number of SGD iterations that are performed for each epoch . Batch augmentation ( BA ) can also be used as transforms over inputs of intermediate layers . For example , we can use the common Dropout regularization method ( Srivastava et al. , 2014 ) to generate multiple instances of the same sample in a given layer , each with its dropout mask . Batch augmentation can be easily implemented in any framework with a reference PyTorch ( Paszke et al. , 2017 ) implementation to be available soon at https : //github.com/ paper-submissions/augment-batch . To further highlight the ease of incorporating these ideas , we note that BA can be added to any training code by merely modifying the input pipeline – augmenting each batch that is fed to the model . 3 EXPERIMENTS . To evaluate the impact of batch augmentation , we used several common datasets and neural network based models . For each one of the models , unless explicitly stated , we tested our approach using the original training regime and data augmentation described by its authors . For simplicity , we did not change the learning rate used , although this is possibly sub-optimal . 3.1 CIFAR10/100 . The Cifar10 dataset introduced by Krizhevsky ( 2009 ) is a popular image classification dataset containing 50 , 000 training images , together with a 10 , 000 test set . Each image is of size 32 × 32 and belongs to one of 10 classes of vehicles and animals . The Cifar100 dataset consists of the same number of training and validation images and the same spatial size , but with an increase to 100 in the number of possible classes for each image . For both datasets , we used the common data augmentation technique as described by He et al . ( 2016 ) . In this method , the input image is padded with 4 zero-valued pixels at each side , top , and bottom . A random 32 × 32 part of the padded image is then cropped and with a 0.5 probability flipped horizontally . This augmentation method has a rather small space of possible transforms ( 9 · 9 · 2 = 162 ) , and so it is quickly exhausted by even a M ≈ 10s of simultaneous instances . We therefore speculated that using a more aggressive augmentation technique , with larger option space , will yield more noticeable difference when batch augmentation is used . We chose to use the recently introduced ” Cutout ” ( DeVries & Taylor , 2017 ) augmentation method , that was noted to improve the generalization of models on various datasets considerably . Cutout uses randomly positioned zero-valued squares within images , thus increasing the number of possible transforms by ×302 . We first tested batch augmentation on the task discussed by Hoffer et al . ( 2017 ) – using a ResNet44 ( He et al. , 2016 ) over the Cifar10 dataset ( Krizhevsky , 2009 ) together with cutout augmentation ( DeVries & Taylor , 2017 ) . We used the original regime by He et al . ( 2016 ) with a batch of B = 64 . We then compared the learning curve with training using batch augmentation withM ∈ { 2 , 4 , 8 , 16 , 32 } different transforms for each sample in the batch , effectively creating a batch of 64 ·M . As we can see in figure 1 , validation convergence speed has noticeably improved ( in terms of epochs ) , with a significant reduction in final validation classification error ( figure 1b ) . This trend largely continues to improve as M is increased , consistent with our expectation . We verified these results using a variety of models ( table 1 ) using various values of M , depending on our ability to fit the M · B within our compute budget ( specifically , GPU memory ) . Our best result was achieved using DARTS final Cifar10 model Liu et al . ( 2018 ) . DARTS is a differentiable architecture search framework which constructs a graph with a SoftMax parameterized edges . The final model is a subset of the graph whose edges have the highest values . In all our experiments we have observed significant improvements to the final validation accuracy as well , as an increase in accuracy per epoch convergence speed . A typical example can be seen in figure 2 , where using a mere M = 6 , BA improved the results of a Wide-ResNet model by more than 0.5 % ( a relative 17 % decrease in error ) . Moreover , we managed to achieve high validation accuracy much quicker with batch augmentation . We trained a ResNet44 on Cifar10 for half of the iterations needed for the baseline using batch augmentation and larger learning rate . we managed to achieve 93.65 % accuracy while our baseline achieved 93.07 % with double the number of iterations . When the baseline is trained with the same shorten regime there is a significant accuracy degradation . This indicates not only the obvious accuracy gain but a potentially time performance improvement for a given hardware . Finally , we compare with regime adaptation ( RA ) method by Hoffer et al . ( 2017 ) . In this method , the number of epochs is increased so that the number of iteration is fixed when using a larger batch . This makes both RA and BA methods comparable with respect to the number of instances seen for each sample over the course of training . Using the same settings ( ResNet44 , Cifar10 ) , we find an accuracy gain of 0.6 % over the 93.07 % result reported by Hoffer et al . ( 2017 ) . Figures 5 and 6 in Appendix depict these results .

Summary: The paper shows that training with large batch size (e.g., with MxB samples) serves as an effective regularization method for deep networks, thus improving the convergence and generalization accuracy of the models. The enlarged batch of MxB consists of multiple (i.e., B) transforms of each of the M samples from the given batch; the transform is executed by a data augmentation method such as Cutout or Dropout. The authors also provide a theoretical explanation for the working of the method, suggesting that the enlarged batch training decreases the gradient variance during the training of the networks.