Multi-Sample Dropout for Accelerated Training and Better Generalization

1 INTRODUCTION . Dropout ( Hinton et al . ( 2012 ) ) is one of the key regularization techniques for improving the generalization of deep neural networks ( DNNs ) . Because of its simplicity and efficiency , the original dropout and various similar techniques are widely used to train neural networks for various tasks . The use of dropout prevents the trained network from overfitting to the training data by randomly discarding ( i.e. , `` dropping '' ) 50 % of the neurons at each training iteration . As a result , the neurons can not depend on each other , and the trained network achieves better generalization . During inference , neurons are not discarded , so all information is preserved ; instead , each outgoing value is multiplied by 0.5 to make the average value consistent with the training time . The network used for inference can be viewed as an ensemble of many sub-networks randomly created during training . The success of dropout inspired the development of many techniques using various ways for selecting information to discard . For example , DropConnect ( Wan et al . ( 2013 ) ) discards a portion of the connections between neurons randomly selected during training instead of randomly discarding neurons . This paper reports multi-sample dropout , a dropout technique extended in a different way especially for deep convolutional neural networks ( CNNs ) . The original dropout creates a randomly selected subset ( a dropout sample ) from the input during training . The proposed multi-sample dropout creates multiple dropout samples . The loss is calculated for each sample , and then the sample losses are averaged to obtain the final loss used for back propagation . By calculating losses for M dropout samples and ensembling them , network parameters are updated to achieve smaller loss with any of these samples . This is similar to performing M training repetitions for each input image in the same minibatch . Therefore , it significantly reduces the number of iterations needed for training . We observed that multi-sample dropout also improved accuracy of the trained network ; it can reduce the gradient noises caused by dropout . We do not discard neurons in the inference , as with the original dropout . Experiments demonstrated that it achieves smaller losses and errors for both the training set and validation set on image classification tasks . As far as the authors know , no other dropout technique uses a similar approach to accelerate training . In CNNs , dropout is typically applied to layers near the end of the network . VGG16 ( Simonyan & Zisserman ( 2014 ) ) , for example , uses dropout for 2 fully connected layers following 13 convolution layers . Because the execution time for the fully connected layers is much shorter than that for the convolution layers , duplicating the fully connected layers for each of multiple dropout samples does not significantly increase the total execution time per iteration . Experiments using the ImageNet , CIFAR-10 , CIFAR-100 , and Street View House Numbers ( SVHN ) datasets showed that , with an increasing number of dropout samples created at each iteration , the improvements obtained ( reduced number of iterations needed for training and higher accuracy ) became more significant at the expense of a longer execution time per iteration and greater memory consumption . Consideration of the reduced number of iterations along with the increased time per iteration revealed that the total training time was the shortest with a moderate number of dropout samples , such as 8 . Multi-sample dropout can be easily implemented on deep learning frameworks without adding a new operator by duplicating a part of the network after the dropout layer while sharing the weights among the fully connected layers duplicated for each dropout sample . The main contribution of this paper is multi-sample dropout , a new regularization technique for accelerating the training of deep neural networks compared to the original dropout . Evaluation of multi-sample dropout on image classification tasks demonstrated that it increases accuracy for both the training and validation sets as well as accelerating the training . 2 MULTI-SAMPLE DROPOUT . 2.1 OVERVIEW . This section describes the multi-sample dropout technique . The basic idea is quite simple : create multiple dropout samples instead of only one . Figure 1 depicts an easy way to implement multisample dropout ( with two dropout samples ) using an existing deep learning framework with only common operators . The dropout layer and several layers after the dropout are duplicated for each dropout sample ; in the figure , the `` dropout , '' `` fully connected , '' and `` softmax + loss func '' layers are duplicated . Different masks are used for each dropout sample in the dropout layer so that a different subset of neurons is used for each dropout sample . In contrast , the parameters ( i.e. , connection weights ) are shared between the duplicated fully connected layers . The loss is computed for each dropout sample using the same loss function , e.g. , cross entropy , and the final loss value is obtained by averaging the loss values for all dropout samples . This final loss value is used as the objective function for optimization during training . We select the class label as the prediction based on the average of outputs from the last fully connected layer . Although a configuration with two dropout samples is shown in Figure 1 , multi-sample dropout can be configured to use any number of dropout samples . The original dropout can be seen as a special case of multi-sample dropout where the number of samples is set to one . During inference , Neurons are not discarded as is done in the original dropout . The loss can be calculated using only one dropout sample because the dropout samples become identical at the inference time if we do not drop any neurons . Hence , We always use only one dropout sample at inference regardless of the training method . When dropout is applied to a layer near the end of the network , the additional execution time due to the duplicated operations is not significant ; this characteristic makes multi-sample dropout more suitable for deep CNNs . We can apply multi-sample dropout for shallow networks , such as the multilayer perceptron . As we show an example in Appendix , multi-sample dropout reduces the number of iterations for training , but the costs of the increased execution time per iteration may surpass the benefits . If the network includes multiple dropout layers , we can apply multi-sample dropout at any of these dropout layers . Multi sampling at an earlier dropout layer may increase diversity among dropout samples and increase the benefits in trade for the higher additional costs due to more duplicated layers . For example , WideResNet ( Zagoruyko & Komodakis ( 2016 ) ) repeatedly executes dropout since it employs residual blocks which include dropout layers . However , we do not need to duplicate the entire network for applying multi-sample dropout to WideResNet ; we can apply the multi sampling at the last residual block for example . 2.2 WHY MULTI-SAMPLE DROPOUT ACCELERATES TRAINING . Intuitively , the effect of multi-sample dropout with M dropout samples is similar to that of enlarging the size of a minibatch M times by duplicating each sample in the minibatch M times . For example , if a minibatch consists of two data samples 〈A , B〉 , training a network by using multi-sample dropout with two dropout samples closely corresponds to training a network by using the original dropout and a minibatch of 〈A , A , B , B〉 . Using a larger minibatch size with duplicated samples may not make sense because it increases the computation time M times . In contrast , multi-sample dropout can enjoy similar gains without a huge increase in computation cost for deep CNNs because it duplicates only the operations after dropout . Because of the non-linearity of the activation functions , the original dropout with duplicated samples and multi-sample dropout do not give exactly the same results . However , similar acceleration was observed in the training in terms of the number of iterations , as shown by the experimental results . 2.3 OTHER SOURCES OF DIVERSITY AMONG SAMPLES . The key to faster training with multi-sample dropout is the diversity among dropout samples ; if there is no diversity , the multi-sampling technique gives no gain and simply wastes computation resources . Although dropout is one of the best sources of diversity , the multi-sampling technique can be used with other sources of diversity . For example , variants of dropout , such as DropConnect , can be enhanced by using the multi-sampling technique . To demonstrate that benefits can be obtained from other sources of diversity , two additional diversitycreation techniques are employed in this paper : 1 ) horizontal flipping and 2 ) zero padding at a pooling layer . As shown in Appendix , improvements due to these additional sources of diversity are visible but much less significant than the improvements from dropout . Such results show that dropout is an ideal source of diversity to use in our multi-sampling technique . In this paper , these additional sources of diversity are used only in the training phase , but not in the inference . 3 EXPERIMENTAL RESULTS . 3.1 IMPLEMENTATION . This section describes the effects of using the multi-sample dropout for various image classification tasks using the ImageNet , CIFAR-10 , CIFAR-100 , and SVHN datasets . For the ImageNet dataset , as well as for the full dataset with 1,000 classes , a reduced dataset with only the first 100 classes was tested ( ImageNet-100 ) . For most of the experiments , we use eight as the number of dropout samples , which generally gives good tradeoff between benefits and additional cost . For the CIFAR10 , CIFAR-100 , and SVHN datasets , an 8-layer network with six convolutional layers and batch normalization ( Ioffe & Szegedy ( 2015 ) ) followed by two fully connected layers with dropout was used ( see appendix for detail ) . This network executes dropout twice with dropout ratios of 40 % and 30 % , which are tuned for the original dropout but are used here for all cases unless otherwise specified . The same network architecture except for the number of neurons in the output layer was used for the CIFAR-10 , SVHN ( 10 output neurons ) , and CIFAR-100 ( 100 output neurons ) datasets . The network was trained using the Adam optimizer ( Kingma & Ba ( 2014 ) ) with a batch size of 100 . These tasks were run on a NVIDIA K20m GPU with CUDA 9.0 using the Chainer v4.0 ( Tokui et al . ( 2015 ) ) as the framework . For the ImageNet datasets , VGG16 ( Simonyan & Zisserman ( 2014 ) ) was used as the network architecture , and the network was trained using stochastic gradient descent with momentum as the optimization method with a batch size of 100 samples . The initial learning rate of 0.01 was exponentially decayed by multiplying it by 0.95 at each epoch . Weight decay regularization was used with a decay rate of 5 · 10−4 by following the original paper . In the VGG16 architecture , dropout was applied for the first two fully connected layers with 50 % as dropout ratio . A NVIDIA V100 GPU with CUDA 10.0 was used for the training with ImageNet datasets . For all datasets , data augmentation was used by extracting a patch from a random position of the input image and by performing random horizontal flipping during training ( Krizhevsky et al . ( 2012 ) ) . For the validation set , the patch from the center position was extracted and fed into the classifier without horizontal flipping .

Summary: This paper propose an ensemble of dropout: it applies multiple copies of a neural net with different dropout configurations (i.e., dropout masks) to the same mini-batch, and the training loss is computed as the sum of losses incurred on the multiple copies. They claim that such ensembling can improve training performance over the original dropout without increasing too much computation. Experiments on several datasets show that the proposed method can achieve slightly better validation accuracies than the original dropout.