Ada-Boundary: Accelerating the DNN Training via Adaptive Boundary Batch Selection

1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved remarkable performance in many fields , especially , in computer vision and natural language processing ( Krizhevsky et al. , 2012 ; Goodfellow et al. , 2016 ) . Nevertheless , as the size of data grows very rapidly , the training step via stochastic gradient descent ( SGD ) based on mini-batches suffers from extremely high computational cost , which is mainly due to slow convergence . The common approaches for expediting convergence include some SGD variants ( Zeiler , 2012 ; Kingma and Ba , 2015 ) that maintain individual learning rates for parameters and batch normalization ( Ioffe and Szegedy , 2015 ) that stabilizes gradient variance . Recently , in favor of the fact that not all samples have an equal impact on training , many studies have attempted to design sampling schemes based on the sample importance ( Wu et al. , 2017 ; Fan et al. , 2017 ; Katharopoulos and Fleuret , 2018 ) . Curriculum learning ( Bengio et al. , 2009 ) inspired by human ’ s learning is one of the representative methods to speed up the training step by gradually increasing the difficulty level of training samples . In contrast , deep learning studies focus on giving higher weights to harder samples during the entire training process . When the model requires a lot of epochs for convergence , it is known to converge faster with the batches of hard samples rather than randomly selected batches ( Schaul et al. , 2016 ; Loshchilov and Hutter , 2016 ; Gao and Jojic , 2017 ) . There are various criteria for judging the hardness of a sample , e.g. , the rank of the loss computed from previous epochs ( Loshchilov and Hutter , 2016 ) . Here , a natural question arises : Does the “ hard ” batch selection always speed up DNN training ? Our answer is partially yes : it is helpful only when training an easy data set . According to our indepth analysis , as demonstrated in Figure 1 ( a ) , the hardest samples in a hard data set ( e.g. , CIFAR10 ) were too hard to learn . They are highly likely to make the decision boundary bias towards themselves , as shown in Figure 1 ( b ) . On the other hand , in an easy data set ( e.g. , MNIST ) , the hardest samples , though they are just moderately hard , provide useful information for training . In practice , it was reported that hard batch selection succeeded to speed up only when training the easy MNIST data set ( Loshchilov and Hutter , 2016 ; Gao and Jojic , 2017 ) , and our experiments in Section 4.4 also confirmed the previous findings . This limitation calls for a new sampling scheme that supports both easy and hard data sets . In this paper , we propose a novel adaptive batch selection strategy , called Ada-Boundary , that accelerates training and is better generalized to hard data sets . As opposed to existing hard batch selection , Ada-Boundary picks up the samples with the most appropriate difficulty , considering the learning progress of the model . The samples near the current decision boundary are selected with high probability , as shown in Figure 2 ( a ) . Intuitively speaking , the samples far from the decision boundary are not that helpful since they are either too hard or too easy : those on the incorrect ( or correct ) side are too hard ( or easy ) . This is the reason why we regard the samples around the decision boundary , which are moderately hard , as having the appropriate difficulty at the moment . Overall , the key idea of Ada-Boundary is to use the distance of a sample to the decision boundary for the hardness of the sample . The beauty of this design is not to require human intervention . The current decision boundary should be directly influenced by the learning progress of the model . The decision boundary of a DNN moves towards eliminating the incorrect samples as the training step progresses , so the difficulty of the samples near the decision boundary gradually increases as the model is learned . Then , the decision boundary keeps updated to identify the confusing samples in the middle of SGD , as illustrated in Figure 2 ( b ) . This approach is able to accelerate the convergence speed by providing the samples suited to the model at every SGD iteration , while it is less prone to incur an overfitting issue . We have conducted extensive experiments to demonstrate the superiority of Ada-Boundary . Two popular convolutional neural network ( CNN ) 1 models are trained using three benchmark data sets . Compared to random batch selection , Ada-Boundary significantly reduces the execution time by 14.0–33.5 % . At the same time , it provides a relative improvement of test error by 7.34–14.8 % in the final epoch . Moreover , compared to the state-of-the-art hard batch selection ( Loshchilov and Hutter , 1The idea is applicable to the DNNs other than CNNs , and we leave this extension as the future work . 2016 ) , Ada-Boundary achieves the execution time smaller by 18.0 % and the test error smaller by 13.7 % in the CIFAR-10 data set . 2 Ada-Boundary COMPONENTS The main challenge for Ada-Boundary is to evaluate how close a sample is to the decision boundary . In this section , we introduce a novel distance measure and present a method of computing the sampling probability based on the measure . 2.1 SAMPLE ’ S DISTANCE BASED ON SOFTMAX DISTRIBUTION . To evaluate the sample ’ s distance to the decision boundary , we note that the softmax distribution , which is the output of the softmax layer in neural networks , clearly distinguishes how confidently the learner predicts and whether the prediction is right or wrong , as demonstrated in Figure 3 . Let h ( y|xi ; θt ) be the softmax distribution of a given sample xi over y ∈ { 1 , 2 , . . . , k } labels , where θt is the parameter of a neural network at time t. Then , the distance from a sample xi with the true label yi to the decision boundary of the neural network with θt is defined by the directional distance function in Eq . ( 1 ) . More specifically , the function consists of two terms related to the direction and magnitude of the distance , determined by the model ’ s correctness and confidence , respectively . The correctness is determined by verifying whether the label with the highest probability matches the true label yi , and the confidence is computed by the standard deviation of the softmax distribution . Intuitively , the standard deviation is a nice indicator of the confidence because the value gets closer to zero when the learner confuses . dist ( xi , yi ; θ t ) = correctness︷ ︸︸ ︷ sign ( xi , yi ) · confidence︷ ︸︸ ︷ std ( h ( y|xi ; θt ) ) sign ( xi , yi ) = { +1 , argmaxy∈ { 1,2 , ... , k } h ( y|xi ; θt ) = yi −1 , otherwise ( 1 ) One might argue that the cross-entropy loss , H ( p , q ) = −p ( xi ) log ( q ( xi ) ) where p ( xi ) and q ( xi ) are the true and softmax distributions for xi , can be adopted for the distance function . However , because p ( xi ) is formulated as a one-hot true label vector , the cross-entropy loss can not capture the prediction probability for false labels , which is an important factor of confusing samples . Another advantage is that our distance function is bounded as opposed to the loss . For k labels , the maximum value of std ( h ( y|xi ; θt ) ) is k−1 √ ( k − 1 ) when h ( m|xi ; θt ) = 1 and ∀l =mh ( l|xi ; θt ) = 0 . Thus , dist ( xi , yi ; θt ) is bounded as in Eq . ( 2 ) . − k−1√k − 1 ≤ dist ( xi , yi ; θt ) ≤ k−1 √ k − 1 ( 2 ) 2.2 SAMPLING PROBABILITY BASED ON QUANTIZATION INDEX . The rank-based approach introduced by Loshchilov and Hutter ( 2016 ) is a common way to make the sampling probability of being selected for the next mini-batch . This approach sorts the samples by a certain importance measure in descending order , and exponentially decays the sampling probability of a given sample according to its rank . Let N denote the total number of samples . Then , each r-th ranked sample is selected with the probability p ( r ) which drops by a factor of exp ( log ( se ) /N ) . Here , se is the selection pressure parameter that affects the probability gap between the most and the least important samples . When normalized to sum up to 1.0 , the probability of the r-th ranked sample ’ s being selected is defined by Eq . ( 3 ) . p ( r ) = 1/ exp ( log ( se ) /N ) r∑N j=1 1/ exp ( log ( se ) /N ) j ( 3 ) In the existing rank-based approach , the rank of a sample is determined by |dist ( xi , yi ; θt ) | in ascending order , because it is inversely proportional to the sample importance . However , if the mass of the true sample distribution is skewed to one side ( e.g. , easy side ) as shown in Figure 4 , the mini-batch samples are selected with high probability from the skewed side rather than around the decision boundary where |dist ( xi , yi ; θt ) | is very small . This problem was attributed to unconditionally fixed probability to a given rank . In other words , the samples with similar ranks are selected with similar probabilities regardless of the magnitude of the distance values . To incorporate the impact of the distance into batch selection , we adopt the quantization method ( Gray and Neuhoff , 1998 ; Chen and Wornell , 2001 ) and use the quantization index q instead of the rank r. Let Δ be the quantization step size and d be the output of the function dist ( xi , yi ; θt ) of a given sample xi . Then , the index q is obtained by the quantizer Q ( d ) as in Eq . ( 4 ) . The quantization index gets larger as a sample moves away from the decision boundary . In addition , the difference between two indexes reflects the difference in the actual distances . q = Q ( d ) , Q ( d ) = |d|/Δ ( 4 ) In Eq . ( 4 ) , we set Δ to be k−1 √ k − 1/N such that the index q is bounded to N ( the total number of samples ) by Eq . ( 2 ) . The sampling probability of a given sample xi with the true label yi is defined as Eq . ( 5 ) . As shown in Figure 4 , our quantization-based method provides a well-balanced distribution , even if the true sample distribution is skewed . p ( xi , yi ) = 1/ exp ( log ( se ) /N ) Q ( dist ( xi , yi ; θ t ) ) ∑N j=1 1/ exp ( log ( se ) /N ) Q ( dist ( xj , yj ; θt ) ) ( 5 ) 3 Ada-Boundary ALGORITHM

Summary: The paper introduces an adaptive importance sampling strategy, as opposed to uniform sampling, for batch normalization. The key idea is to assign higher importance to those correctly classified training samples with relatively smaller soft-max prediction variance, hopefully to push the deep nets to learn faster from uncertain samples near the decision boundary. Experimental results on several benchmark datasets (MNIST, CIFAR-10) and commonly used deep nets (LeNet, ResNet) are reported to show the power of boundary batch selection in improving the overall training efficiency.