Carpe Diem, Seize the Samples Uncertain "at the Moment" for Adaptive Batch Selection

1 INTRODUCTION . Stochastic gradient descent ( SGD ) for randomly selected mini-batch samples is commonly used to train deep neural networks ( DNNs ) . However , many recent studies have pointed out that the performance of DNNs is heavily dependent on how well the mini-batch samples are selected ( Shrivastava et al. , 2016 ; Chang et al. , 2017 ; Katharopoulos & Fleuret , 2018 ) . In earlier approaches , a sample ’ s difficulty is employed to identify proper mini-batch samples , and these approaches achieve a more accurate and robust network ( Han et al. , 2018 ) or expedite the training convergence of SGD ( Loshchilov & Hutter , 2016 ) . However , the two opposing difficulty-based strategies , i.e. , preferring easy samples ( Kumar et al. , 2010 ; Han et al. , 2018 ) versus hard samples ( Loshchilov & Hutter , 2016 ; Shrivastava et al. , 2016 ) , work well in different situations . Thus , for practical reasons to cover more diverse situations , recent approaches begin to exploit a sample ’ s uncertainty that indicates the consistency of previous predictions ( Chang et al. , 2017 ; Song et al. , 2019 ) . An important question here is how to evaluate the sample ’ s uncertainty based on its historical predictions during the training process . Intuitively , because a series of historical predictions can be seen as a series of data indexed in chronological order , the uncertainty can be measured based on two forms of handling time-series observations : ( i ) a growing window ( Figure 1 ( a ) ) that consistently increases the size of a window to use all available observations and ( ii ) a sliding window ( Figure 1 ( b ) ) that maintains a window of a fixed size on the most recent observations by deleting outdated ones . While the state-of-the-art algorithm , Active Bias ( Chang et al. , 2017 ) , adopts the growing window , we propose to use the sliding window in this paper . In more detail , Active Bias recognizes uncertain samples based on the inconsistency of the predictions in the entire history of past SGD iterations . Then , it emphasizes such uncertain samples by choosing them with high probability for the next mini-batch . However , according to our experiments presented in Section 5.2 , such uncertain samples slowed down the convergence speed of training , though they ultimately reduced the generalization error . This weakness is attributed to the inherent limitation of the growing window , where older observations could be too outdated ( Torgo , 2011 ) . In other words , the outdated predictions no longer represent a network ’ s current behavior . As illustrated in Figure 2 , when the label predictions of two samples were inconsistent for a long time , Active Bias invariably regards them as highly uncertain , although their recent label predictions become consistent along with the network ’ s training progress . This characteristic evidently entails the risk of emphasizing uninformative samples that are too easy or too hard at the current moment , thereby slowing down the convergence speed of training . Therefore , we propose a simple but effective batch selection method , called Recency Bias , that takes advantage of the sliding window to evaluate the uncertainty in fresher observations . As opposed to Active Bias , Recency Bias excludes the outdated predictions by managing a sliding window of a fixed size and picks up the samples predicted inconsistently within the sliding window . Thus , as shown in Figure 2 , the two samples uninformative at the moment are no longer selected by Recency Bias simply because their recent predictions are consistent . Consequently , since informative samples are effectively selected throughout the training process , this strategy not only accelerates the training speed but also leads to a more accurate network . To validate the superiority of Recency Bias , two popular convolutional neural networks ( CNNs ) were trained for two independent tasks : image classification and fine tuning . We compared Recency Bias with not only random batch selection ( baseline ) but also two state-of-the-art batch selection strategies . Compared with three batch selection strategies , Recency Bias provided a relative reduction of test error by 1.81 % –20.5 % in a fixed wall-clock training time . At the same time , it significantly reduced the execution time by 24.6 % –59.3 % to reach the same test error . 2 RELATED WORK . Let D = { ( xi , yi ) |1 ≤ i ≤ N } be the entire training dataset composed of a sample xi with its true label yi , where N is the total number of training samples . Then , a straightforward strategy to construct a mini-batchM = { ( xi , yi ) |1 ≤ i ≤ b } is to select b samples uniformly at random ( i.e. , P ( xi|D ) = 1/N ) from the training dataset D. Because not all samples have an equal impact on training , many research efforts have been devoted to develop advanced sampling schemes . Bengio et al . ( 2009 ) first took easy samples and then gradually increased the difficulty of samples using heuristic rules . Kumar et al . ( 2010 ) determined the easiness of the samples using their prediction errors . Recently , Tsvetkov et al . ( 2016 ) used Bayesian optimization to learn an optimal curriculum for training dense , distributed word representations . Sachan & Xing ( 2016 ) emphasized that the right curriculum must introduce a small number of the samples dissimilar to those previously seen . Fan et al . ( 2017 ) proposed a neural data filter based on reinforcement learning to select training samples adaptively . However , it is common for deep learning to emphasize hard samples because of the plethora of easy ones ( Katharopoulos & Fleuret , 2018 ) . Loshchilov & Hutter ( 2016 ) proposed a difficulty-based sampling scheme , called Online Batch , that uses the rank of the loss computed from previous epochs . Online Batch sorts the previously computed losses of samples in descending order and exponentially decays the sampling probability of a sample according to its rank r. Then , the r-th ranked sample x ( r ) is selected with the probability dropping by a factor of exp ( log ( se ) /N ) , where se is the selection pressure parameter that affects the probability gap between the most and the least important samples . When normalized to sum to 1.0 , the probability P ( x ( r ) |D ; se ) is defined by Eq . ( 1 ) . It has been reported that Online Batch accelerates the convergence of training but deteriorates the generalization error because of the overfitting to hard training samples ( Loshchilov & Hutter , 2016 ) . P ( x ( r ) |D ; se ) = 1/ exp ( log ( se ) /N ) r∑N j=1 1/ exp ( log ( se ) /N ) j ( 1 ) Most close to our work , Chang et al . ( 2017 ) devised an uncertainty-based sampling scheme , called Active Bias , that chooses uncertain samples with high probability for the next batch . Active Bias maintains the historyHt−1i that stores all h ( yi|xi ) before the current iteration t ( i.e. , growing window ) , where h ( yi|xi ) is the softmax probability of a given sample xi for its true label yi . Then , it measures the uncertainty of the sample xi by computing the variance over all h ( yi|xi ) inHt−1i and draws the next mini-batch samples based on the normalized probability P ( xi|D , Ht−1i ; ) in Eq . ( 2 ) , where is the smoothness constant to prevent the low variance samples from never being selected again . As mentioned earlier in Section 1 , Active Bias slows down the training process because the oldest part in the historyHt−1i no longer represents the current behavior of the network . P ( xi|D , Ht−1i ; ) = ˆstd ( Ht−1i ) + ∑N j=1 ( ˆstd ( Ht−1j ) + ) , ˆstd ( Ht−1i ) = √√√√var ( h ( yi|xi ) ) + var ( h ( yi|xi ) ) 2|Ht−1i | ( 2 ) For the completeness of the survey , we include the recent studies on submodular batch selection . Joseph et al . ( 2019 ) and Wang et al . ( 2019 ) designed their own submodular objectives that cover diverse aspects , such as sample redundancy and sample representativeness , for more effective batch selection . Differently from their work , we explore the issue of truly uncertain samples in an orthogonal perspective . Our uncertainty measure can be easily injected into their submodular optimization framework as a measure of sample informativeness . In Section 5 , we will confirm that Recency Bias outperforms Online Batch and Active Bias , which are regarded as two state-of-the-art adaptive batch selection methods for deep learning . 3 Recency Bias COMPONENTS 3.1 CRITERION OF AN UNCERTAIN SAMPLE . The main challenge of Recency Bias is to identify the samples whose recent label predictions are highly inconsistent , which are neither too easy nor too hard at the moment . Thus , we adopt the predictive uncertainty ( Song et al. , 2019 ) in Definition 3.1 that uses the information entropy ( Chandler , 1987 ) to measure the inconsistency of recent label predictions . Here , the sample with high predictive uncertainty is regarded as uncertain and selected with high probability for the next mini-batch . Definition 3.1 . ( Predictive Uncertainty ) Let ŷit = Φ ( xi , θt ) be the predicted label of a sample xi at time t andHxi ( q ) = { ŷt1 , ŷt2 , . . . , ŷtq } be the label history of the sample xi that stores the predicted labels at the previous q times , where Φ is a neural network . The label history Hxi ( q ) corresponds to the sliding window of size q to compute the uncertainty of the sample xi . Next , p ( yi|xi ; q ) is formulated such that it provides the probability of the label yi ∈ { 1 , 2 , ... , k } estimated as the label of the sample xi based onHxi ( q ) as in Eq . ( 3 ) , where [ · ] is the Iverson bracket1 . p ( yi|xi ; q ) = ∑ ŷi∈Hxi ( q ) [ ŷi = yi ] |Hxi ( q ) | ( 3 ) Then , to quantify the uncertainty of the sample xi , the predictive uncertainty F ( xi ; q ) is defined using the empirical entropy as in Eq . ( 4 ) . Because the uncertainty is bounded , we add the standardization term δ to normalize the value to [ 0 , 1 ] . For k classes , δ is the maximum entropy when ∀jp ( j|xi ; q ) = 1/k . F ( xi ; q ) = − ( 1/δ ) k∑ j=1 p ( j|xi ; q ) log p ( j|xi ; q ) δ = − log ( 1/k ) ( 4 ) 1The Iverson bracket [ p ] returns 1 if p is true ; 0 otherwise . 3.2 SAMPLING PROBABILITY FOR MINI-BATCH CONSTRUCTION . To construct next mini-batch samples , we assign the sampling probability according to the predictive uncertainty in Definition 3.1 . Motivated by Loshchilov & Hutter ( 2016 ) , the sampling probability of a given sample xi is exponentially decayed with its predictive uncertainty F ( xi ; q ) . In detail , we adopt the quantization method ( Chen & Wornell , 2001 ) and use the quantization index to decay the sampling probability . The index is obtained by the simple quantizer Q in Eq . ( 5 ) , where ∆ is the quantization step size . Compared with the rank-based index ( Loshchilov & Hutter , 2016 ) , the quantization index is known to well reflect the difference in actual values ( Widrow et al. , 1996 ) . Q ( F ( xi ; q ) ) = d ( 1− F ( xi ; q ) ) /∆e , 0 ≤ F ( xi ; q ) ≤ 1 ( 5 ) In Eq . ( 5 ) , we set ∆ to be 1/N such that the index is bounded to N ( the total number of samples ) . Then , the sampling probability P ( xi|D ; se ) is defined as in Eq . ( 6 ) . The higher the predictive uncertainty , the smaller the quantization index . Therefore , a higher sampling probability is assigned for uncertain samples in Eq . ( 6 ) . P ( xi|D ; se ) = 1/ exp ( log ( se ) /N ) Q ( F ( xi ; q ) ) ∑N j=1 1/ exp ( log ( se ) /N ) Q ( F ( xj ; q ) ) ( 6 ) Meanwhile , it is known that using only some part of training data exacerbates the overfitting problem at a late stage of training ( Loshchilov & Hutter , 2016 ; Zhou & Bilmes , 2018 ) . Thus , to alleviate the problem , we include more training samples as the training progresses by exponentially decaying the selection pressure se as in Eq . ( 7 ) . At each epoch e from e0 to eend , the selection pressure se exponentially decreases from se0 to 1 . Because this technique gradually reduces the sampling probability gap between the most and the least uncertain samples , more diverse samples are selected for the next mini-batch at a later epoch . When the selection pressure se becomes 1 , the mini-batch samples are randomly chosen from the entire dataset . se = se0 ( exp ( log ( 1/se0 ) / ( eend − e0 ) ) ) e−e0 ( 7 ) 4 Recency Bias ALGORITHM Algorithm 1 Recency Bias Algorithm INPUT : D : data , epochs , b : batch size , q : window size , se0 : initial selection pressure , γ : warm-up OUTPUT : θt : model parameter 1 : t← 1 ; 2 : θt ← Initialize the model parameter ; 3 : for i = 1 to epochs do 4 : / * Sampling Probability Derivation * / 5 : if i > γ then 6 : se ← Decay_Selection_Pressure ( se0 , i ) ; / * Decaying se by Eq . ( 7 ) * / 7 : for m = 1 to N do / * Updating the index and the sampling probability in a batch * / 8 : q_dict [ xm ] = Q ( F ( xm ; q ) ) ; / * By Eq . ( 5 ) * / 9 : p_table← Compute_Prob ( q_dict , se ) ; / * By Eq . ( 6 ) * / 10 : / * Network Training * / 11 : for j = 1 to N/b do / * Mini-batch * / 12 : if i ≤ γ then / * Warm-up * / 13 : { ( x1 , y1 ) , . . . , ( xb , yb ) } ← Randomly select next mini-batch samples ; 14 : else / * Adaptive batch selection * / 15 : { ( x1 , y1 ) , . . . , ( xb , yb ) } ← Select next mini-batch samples based on p_table ; 16 : losses , labels← Inference_Step ( { ( x1 , y1 ) , . . . , ( xb , yb ) } , θt ) ; / * Forward * / 17 : θt+1 ← SGD_Step ( losses , θt ) ; / * Backward * / 18 : Update_Label_History ( labels ) ; / * By Definition 3.1 * / 19 : t← t+ 1 ; 20 : return θt ; Algorithm 1 describes the overall procedure of Recency Bias . The algorithm requires a warm-up period of γ epochs because the quantization index for each sample is not confirmed yet . During the warm-up period , which should be at least q epochs ( γ ≥ q ) to obtain the label history of size q , randomly selected mini-batch samples are used for the network update ( Lines 12–13 ) . After the warm-up period , the algorithm decays the selection pressure se and updates not only the quantization index but also the sampling probability in a batch at the beginning of each epoch ( Lines 4–9 ) . Subsequently , the uncertain samples are selected for the next mini-batch according to the updated sampling probability ( Line 14–15 ) , and then the label history is updated along with the network update ( Lines 16–19 ) . Overall , the key technical novelty of Recency Bias is to incorporate the notion of a sliding window ( Line 8 ) rather than a growing window into adaptive batch selection , thereby improving both training speed and generalization error . Time Complexity : The main “ additional ” cost of Recency Bias is the derivation of the sampling probability for each sample ( Lines 4–9 ) . Because only simple mathematical operations are needed per sample , its time complexity is linear to the number of samples ( i.e. , O ( N ) ) , which is negligible compared with that of the forward and backward steps of a complex network ( Lines 16–17 ) . Therefore , we contend that Recency Bias does not add the complexity of an underlying optimization algorithm .

Summary: This paper explores a well motivated but very heuristic idea for selecting the next samples to train on for training deep learning models. This method relies on looking at the uncertainty of predictions of in the recent history of statements and preferring those instances that have a predictive uncertainty over the recent predictions.  This allows the training method to train on instances that are neither too hard nor too easy and focus on reducing the uncertainty whenever it has the greatest potential gain to do so.