Boost Neural Networks by Checkpoints

1 Introduction . DNN ensembles often outperform individual networks in either accuracy or robustness [ Hansen and Salamon , 1990 , Zhou et al. , 2002 ] . Particularly , in the cases when memory is restricted , or complex models are difficult to deploy , ensembling light-weight networks is a good alternative to achieve the performance comparable with deep models . However , since training DNNs is computationally expensive , straightforwardly ensembling multiple networks is not acceptable in many real-world scenarios . A trick to avoid the exorbitant computational cost is to fully utilize the middle stages — or so-called checkpoints — in one training process , instead of training additional networks . Here we refer to this technique as Checkpoint Ensemble ( CPE ) . Despite its merit of no additional training computation , an obvious problem of CPE is that the checkpoints sampled from one training process are often very similar , which violates the consensus that we desire the base models in an ensemble are accurate but sufficiently diverse . To enhance the diversity , conventional ensembles often differ the base models from initialization , objective function , or hyperparameters [ Breiman , 1996 , Freund and Schapire , 1997 , Zhou et al. , 2002 ] , whilst the recent CPE methods try to achieve this goal by cyclical learning ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . rate scheduling , with the assumption that the high learning rates are able to force the model jumping out of the current local optima and visiting others [ Huang et al. , 2017a , Garipov et al. , 2018 , Zhang et al. , 2020 ] . However , this effect is not theoretically promised and the cyclically high learning rates may incur oscillation or even divergence when training budget is limited . In this paper , we are motivated to efficiently exploit the middle stages of neural network training process , obtaining significant performance improvement only at the cost of additional storage , meanwhile the convergence is promised . We design a novel boosting strategy to ensemble the checkpoints , named CBNN ( Checkpoint-Boosted Neural Network ) . In contrast to cyclical learning rate scheduling , CBNN encourages checkpoints diversity by adaptively reweighting training samples . We analyze its advantages in Section 4 from two aspects . Firstly , CBNN is theoretically proved to be able to reduce the exponential loss , which aims to accelerate model convergence even if the network does not have sufficiently low error rate . Secondly , reweighting training samples is equivalent to tuning the loss function . Therefore , each reweighting process affects the distribution of local minimum on the loss surface , and thereby changes the model ’ s optimization direction , which explains the high checkpoint diversity of CBNN . Moreover , the adaptability of CBNN makes it effective in tackling with imbalanced datasets ( datasets with imbalanced class distribution ) . This is due to CBNN ’ s ability to automatically allocate high weights to the “ minority-class ” samples , while most commonly used imbalanced learning methods require manually assigning sample weights [ Buda et al. , 2018 , Johnson and Khoshgoftaar , 2019 ] . 2 Related Work . In general , training a collection of neural networks with conventional ensemble strategies such as Bagging [ Breiman , 1996 ] and Boosting [ Freund and Schapire , 1997 ] improves the predictors on accuracy and generalization . For example , [ Moghimi et al. , 2016 ] proposed the BoostCNN algorithm , where a finite number of convolutional neural networks ( CNNs ) are trained based on a boosting strategy , achieving a superior performance than a single model . Several other strategies such as assigning different samples to the base models [ Zhou et al. , 2018 ] or forming ensembles with shared low-level layers of neural networks [ Zhu et al. , 2018 ] also combine the merits of DNNs and ensemble learning . However , these methods are computationally expensive . Alternatively , the “ implicit ensemble ” techniques [ Wan et al. , 2013 , Srivastava et al. , 2014 , Huang et al. , 2016 ] or adding random noise layers [ Liu et al. , 2018 ] avoid this problem . Such methods are sometimes seen as regularization approaches and can work in coordination with our proposed method . Recently , checkpoint ensemble has become increasingly popular as it improves the predictors “ for free ” [ Huang et al. , 2017a , Zhang et al. , 2020 ] . It was termed as “ Horizontal Voting ” in [ Xie et al. , 2013 ] , where the outputs of the checkpoints are straightforwardly ensembled as the final prediction . However , these checkpoints share a big intersection of misclassified samples which only bring very marginal improvements . Therefore , the necessary adjustments to the training process should be made to enhance this diversity . For instance , [ Huang et al. , 2017a ] proposed the Snapshot Ensemble method , adopting the cyclic cosine annealing learning rate to enforce the model jumping out of the current local optima . Similarly , another method FGE ( Fast Geometric Ensembling ) [ Garipov et al. , 2018 ] copies a trained model and further fine-tunes it with a cyclical learning rate , saving checkpoints and ensembling them with the trained model . More recently , [ Zhang et al. , 2020 ] proposed the Snapshot Boosting , where they modified the learning rate restarting rules and set different sample weights during each training stage to further enhance the diversity of checkpoints . Although there are weights of training samples in Snapshot Boosting , these weights are updated only after the learning rate is restarted and each update begins from the initialization . Such updates are not able to significantly affect the loss function , hence it is essentially still a supplementary trick for learning rate scheduling to enhance the diversity . In this paper , our proposed method aims to tune the loss surface during training by the adaptive update of sample weights , which induces the network to visit multiple different local optima . Our experimental results further demonstrate that our method , CBNN significantly promotes the diversity of checkpoints and thereby achieves the highest test accuracy on a number of benchmark datasets with different state-of-the-art DNNs . Algorithm 1 Checkpoint-Boosted Neural Networks Input : Number of training samples : n ; number of classes : k ; deviation rate : η ; total training iterations : T ; training iterations per checkpoint : t Require : Estimated weight of the trained model : λ0 Output : An ensemble of networks { G1 ( x ) , G2 ( x ) . . . } ( Note thatGj ( x ) ∈ Rk , Gyj ( x ) = 1 ifGj ( x ) predicts x belonging to the y-th class , otherwise Gyj ( x ) = 0 ) 1 : Initialize m← 1 // number of base models 2 : Uniformly initialize the sample weights : ωmi ← 1/n , i← 1 , ... , n 3 : while T −mt > 0 & ∑m−1 j=0 λj < 1/η do 4 : Train model for min ( t , T −mt ) iterations , get Gm ( x ) 5 : Calculate the weighted error rate on training set : em ← ∑n i=1 ωmiI ( G yi m ( xi ) = 0 ) 6 : Calculate the weight of m-th checkpoint : λm ← log ( ( 1− em ) /em ) + log ( k − 1 ) 7 : Update the sample weights : ωm+1 , i ← ωmiexp ( −ηλmGyim ( xi ) ) / ∑n i=1 ωmiexp ( −ηλmGyim ( xi ) ) 8 : Save the checkpoint model at current step . 9 : m← m+ 1 10 : end while 11 : Train model for max ( t , T − ( m− 1 ) t ) iterations , get Gm ( x ) . 12 : Execute step 5 and 6 to calculate λm . 13 : return G ( x ) = ∑m j=1 λjGj ( x ) / ∑m j=1 λj 3 Methodology . 3.1 Overview of Checkpoint-Boosted Neural Networks . CBNN aims to accelerate convergence and improve accuracy by saving and ensembling the middle stage models during neural networks training . Its ensemble scheme is inspired by boosting strategies [ Freund and Schapire , 1997 , Hastie et al. , 2009 , Mukherjee and Schapire , 2013 , Saberian and Vasconcelos , 2011 ] , whose main idea is to sequentially train a set of base models with weights for training samples . After one base model is trained , it enlarges the weights of the misclassified samples ( meanwhile , the weights of the correctly classified samples are reduced after re-normalization ) . Intuitively , this makes the next base model focus on the samples that were previously misclassified . In testing phase , the output of boosting is the weighted average of the base models ’ predictions , where the weights of the models depend on their error rates on the training set . We put the sample weights on the loss function , which is also reported in many cost-sensitive learning and imbalanced learning methods [ Buda et al. , 2018 , Johnson and Khoshgoftaar , 2019 ] . In general , given a DNN model p ∈ R|G| and training dataset D = { ( xi , yi ) } ( |D| = n , xi ∈ Rd , yi ∈ R ) where |G| is the number of model parameters and xi , yi denote the features and label of the i-th sample , the weighted loss is formulated as L = ∑n i=1 ωil ( yi , f ( xi ; p ) ) ∑n i=1 ωi + r ( p ) , ( 1 ) where l is a loss function ( e.g . cross-entropy ) , f ( xi ; p ) is the output of xi with the model parameters p , and r ( p ) is the regularization term . The next section introduces the training procedure of CBNN , including the weight update rules and ensemble approaches , which are also summarized in Algorithm 1 . 3.2 Training Procedure . We start training a network with initializing the sample weight vector as a uniform distribution : Ω1 = [ ω1,1 , ... , ω1 , i , ... , ω1 , n ] , ω1 , i = 1 n , ( 2 ) where n denotes the number of training samples . After every t iterations of training , we update the weights of training samples and save a checkpoint , where the weight of the m-th checkpoint is calculated by λm = log 1− em em + log ( k − 1 ) , ( 3 ) in which k is the number of classes , and em = n∑ i=1 ωmiI ( G yi m ( xi ) = 0 ) ( 4 ) denotes the weighted error rate . Note that Gm ( x ) ∈ Rk yields a one-hot prediction , i.e. , given the predicted probabilities of x , [ py ( x ) ] ky=1 , Gym ( x ) = { 1 , py ( x ) > py ′ ( x ) , ∀y′ 6= y , y′ ∈ [ 1 , k ] 0 , else . ( 5 ) Briefly , Gym ( x ) = 1 implies that the model Gm ( · ) correctly predict the sample x and Gym ( x ) = 0 implies the misclassification . According to Equation 3 , the weight of a checkpoint increases with the decrease of its error rate . The term log ( k − 1 ) may be confusing , which is also reported in the multi-class boosting algorithm SAMME [ Hastie et al. , 2009 ] . Intuitively , log ( k − 1 ) makes the weight of the base models λm positive , since the error rate in a k-classification task is no less than ( k − 1 ) /k ( random choice is able to obtain 1/k accuracy ) . In fact , the positive λm is essential for convergence , which will be further discussed in Section 4.1 . We then update the sample weights with the rule that the weight of the sample ( xi , yi ) increases if Gyim ( xi ) = 0 ( i.e. , it is misclassified ) and decrease if G yi m ( xi ) = 1 , which is in accordance with what we introduced in Section 3.1 . They are updated by ωm+1 , i = ωmi Zm exp ( −ηλmGyim ( xi ) ) , ( 6 ) where η is a positive hyperparameter named “ deviation rate ” and Zm = n∑ i=1 ωmiexp ( −ηλmGyim ( xi ) ) ( 7 ) is a normalization term to keep ∑n i=1 ωmi = 1 . It is very important to consider the tradeoff of the deviation rate η . Here we firstly present an intuitive explanation . In Equation 6 , a high deviation rate speeds up the weight update , which seems to help enhance the checkpoint diversity . However , this will lead to very large weights until some specific samples dominate the training set . From grid search , we find that setting η = 0.01 achieves significant performance improvement . In Section 4.1 , we are going to prove that CBNN reduces the exponential loss with the condition M∑ m=1 λm < 1/η , ( 8 ) i.e. , the sum of the weights of M − 1 checkpoints λm ( m = 1 , 2 , . . . , M − 1 ) and the finally trained model λM should be limited . To meet this condition , therefore , sometimes we need to limit the number of checkpoints . However , we can not acquire λM beforehand . A simple way to address this issue is to estimate the error rate of the finally trained model and calculate its weight ( denoted by λ0 ) according to Equation 3 . For example , if regard that the error rate of the network in a 100-class task is not less than 0.05 , we can estimate λ0 as log ( 0.95/0.05 ) + log ( 99 ) = 7.54 . Then , we stop saving checkpoints and updating the sample weights until ∑m j=0 λj > 1/η . The final classifier is an ensemble of the finally trained model GM ( X ) and M − 1 checkpoints Gm ( X ) , m = 1 , 2 , . . . , M − 1 : G ( x ) = M∑ m=1 λmGm ( x ) / M∑ m=1 λm . ( 9 ) For saving storage and test time , we can select part of the M − 1 checkpoints to ensemble , while selecting with equal interval is expected to achieve high diversity .

Summary: The authors made a study on how to ensemble checkpoints with a boosting scheme, leading to faster convergence and more diversity. By adaptively reweight the training samples, Checkpoint-Boosted Neural Network (CBNN) can accelerate convergence and encourage diversity among individual ensemble members. The empirical evaluation demonstrated that under the same training budget, CBNN outperforms a number of baselines including Snapshot Ensemble. The authors also showed that it achieves great performance on imbalanced dataset because of CBNN's adaptability.