Boost Neural Networks by Checkpoints

1 Introduction . DNN ensembles often outperform individual networks in either accuracy or robustness [ Hansen and Salamon , 1990 , Zhou et al. , 2002 ] . Particularly , in the cases when memory is restricted , or complex models are difficult to deploy , ensembling light-weight networks is a good alternative to achieve the performance comparable with deep models . However , since training DNNs is computationally expensive , straightforwardly ensembling multiple networks is not acceptable in many real-world scenarios . A trick to avoid the exorbitant computational cost is to fully utilize the middle stages — or so-called checkpoints — in one training process , instead of training additional networks . Here we refer to this technique as Checkpoint Ensemble ( CPE ) . Despite its merit of no additional training computation , an obvious problem of CPE is that the checkpoints sampled from one training process are often very similar , which violates the consensus that we desire the base models in an ensemble are accurate but sufficiently diverse . To enhance the diversity , conventional ensembles often differ the base models from initialization , objective function , or hyperparameters [ Breiman , 1996 , Freund and Schapire , 1997 , Zhou et al. , 2002 ] , whilst the recent CPE methods try to achieve this goal by cyclical learning ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . rate scheduling , with the assumption that the high learning rates are able to force the model jumping out of the current local optima and visiting others [ Huang et al. , 2017a , Garipov et al. , 2018 , Zhang et al. , 2020 ] . However , this effect is not theoretically promised and the cyclically high learning rates may incur oscillation or even divergence when training budget is limited . In this paper , we are motivated to efficiently exploit the middle stages of neural network training process , obtaining significant performance improvement only at the cost of additional storage , meanwhile the convergence is promised . We design a novel boosting strategy to ensemble the checkpoints , named CBNN ( Checkpoint-Boosted Neural Network ) . In contrast to cyclical learning rate scheduling , CBNN encourages checkpoints diversity by adaptively reweighting training samples . We analyze its advantages in Section 4 from two aspects . Firstly , CBNN is theoretically proved to be able to reduce the exponential loss , which aims to accelerate model convergence even if the network does not have sufficiently low error rate . Secondly , reweighting training samples is equivalent to tuning the loss function . Therefore , each reweighting process affects the distribution of local minimum on the loss surface , and thereby changes the model ’ s optimization direction , which explains the high checkpoint diversity of CBNN . Moreover , the adaptability of CBNN makes it effective in tackling with imbalanced datasets ( datasets with imbalanced class distribution ) . This is due to CBNN ’ s ability to automatically allocate high weights to the “ minority-class ” samples , while most commonly used imbalanced learning methods require manually assigning sample weights [ Buda et al. , 2018 , Johnson and Khoshgoftaar , 2019 ] . 2 Related Work . In general , training a collection of neural networks with conventional ensemble strategies such as Bagging [ Breiman , 1996 ] and Boosting [ Freund and Schapire , 1997 ] improves the predictors on accuracy and generalization . For example , [ Moghimi et al. , 2016 ] proposed the BoostCNN algorithm , where a finite number of convolutional neural networks ( CNNs ) are trained based on a boosting strategy , achieving a superior performance than a single model . Several other strategies such as assigning different samples to the base models [ Zhou et al. , 2018 ] or forming ensembles with shared low-level layers of neural networks [ Zhu et al. , 2018 ] also combine the merits of DNNs and ensemble learning . However , these methods are computationally expensive . Alternatively , the “ implicit ensemble ” techniques [ Wan et al. , 2013 , Srivastava et al. , 2014 , Huang et al. , 2016 ] or adding random noise layers [ Liu et al. , 2018 ] avoid this problem . Such methods are sometimes seen as regularization approaches and can work in coordination with our proposed method . Recently , checkpoint ensemble has become increasingly popular as it improves the predictors “ for free ” [ Huang et al. , 2017a , Zhang et al. , 2020 ] . It was termed as “ Horizontal Voting ” in [ Xie et al. , 2013 ] , where the outputs of the checkpoints are straightforwardly ensembled as the final prediction . However , these checkpoints share a big intersection of misclassified samples which only bring very marginal improvements . Therefore , the necessary adjustments to the training process should be made to enhance this diversity . For instance , [ Huang et al. , 2017a ] proposed the Snapshot Ensemble method , adopting the cyclic cosine annealing learning rate to enforce the model jumping out of the current local optima . Similarly , another method FGE ( Fast Geometric Ensembling ) [ Garipov et al. , 2018 ] copies a trained model and further fine-tunes it with a cyclical learning rate , saving checkpoints and ensembling them with the trained model . More recently , [ Zhang et al. , 2020 ] proposed the Snapshot Boosting , where they modified the learning rate restarting rules and set different sample weights during each training stage to further enhance the diversity of checkpoints . Although there are weights of training samples in Snapshot Boosting , these weights are updated only after the learning rate is restarted and each update begins from the initialization . Such updates are not able to significantly affect the loss function , hence it is essentially still a supplementary trick for learning rate scheduling to enhance the diversity . In this paper , our proposed method aims to tune the loss surface during training by the adaptive update of sample weights , which induces the network to visit multiple different local optima . Our experimental results further demonstrate that our method , CBNN significantly promotes the diversity of checkpoints and thereby achieves the highest test accuracy on a number of benchmark datasets with different state-of-the-art DNNs . Algorithm 1 Checkpoint-Boosted Neural Networks Input : Number of training samples : n ; number of classes : k ; deviation rate : η ; total training iterations : T ; training iterations per checkpoint : t Require : Estimated weight of the trained model : λ0 Output : An ensemble of networks { G1 ( x ) , G2 ( x ) . . . } ( Note thatGj ( x ) ∈ Rk , Gyj ( x ) = 1 ifGj ( x ) predicts x belonging to the y-th class , otherwise Gyj ( x ) = 0 ) 1 : Initialize m← 1 // number of base models 2 : Uniformly initialize the sample weights : ωmi ← 1/n , i← 1 , ... , n 3 : while T −mt > 0 & ∑m−1 j=0 λj < 1/η do 4 : Train model for min ( t , T −mt ) iterations , get Gm ( x ) 5 : Calculate the weighted error rate on training set : em ← ∑n i=1 ωmiI ( G yi m ( xi ) = 0 ) 6 : Calculate the weight of m-th checkpoint : λm ← log ( ( 1− em ) /em ) + log ( k − 1 ) 7 : Update the sample weights : ωm+1 , i ← ωmiexp ( −ηλmGyim ( xi ) ) / ∑n i=1 ωmiexp ( −ηλmGyim ( xi ) ) 8 : Save the checkpoint model at current step . 9 : m← m+ 1 10 : end while 11 : Train model for max ( t , T − ( m− 1 ) t ) iterations , get Gm ( x ) . 12 : Execute step 5 and 6 to calculate λm . 13 : return G ( x ) = ∑m j=1 λjGj ( x ) / ∑m j=1 λj 3 Methodology . 3.1 Overview of Checkpoint-Boosted Neural Networks . CBNN aims to accelerate convergence and improve accuracy by saving and ensembling the middle stage models during neural networks training . Its ensemble scheme is inspired by boosting strategies [ Freund and Schapire , 1997 , Hastie et al. , 2009 , Mukherjee and Schapire , 2013 , Saberian and Vasconcelos , 2011 ] , whose main idea is to sequentially train a set of base models with weights for training samples . After one base model is trained , it enlarges the weights of the misclassified samples ( meanwhile , the weights of the correctly classified samples are reduced after re-normalization ) . Intuitively , this makes the next base model focus on the samples that were previously misclassified . In testing phase , the output of boosting is the weighted average of the base models ’ predictions , where the weights of the models depend on their error rates on the training set . We put the sample weights on the loss function , which is also reported in many cost-sensitive learning and imbalanced learning methods [ Buda et al. , 2018 , Johnson and Khoshgoftaar , 2019 ] . In general , given a DNN model p ∈ R|G| and training dataset D = { ( xi , yi ) } ( |D| = n , xi ∈ Rd , yi ∈ R ) where |G| is the number of model parameters and xi , yi denote the features and label of the i-th sample , the weighted loss is formulated as L = ∑n i=1 ωil ( yi , f ( xi ; p ) ) ∑n i=1 ωi + r ( p ) , ( 1 ) where l is a loss function ( e.g . cross-entropy ) , f ( xi ; p ) is the output of xi with the model parameters p , and r ( p ) is the regularization term . The next section introduces the training procedure of CBNN , including the weight update rules and ensemble approaches , which are also summarized in Algorithm 1 . 3.2 Training Procedure . We start training a network with initializing the sample weight vector as a uniform distribution : Ω1 = [ ω1,1 , ... , ω1 , i , ... , ω1 , n ] , ω1 , i = 1 n , ( 2 ) where n denotes the number of training samples . After every t iterations of training , we update the weights of training samples and save a checkpoint , where the weight of the m-th checkpoint is calculated by λm = log 1− em em + log ( k − 1 ) , ( 3 ) in which k is the number of classes , and em = n∑ i=1 ωmiI ( G yi m ( xi ) = 0 ) ( 4 ) denotes the weighted error rate . Note that Gm ( x ) ∈ Rk yields a one-hot prediction , i.e. , given the predicted probabilities of x , [ py ( x ) ] ky=1 , Gym ( x ) = { 1 , py ( x ) > py ′ ( x ) , ∀y′ 6= y , y′ ∈ [ 1 , k ] 0 , else . ( 5 ) Briefly , Gym ( x ) = 1 implies that the model Gm ( · ) correctly predict the sample x and Gym ( x ) = 0 implies the misclassification . According to Equation 3 , the weight of a checkpoint increases with the decrease of its error rate . The term log ( k − 1 ) may be confusing , which is also reported in the multi-class boosting algorithm SAMME [ Hastie et al. , 2009 ] . Intuitively , log ( k − 1 ) makes the weight of the base models λm positive , since the error rate in a k-classification task is no less than ( k − 1 ) /k ( random choice is able to obtain 1/k accuracy ) . In fact , the positive λm is essential for convergence , which will be further discussed in Section 4.1 . We then update the sample weights with the rule that the weight of the sample ( xi , yi ) increases if Gyim ( xi ) = 0 ( i.e. , it is misclassified ) and decrease if G yi m ( xi ) = 1 , which is in accordance with what we introduced in Section 3.1 . They are updated by ωm+1 , i = ωmi Zm exp ( −ηλmGyim ( xi ) ) , ( 6 ) where η is a positive hyperparameter named “ deviation rate ” and Zm = n∑ i=1 ωmiexp ( −ηλmGyim ( xi ) ) ( 7 ) is a normalization term to keep ∑n i=1 ωmi = 1 . It is very important to consider the tradeoff of the deviation rate η . Here we firstly present an intuitive explanation . In Equation 6 , a high deviation rate speeds up the weight update , which seems to help enhance the checkpoint diversity . However , this will lead to very large weights until some specific samples dominate the training set . From grid search , we find that setting η = 0.01 achieves significant performance improvement . In Section 4.1 , we are going to prove that CBNN reduces the exponential loss with the condition M∑ m=1 λm < 1/η , ( 8 ) i.e. , the sum of the weights of M − 1 checkpoints λm ( m = 1 , 2 , . . . , M − 1 ) and the finally trained model λM should be limited . To meet this condition , therefore , sometimes we need to limit the number of checkpoints . However , we can not acquire λM beforehand . A simple way to address this issue is to estimate the error rate of the finally trained model and calculate its weight ( denoted by λ0 ) according to Equation 3 . For example , if regard that the error rate of the network in a 100-class task is not less than 0.05 , we can estimate λ0 as log ( 0.95/0.05 ) + log ( 99 ) = 7.54 . Then , we stop saving checkpoints and updating the sample weights until ∑m j=0 λj > 1/η . The final classifier is an ensemble of the finally trained model GM ( X ) and M − 1 checkpoints Gm ( X ) , m = 1 , 2 , . . . , M − 1 : G ( x ) = M∑ m=1 λmGm ( x ) / M∑ m=1 λm . ( 9 ) For saving storage and test time , we can select part of the M − 1 checkpoints to ensemble , while selecting with equal interval is expected to achieve high diversity .

Summary: This paper proposed to ensemble the checkpoint models during the training to boost the performance of neural networks. To improve diversity over checkpoints, the proposed method changes the weights for each sample based on the mis-classification rate along the training process. When finally doing the ensemble over the checkpoints, a weight for each checkpoint based on mis-classification rate is also assigned. The experimental results show that comparing with existing checkpoint-based ensemble models and single model,  the proposed method has 1) better accuracy; 2) high diversity over different checkpoints; 3) low training cost.