Boost Neural Networks by Checkpoints
Authors: Feng Wang
Guoyizhe Wei
Qiao Liu
Jinxiang Ou
Xian Wei
Hairong Lv
Venue: NeurIPS 2021 Poster (2021)
Keywords: Ensembles
Boosting
Deep Neural Networks

Abstract: Training multiple deep neural networks (DNNs) and averaging their outputs is a simple way to improve the predictive performance. Nevertheless, the multiplied training cost prevents this ensemble method to be practical and efficient. Several recent works attempt to save and ensemble the checkpoints of DNNs, which only requires the same computational cost as training a single network. However, these methods suffer from either marginal accuracy improvements due to the low diversity of checkpoints or high risk of divergence due to the cyclical learning rates they adopted. In this paper, we propose a novel method to ensemble the checkpoints, where a boosting scheme is utilized to accelerate model convergence and maximize the checkpoint diversity. We theoretically prove that it converges by reducing exponential loss. The empirical evaluation also indicates our proposed ensemble outperforms single model and existing ensembles in terms of accuracy and efficiency. With the same training budget, our method achieves 4.16% lower error on Cifar-100 and 6.96% on Tiny-ImageNet with ResNet-110 architecture. Moreover, the adaptive sample weights in our method make it an effective solution to address the imbalanced class distribution. In the experiments, it yields up to 5.02% higher accuracy over single EfficientNet-B0 on the imbalanced datasets.

Boost Neural Networks by Checkpoints
Feng Wang', Guoyizhe Wei', Qiao Liu", Jinxiang Ou', Xian Wei Hairong Ly
Department of Automation, Tsinghua University
'Department Statistics. Stanford University
Software Engineering Institute, East China Normal University
"Fuzhou Institute of Data Technology
(wangf19, wgyz19, ojx19) @mails tsinghua. edu. liugiao@stanford edu, xian. wei@tum. .m Ivhairong@tsinghua. edu. cn
Abstract
Training multiple deep neural networks (DNNs) and averaging their outputs a simple way to improve the predictive performance. Nevertheless, the multiplied training cost prevents this ensemble method be practical and efficient. Several recent works attempt to save and ensemble the checkpoints of DNNs, which only requires the same computational cost as training a single network. However, these methods suffer from either marginal accuracy improvements due to the low diversity of checkpoints or high risk of divergence due the cyclical learning rates they adopted. In this paper, we propose novel method to ensemble the checkpoints, where boosting scheme is utilized accelerate model convergence and maximize the checkpoint diversity. We theoretically prove that it converges by reducing exponential loss. The empirical evaluation also indicates our proposed ensemble outperforms single model and existing ensembles in terms of accuracy and efficiency. With the same training budget, our method achieves 4. 16% lower eπror on Cifar-100 and 6.96% on ImageNe with ResNet- 1 10 architecture. Moreover, the adaptive sample weights in our method make an effective solution address the imbalanced class distribution. In the experiments, it yields up 5.02% higher accuracy over single FfffccienNet-BO0 on the imbalanced datasets.
Introduction
DNN ensembles often outperform individual networks in either accuracy or robustness [Hansen and Salamon 1990 Zhou et al. 2002]. Particularly, in the cases when memory is restricted, or complex models are difficult deploy, ensembling light-weight networks is a good alternative achieve the performance comparable with deep models. However, since training DNNs is computationally expensive, straightforwardly ensembling multiple networks is not acceptable in many real-world scenarios.
trick avoid the exorbitant computational cost is fully utilize the middle stages so-called checkpoints - in one training process, instead of training additional networks. Here we refer to this technique Checkpoint Ensemble (CPR). Despite its merit of additional training computation, obvious problem of CPE is that the checkpoints sampled from one training process are often very similar, which violates the consensus that we desire the base models an ensemble are accurate but sufficiently diverse. To enhance the diversity, conventional ensembles often differ the base models from initialization, objective function, or hyperparameters [Breiman 1996, Freund and Schapire 1997 Zhou et al., 2002), whilst the recent CPE methods try to achieve this goal by cyclical learning
"Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

rate scheduling, with the assumption that the high learning rates are able to force the model jumping out of the current local optima and visiting others [Huang et al.) 2017a Garipov et al.) 2018 Zhang et al. 2020]. However, this effect is not theoretically promised and the eyclically high learning rates may meur oscillation or even divergence when training budget limited.
In this paper, we are motivated to efficiently exploit the middle stages of neural network training process, obtaining significant performance improvement only at the cost of additional storage, meanwhile the convergence is promised. We design a novel boosting strategy to ensemble the checkpoints, named CBNN Brteckpoint-Boostee Neural Network). In contrast to cyclical learning rate scheduling, CBNN encourages checkpoints diversity by adaptively reweighting training samples. We analyze its advantages in Section from two aspects. Firstly, CBNN is theoretically proved to be able to reduce the exponential loss, which aims to accelerate model convergence even if the network does not have sufficiently low error rate. Secondly, reweighting training samples is equivalent to tuning the loss function. Therefore, each reweighting process affects the distribution of local minimum on the loss surface, and thereby changes the model's optimization direction, which explains the high checkpoint diversity of CBNN.
Moreover, the adaptability of CBNN makes it effective in tackling with imbalanced datasets (datasets with imbalanced class distribution). This is due to CBNN's ability to automatically allocate high weights to the "minority-clase samples, while most commonly used imbalanced learning methods require manually assigning sample weights [Buda et al. 2018 Johnson and Khoshgoftaar 2019].
2
Related Work
In general, training a collection of neural networks with conventional ensemble strategies such as Bagging (Breiman 1996) and Boosting [Freund and Schapire 1997] improves the predictors on accuracy and generalization. For example, |Moghimi et 2016 proposed the BoostCNN algorithm, where a finite number of convolutional neural networks (CNNs) are trained based on boosting strategy, achieving a superior performance than a single model. Several other strategies such as assigning different samples to the base models (Zhou et al. 2018] or forming ensembles with shared low-level layers of neural networks [Zhu et al. 2018 also combine the merits of DNNs and ensemble learning.
However, these methods are computationally expensive. Alternatively, the "implicit ensemble" techniques (Wan et al.] 2013]] (Srivastava et al., 2014, Huang et al. 2016] or adding random noise layers (Liu et al. [018 avoid this problem. Such methods are sometimes seen as regularization approaches and can work in coordination with our proposed method. Recently, checkpoint ensemble has become increasingly popular as improves the predictors "for free" (Huang et al., 2017a| Zhang et al. 2020 It was termed as "Horizontal Voting" in [Xie et al., 2013] where the outputs of the checkpoints are straightforwardly ensembled as the final prediction. However, these checkpoints share a big intersection of misclassified samples which only bring very marginal improvements.
Therefore, the necessary adjustments to the training process should be made to enhance this diversity. For instance, (Huang et al.) 2017 [a] proposed the Snapshot Ensemble method, adopting the cyclic cosine annealing learning rate to enforce the model jumping out of the current local optima. Similarly, another method FGE (Fast Geometric Ensembling) [Garipov et al., 2018) copies trained model and further fine-tunes it with a cyclical learning rate, saving checkpoints and ensembling them with the trained model. More recently, (Zhang et al. [020] proposed the Snapshot Boosting, where they modified the learning rate restarting rules and set different sample weights during each training stage to further enhance the diversity of checkpoints. Although there are weights of training samples in Snapshot Boosting, these weights are updated only after the learning rate is restarted and each update begins from the initialization. Such updates are not able to significantly affect the loss function, hence it is essentially still a supplementary trick for learning rate scheduling to enhance the diversity.
In this paper, our proposed method aims to tune the loss surface during training by the adaptive update of sample weights, which induces the network to visit multiple different local optima. Our experimental results further demonstrate that our method, CBNN significantly promotes the diversity of checkpoints and thereby achieves the highest test accuracy on a number of benchmark datasets with different sfate-of-the-art DNNs.
2

---

Algorithm Checkpoint-Booster Neural Networks
Input: Number of training samples: n; number of classes: k; deviation rate: 77;
total training iterations: T; training iterations per checkpoint:
Require: Estimated weight of the trained model: Àu
Output: An ensemble of networks (Z), G2(z) (Note that Gz(œ) E C)(x) -
predicts x belonging to the y-th class, otherwise (a = 0)
I: Initialize < // number of base models
2: Uniformly initialize the sample weights:
Wmit < 1/n, i マ 1,...,
3: while 7' mf 0 & Σj= Ay < 1/n
4: Train model for min(l. T ml) iterations, get Gm (a:)
5: Calculate the weighted enror rate on training set:
6m <- Σi= Wms (G (2:)) = 0)
6:
Calculate the weight of m-th checkpoint:
Am f- (1-mm) + log(k - 1)
7: Update the sample weights:
Wm+1.i f- expexp (-rjAm (z;)) Ei= Wmsexp (-riAm Gh (x:))
8: Save the checkpoint model at current step.
9: m f m
10: end while
11: Train model for (1, T - (m - 1)t) iterations, get Gm (z).
12: Execute step 5 and 6 calculate Am-
13: return G(z)) = Σド= (r)) >
3 Methodology
3.1
Overview of Checkpoint- -Boosted Neural Networks
CBNN aims accelerate convergence and improve accuracy by saving and ensembling the middle stage models during neural networks training. Its ensemble scheme is inspired by boosting strategies
[Freund and Schapire, 1997 Hastie et al., 2009 Mukherjee and Schapire 2013, Saberian and Vasconcelos] 2011]. whose main idea is to sequentially train a set of base models with weights for training samples. Áfter one base model is trained, it enlarges the weights of the misclassified samples (meanwhile, the weights of the correctly classified samples are reduced after re-normalization) Intuitively, this makes the next base model focus on the samples that were previously misclassified. In testing phase, the output of boosting is the weighted average of the base models' predictions, where the weights of the models depend on their error rates the training set.
We put the sample weights on the loss function, which is also reported many cost-sensitive learning and imbalanced learning methods [Buda et al. 2018 Johnson and Khoshgoftaar 2019]. In general, given DNN model IR [G] and training dataset D {(x;, y1) (D) = 2; R yi I) where |G] the number of weighted loss is formulated as
model parameters and a4, yt denote the features and label of the i-th sample, the
C En (oll(ifflzzss r(p),
El Wi
where is loss function (e.g. cross-entropy), f(x is the output of x; with the model parameters and r(p) is the regularization term.
The next section introduces the training procedure of CBNN, including the weight update rules and ensemble approaches, which are also summarized in Algorithm
3.2 Training Procedure
We start training a network with initializing the sample weight vector uniform distribution:
Ω = [W Iw.,,.... Wi ..., wi,n), W1,i n
3

---

where n denotes the number of training samples. After every t iterations of training, we update the weights of training samples and save a checkpoint, where the weight of the m-th checkpoint calculated by
1- - em
Am
log
log(k: 1),
(3)
Cm
in which k is the number of classes, and
cm Σ Wmil (G = 0)
(4)
denotes the weighted enror rate. Note that (m) E R yields one-hot prediction, i.e. given the predicted probabilities of x, ip" (x)]
Gh (2) - {
1, >p(()) > pd (I)), Vy vV.0" E [1,
(5)
clse
Briefly, G% (x) 1 implies that the model Gm (-) correctly predict the sample and G% (x) 0 implies the misclassification. According to Equation 3 the weight of a checkpoint increases with the decrease of its error rate. The term log(k 1) may be confusing. which is also reported in the multi-class boosting algorithm SAMME [Hastie et al., 2009]. Intuitively, log(k - 1) makes the weight of the base models A.. positive, since the error rate in k-classification task is less than (I: - 1)/k (random choice is able to obtain 1 1/k accuracy). In fact, the positive Am essential for convergence, which will be further discussed in Section 4.1
We then update the sample weights with the rule that the weight of the sample (æ;, wi) increases G% (1;) = 0 (i.e.. it is misclassified) and decrease if G (zx) 1, which is in accordance with what we introduced in Section 3.1. They are updated by
Wmi Zm
Wm-1.
exp (-n (H()))
(6)
where 1) is a positive hyperparameter named "deviation rate" and
Zm
Wms (-1AAm G (z:))
(7)
a normalization term to keep Σ Wmi = 1. very important to consider the tradeoff of the deviation rate y). Here we firstly present an intuitive explanation. In Equation 6, a high deviation rate speeds up the weight update, which seems to help enhance the checkpoint diversity. However, this will lead to very large weights until some specific samples dominate the training set. From grid search, we find that setting = 0.01 achieves significant performance improvement.
In Section 4.1
we are going to prove that CBNN reduces the exponential loss with the condition
M
Σ
Am < 1/11
(8)
i.e., the sum of the weights of M - 1 checkpoints Am (m = 1,2,., M M-1 1) and the finally trained model das should be limited. To meet this condition, therefore, sometimes we need to limit the number of checkpoints. However, we cannot acquire AM beforehand. A simple way to address this issue to estimate the error rate of the finally trained model and calculate its weight (denoted by An) according to Equation 3 For example, if regard that the error rate of the network in 100-class task not less than 0.05, we can estimate Ån as (0.(0.95/0.05. + log(99) = 7.54. Then, we stop saving checkpoints and updating the sample weights until Σ A 1/n.
The final classifier is an ensemble of the finally trained model GM((X) and M - 1 checkpoints (X), m 1,,,..., 1:
M
G(x) Σ Am Gm())) (x)/
Σ Am-
(9)
For saving storage and test time, we can select part of the M checkpoints to ensemble, while selecting with equal interval is expected to achieve high diversity.

---

Theoretical Analyses
4.1 Convergence of CBNN
According to Equation the output of the final model, G(r), is the weighted average output of the base model set [Gm (2 m= where (z(z) [0, 1] denotes the predicted probability that the sample a belongs to the y-th category. Therefore, we desire as high G (x:) as possible on each sample I;
and the exponential loss
Lep 1 Σ exp( exp(-G"Mi (x,)
(10)
often used to measure the error of the model. We analyze the convergence our method by presenting the upper bound of this exponential loss in CBNN and proving that this bound decreases as the number of base models, M, increases.
Theorem 1. The exponential loss Leap in CBNN is bounded above by Lezp 1I Zm. Proof.
M Σ m=1
M Am G% (z:)/
M Am G (Ii) (#a))
Leep
exp
Σ
Am
exp
-1
i=1
m=1
m=1
M
exp (-n)mm Gth (I:))
elppexp (-nd (x))
II
exp 1m G (2
n
m=1
M
Zz
e2p (
exp (-n 4% (ri))
1=1
m=3
.....MM-
wM,eep (MM.exp(-TAMMM(
II 7m
1=1
m=1
(11)
Theorem 2. The upper bound of Lezp in CBNN decreases as M increases.
Proof.
Zm
Σ empexp (-1iAm Gt (z+))
Σ
Wmmexp ( (-ndm) +
Wmi
(12)
G (m.)==1
G (x:)==0
=11 em exp (-T/Am) Em <1
Note that, according to Formula 12 Zm < when Am is positive, which explains why we add
log(k: 1) in Equation 3 Theorem and 2 demonstrate exponential decrease Leep bound as we save more base models. Moreover, according to Formula T2 even a moderately low error rate e. of the base model can lead to significant decrease the Lezp upper bound.
4.2 Diversity of Checkpoints
We also demonstrate the superior performance of CBNN from the viewpoint of checkpoint diversity. As reported in the prior works [Auer et al.) 1996 Dauphin et al. [2044), there is a large number of local optima and saddle points on DNN loss surface. However, even the nearby optima may yield a small intersection of misclassified samples from given dataset [Garipov al. 2018).
When we enlarge the weights of the misclassified samples corresponding to a local optimum, the loss L of this optimum increased, whereas it may be only slightly changed (or even decreased) around the other optima. Based on this observation, CBNN adopts loss function in the form of Equation [ and tunes the loss surface adaptively by increasing the weights of the misclassified samples during training. This process enlarges the loss of the current local optimal domain, making it casier for the model to jump out of the current optimum and visit more other optima. In other
5

---

530
50
1.00
272
F
60
-2
-10
-9
50-di epoch
100-ch epoch
150-th spoch
Figure 1: The moving Le-regularized cross-entropy of an EfficientNet-BOO on Cifar-100 in CBNN, where P1. P2, and p3 are three local optima the model. The network tends to converge to pi the beginning (the 50-th epoch, left panel). As the loss surface moves, it converges to p2 (the 100-th epoch, middle panel) and finally to pa (the 150-th epoch, right panel). As seen the moving loss surface makes checkpoints accurate but diverse, hence enhances their value to be ensembled.
words, the checkpoints of this model have higher diversity and ensembling them is expected to yield considerable performance improvements.
Fig T visualizes the Legrelulari cross-entropy loss on Cifar-100 (Krizhevsky al. 2009) of an EfficientNet-B0 (Tan and Le 2019) trained with CBNN strategy, where the three panels respectively display the loss surface at the 50, 100, 150-th epoch in a two-dimensional subspace view is shown that the loss surface has significantly changed during the training process, which thereby induces the network to converge to different local optima at different training stages. Specifically, the model tends to converge to one local optimum the 50-th epoch (p1, Fig Left), whereas the loss around p1 is gradually increased and the model is then forced to converge to Pa at the 100-th epoch (Fig I Middle). As the loss surface is further modified, the model leaves pa and finally converges to Ps (Fig) Right).
5 Experiments
We compare the effectiveness of CBNN with competitive baselines in this section. All the experiments are conducted on four benchmark datasets: Cifar-10, Cifar-100 [Krizhessky et al. 2009). Tiny ImageNet [Le and Yang 2015), and ImageNet ILSVRC 2012
[Deng et al. 2009
5.1 Experiment Setup
DNN Architectures. We test the ensemble methods the three commonly used network architec-
tures, including ResNet [He et al. 2016), DenseNet (Huang et al. [0017)] and EfficientNet [Tan and
L0190 We adopt an 110-layer ResNet and 40-layer DenseNet introduced in (He et al.) [016 and [Huang et al. (0176), with 32x input size. For EfficientNet, we use EfficientNet-B0 and B3 with
images resized to 224 224 pixels. Despite EfficientNet-B3 suggested larger image size (Tan and Le) 2019). 224 would be sufficient to achieve high performance while reducing memory footprint and computational cost. All the models are trained with 0.2 dropout rate (0.3 for EfficientNet-B3) and images are augmented by AutoAugment (Cubuk et al. 2019
Baseline Methods. Firstly, we train Single Model (referred to as Single) as baseline, adopting a standard decaying learning rate that is initialized to 0.05 and drops by 96% every two epochs with five epochs warmup |Gotmare et al. 2018). Here compare CBNN with state-of-tt of-te-of-the-att -tth--rtt art checkpoint ensemble models, including Snapshot Ensemble (SSE) [Huang et al. [0177]] Snapshot Boosting (SSB) [Zhang al., 2020), and Fast Geometrie Ensemble (FGE) Garipov et al. 2018). In Snapshot Ensemble, the learning rate scheduling rules follow (Hung et al. 201 7a and we set 0.2, which achieves better performance in experiments. Similarly, we r 0.1, 72 0.5.
"p,, P2, Ps [l (IG the number model parameters) are parameter vectors the model iterated to 50, 100 and 150 epochs, respectively. We adopt similar aftine transformation as [Garipov et al. 22018). setting = P2. v=(p1 Pz) < P1 P2,Ps p2 > /llps pzll2 (ps pz). and a point (z, y) in each panel denotes the vector p p2 u/ u/|lu] l/llol/

---

Table I: Error rates (%) on the four datasets with different ensemble methods. We repeat these experiments five times to get the average test error and standard deviation. Note that we omit the deviations of Cifar- 10 since they are relatively low and similar. The best results of each model and dataset are bolded.
Model
Method
Cifar-10
Cifar-100
Tiny-ImageNet
ImageNet
Single
5.63
27.67 + 0.25
45.10 + 0.36
SSE
5.41
24.27 + 0.26
39.08 + 0.34
ResNet-110
SSB
5.61
27.09 + 0.35
43.68 + 0.49
FGE
5.50
26.21 + 0.23
41.16 0.31
CBNN
5.25
23.51 + 0.20
38.14 + 0.22
Single
5.13
23.15 + 0.22
38.24 + 0.34
SSE
4.78
21.45 + 0.29
36.35 + 0.30
DenseNet-40
SSB
5.13
22.59 + 0.31
37.99 = 0.44
FGE
4.82
22.07 + 0.28
37.72 + 0.27
CBNN
4.78
20.66 + 0.17
34.28 + 0.20
Single
3.88
20.87 + 0.25
34.50 0.35
23.70
SSE
3.56
19.70 + 0.25
33.02
23.51
EfficientNet-BOO
SSB
3.87
20.12 + 0.41
34.02
23.75
FGE
3.68
19.58 + 0.24
3.37
23.43
CBNN
3.37
18.06 + 0.20
31.16 0.26
22.69 0.19
Single
3,45
17.28 + 0.22
31.86 0.30
1 0.15
SSE
3.30
16.84 + 0.29
31.01
19.16 0.22
EfficientNet-B3
SSB
3.43
17.25 + 0.32
3.44
19.59 0.34
FGE
3.27
16.96 + 0.24
30.97
18.68 0.20
CBNN
3.28
16.07 + 0.13
29.85 + 0.23
17.55 = 0.14
P1 = 2 and p2 = for Snapshot Boosting. Fast Geometric Ensemble (FGE), we train the first model with standard learning rate (the same Single Model) and the other models with the settings according to [Garipov et al. 2018). and set 0₁ = 5 10 -2, a2 10 to all the datasets and DNN architectures. Our method, CBNN adopts the learning rate used in training Single Model as well, and setting 7 0.01.
Training Budget. Checkpoint ensembles are designed to save training costs, therefore, to fully evaluate their effects, the networks should be trained for relatively fewer epochs. In our experiments. we train the DNNs from scratch for 200 epochs on Cifar- 10, Cifar- 100, Tiny-ImageN epochs on ImageNet. We save six checkpoint models for SSE and FGE, however, considering large scale and complexity of ImageNet, saving three base models is optimal for SSE and FGE since more learning rate cycles may incur divergence. This number of SSB automatically set by itself. Although method supports greater number of base models which leads to higher accuracy, for clear comparison, we only select the same number checkpoints as SSE and FGE.
and 300
5.2 Evaluation of CBNN
The evaluation results are summarized Table where all the methods listed are at the same training cost. It seen that in most cases, CBNN achieves the lowest error rate on the four tested datasets with different DNN architectures. Our method achieves a significant error reduction on Cifar- 100 and ImageNe.. For instance, with EfficientNet-Be as the base model, using CBNN decreases the mean error rate by 2.8 1.81% and 3.34%, on Cifar- 100 and Tiny-ImageNet, respectively. In the experiments, networks tend converge with lower speed ImageNet mainly due to the higher level of complexity and larger number training images in this dataset. Although we reduce the learning rate cycles of SSE and FGE (SSB automatically restarts the learning rate), these ensemble methods only slightly increase the accuracy on ImageNet with EfficientNet-B3.
We quantitatively evaluate the efficiency of ensemble methods by comparing the GPU hours they take to achieve certain test accuracy for given DNN and dataset. For the checkpoint ensemble methods, the test accuracy calculated by the ensemble of current model and saved checkpoints. For

---

Table 2: GPU hours required for EfficientNet-BO to reach certain test accuracy. We repeat these experiments five times get average GPU hours and standard deviation. Note that we omit the deviations of Cifar-10600% since they are relatively low and similar. The best results are bolded.
Cifar-10 10
Cifar-100
Method
60%
80%
90%
40%
60%
70%
Single
0.38
0.78 + 0.12
5.05
0.55
0.91 + 0.08
2.71 0.12
7.47 + 0.79
SSE
0.38
0.75 + 0.10
3,92 + 0.21
0.87 0.08
2
5.16
SSB
0.39
0.76 0.10
4.22 - 0.38
0.99
2.62 = 0.12
5.67 - 0.30
PE 0.72
2.13 0.34
10.13 + 0.78
2.11 0
7.78 1.23
17.32
+ 1.30
CBNN
0.36
0.68
0.07
3.62 0.15
0.85 0.07
2.33 0.10
4.82 0.22
example, if we save a checkpoint every 5,000 iterations, the test accuracy of the 12,000-th iteration is calculated by the averaged outputs of the 5,000, 10,000 and 12,000-th iteration. Additionally, we train four models parallel with different random initialization and add up their training time to evaluate the efficiency of conventional ensembles (referred to as Parallel Ensemble, PE). Table 2 summarizes the time consumption of different methods on Nvidia Tesla P40 GPUs, where the checkpoint ensembles are shown to be more efficient. In particular, CBNN reaches 0.7 test accuracy on Cifar- 100 costing only 64.5% training time of the single model. Although the conventional ensembles are able to improve accuracy, in our experiments, the parallel ensemble method often results in 2-3 times training overhead than single model to achieve the same accuracy.
5.3 Diversity of Base Models
0.68
6.08
0.00
3.68
0.00
0.20
0.6G
6.66
0.66
0.66
0.06
2.2
2.2
Majonit Classes
Minesily Clasos
0.75
0.55
0.75
0.90
0.00
5.00
0.27
5.45
1.2
0.68
4.75
0.00
0.49
3.4
0.09
2.93
0.08
0.75
6.70
0.22
0.
0.80
0.57
0.50
6.9
6.9
0.72
0.75
0.74
0.44
6.67
0.26
0.8
60
(a)
(b)
Figure 2: Experimental results of diversity and imbalanced datasets. (a): Pairwise correlation between the softmax outputs of any two checkpoint models of EfficientNet-BO on Cifar- Eifar-100 testing set. Left: CBNN. Right: Snapshot Ensemble. (b): Average weights over classes on imbalanced Cifar- 100 (randomly select 20 classes and remove 450 training images from each of them). The total 100 points represent the 100 classes in Cifar- 100 and the average weight of training samples in each class is measured by y-axis. Note that the average value of the red points (red dash line) is significantly higher than that in dark blue (dark blue dash line).
To measure the diversity of the base models, we compute the pairwise correlation between the softmax outputs of any two checkpoints and then compare our method with Snapshot Ensemble. The results on Cifar- 100 with EfficientNet-BOO are shown in Fig 2(a) where the correlation values of CBNN (the left panel) are significantly lower than that of Snapshot Ensemble (the right panel). These lower correlation values CBNN indicate that any two checkpoints predict with a smaller intersection of misclassified samples, which in other words implies higher diversity.
The high diversity, as well the visualization the moving loss surface in Fig are explained based on the following two aspects: (1) As it is introduced in Section 4.2 the weight update processes in our method change the shape of the loss surface, thus inducing the model to visit more nearby local optima; (2) We optimize the model in a mini-batch manner, where it takes a large step size of gradient decent in training the batch that contains high weighted samples, which increases the probability of the model to jump out of local optima.
8

---

5.4 Effect on Imbalanced Datasets
Table 3: Error rates (%) on imbalanced Cifar-100 and Imag--m (randomly selected 20% classes and removed 90% training images from each of them). tsh and oversample denote Thresholding and Oversampling introduced in section 5.4 We repeat these experiments five times to get average values and present standard deviations of oversampling since this method introduces higher randomness. The best results of each model and dataset are bolded.
Cifar-100
Imageeta
Model
Method
original
tsh
oversample
original
tsh
oversample
Single
29.19
26.93
27.55 + 0.30
45.25
42.58
43.12 0.67
SSE
28.27
25.88
26.14 + 0.42
43.79
41.26
42.06 + 0.65
Efficient Net-BO
SSB
28.66
26.51
27.18 + 0.33
44.98
42.47
43.13 0.69
FGE
28.39
26.43
26.83 + 0.29
43.77
41.03
41.78 + 0.55
CBNN
24.39
40.23
Single
27.08
25.01
25.86 + 0.30
43.34
41.33
42.01 + 0.60
SSE
26.49
24.56
25.11 + 0.33
42.26
40.47
40.36 0.45
Efficient
SSB
26.78
24.89
25,46 + 0.39
42.99
41.30
41.76 +
Net-B3
FGE
26.23
24.16
24.86 + 0.34
42.58
41.29
41.12 + 0.48
CBNN
22.98
39.02
The imbalanced class distribution often negatively impacts the performance of classification models. There are many existing methods tackling with this issue. As (Buda et al. 2018 claimed, most of these methods are mathematically equivalent. Thus, we implement two effective and commonly used approaches introduced in (Buda et al. 2018). Random Minority Oversampling and Thresholding with Prior Class Probabilities, as baselines.
Further, we construct "step imbalance" datasets based on C0f and Tiny-ImageNet in the way introduced in [Buda et al., 2018 i.e., we randomly select kmin of the total i classes as minority classes, setting μ kmin/ / k.
Tmmnn /nmin where Mmazz Amin denote the number of training samples majority and minority classes. We then randomly subsample the selected minority classes to match Here we set μ 0.2, p 10.
Error rates on the imbalanced datasets summarized in Table 3 where it is seen that the single model and the prior checkpoint ensembles suffer from significant eтror increase relative to training with the entire dataset (See Table D]. even if Oversampling or Thresholding are adopted help rebalancing the dataset. In contrast, the imbalance processing only leads to a relatively low reduction on CBNN accuracy.
To further study how the sample weights are adapted to the imbalanced datasets, we obtain the average weights in each class after the CBNN model is trained. As is seen in 2(b) the average weights of the minority classes (the red points) are almost at a higher level than the majority (the dark blue points), which indicates that CBNN can adaptively higher weights for the minority classes to balance the dataset.
6
Conclusion
Proper ensemble strategies improve DNNs in both accuracy and robustness. However, the high training cost of DNNs urges the ensembles to be sufficiently efficient for practical use. Checkpoint Ensembles often outperform the single model by exploiting the value of middle stage models during training, which do not require additional training cost. Our new proposed checkpoint ensemble method, CBNN, theoretically proven to reduce the exponential loss and shown yield more diverse base models, compared to the existing methods. A series of experiments demonstrate the superior performance CBNN in both accuracy and efficiency. In particular, when training DNNs on a large scale dataset with limited computation budget, the prior checkpoint ensemble methods based on cyclical learning rate scheduling result in poor performance (e.g. training EfficientNet-B on ImageNet in Table Moreover, imbalanced datascts, CBNN is able to automatically rebalance

---

the class distribution by adaptive sample weights, which outperforms the commonly used methods with manually rebalancing approaches.
Acknowledgement
This work was supported by the National Nature Science Foundation of China under Grant No. 42050101 and U1736210.
References
Peter Auer, Mark Herbster, and Manfred KK Warmuth. Exponentially many local minima for single neurons. In Advances in neural information processing systems, pages 316-322, 1996.
Ieo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-259, 2018.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 113-123, 2019.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in neural information processing systems, 27:2933-2941 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee, 2009.
Yoav Freund and Robert E Schapire. A decision-theorrticc generalization on-line learning and an application to boosting. Journal of computer and system sciences, 55(1): 19--199, 1997.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems, pages 8799, 2018
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristies: Learning rate restarts, warmup and distillation. International Conference on Learning Representationss 2018.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 12(10)9933-001,, 1990.
Trevor Hastie, Saharon Rosset, Ji Zhu, and Hui Zou. Multi-class adaboost. Statistics and its Interfuce, 2(3):349-360, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pages 646 661. Springer, 2016.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get m for free. International Conference Learning Representations, 2017a.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4700-4708, 2017b.
10

---

Justin M Johnson and Taghi M Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):1-54, 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7, 2015.
Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference Computer Vision (ECCV), pages 369-385, 2018.
Mohammad Moghimi, Serge J Belongie, Mohammad J Saberian, Jian Yang, Nuno Vasconcelos, and Li-Jia Li. Boosted convolutional neural networks. In BMVC, volume page 2016.
Indraneel Mukherjee and Robert Schapire. A theory multiclass boosting. Journal of Machine Learning Research, 14(Febb)437-497, 2013.
Mohammad J Saberian and Nuno Vasconcelos. Multiclass boosting: Theory and algorithms. In NIPS, pages 22322113, 2011.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Hyy Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1229-198, 2014.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114, 2019.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058-1066. 2013.
Jingjing Xie, Bing Xu, and Zhang Chuang. Horizontal and vertical ensemble with deep representation for classification. arXiv preprint arXiv:1106.2755: 27599 2013.
Wentao Zhang, Jiawei Jiang, Yingxia Shao, and Bin Cui. Snapshot boosting: fast ensemble framework for deep neural networks. Science China Information Sciences, 63(1):112102 2020.
Tianyi Zhou, Shengjie Wang, and Jeff A Bilmes. Diverse ensemble evolution: Curriculum data-model marriage. Advances in Neural Information Processing Systems, pages 5905-5916, 2018
Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: many could better than all. Artificial intelligence, 137(1-2) (3(1-2)):399-63. 2002.
Xiatian Zhu, Shaogang Gong, al. Knowledge distillation by on-the-fly native ensemble. In Advances in neural information processing systems, pages 7517-7527, 2018.


Reviews:
[{"review_id": "tEZWN6N9a30", "reviewer": "Reviewer_Ca6w", "summary": "This paper proposed to ensemble the checkpoint models during the training to boost the performance of neural networks. To improve diversity over checkpoints, the proposed method changes the weights for each sample based on the mis-classification rate along the training process. When finally doing the ensemble over the checkpoints, a weight for each checkpoint based on mis-classification rate is also assigned. The experimental results show that comparing with existing checkpoint-based ensemble models and single model,  the proposed method has 1) better accuracy; 2) high diversity over different checkpoints; 3) low training cost. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-written and easy to follow. The proposed method is straightforward and should be novel. The experimental results look promising on both performance and training cost. \nBut I have a few questions and concerns.\n\n1. Is there a reason why ImageNet with Densenet and Resnet are omitted in Tab. 1?\n2. As SSE is the second-best performance, I am wondering if the authors tried to adopts cyclic learning rate schedule as SSE does to improve CBNN? \n3. The theoretical analysis Theorem 1 and Theorem 2 seem trivial and the conclusion is a little bit vacuous. In particular, the conclusion is that a upper bound of the exponential loss is decreasing with more checkpoints. The upper bound can be very loose, which can lead to a vacuous conclusion. It would be better to calculate the tightness of the upper bound empirically. \n4. Although random labeling has the error rate (k-1)/k, the worst case error rate can be worse than this. Therefore, $\\lambda_m > 0$ needs to be assumed.  \n\nOverall, as the experimental results look promising, I think this paper would be interesting to the DL community. However, due the above concerns and lack of the limitation and societal impact discussion, I think the current version of the paper is not ready for publication. I am open to change the score based on the authors' rebuttal. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't think the paper has explicitly mentioned limits of the methods. \nThe paper showed better performance on imbalanced datasets in Sec. 5.4. Since imbalanceness is common in datasets with minority groups, it would be good if the authors can discuss the impact of their method in this aspect. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "RMbAyAwVkpC", "reviewer": "Reviewer_Bmco", "summary": "This paper proposed a novel method to obtain a diverse set of checkpoint models for efficient ensembles. Through the boosting inspired sampling weighting scheme, the proposed method not only facilitates the convergence of the model but also produces checkpoints with diverse predictions with an enhanced ensemble performance. The authors of the paper theoretically prove the convergence of the proposed method and empirically demonstrate that the method is capable of producing ensembles with good performance given a fixed computational budget.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "All in all, this paper is well written and easy to follow. The proposed method seems technically sound. While the idea of sample reweighting is definitely not new in general in the context of deep learning (for example, see [1][2]), I personally find its connection to boosting and also the use of it as a way to promote model diversity among different checkpoints to be very interesting. Specifically, despite the reduction in training time computational cost, most of the previously proposed methods for \"checkpoints ensembling\" still perform poorly when compared to standard ensemble strategies likely due to low diversity among the models. The authors of the paper directly tackle this problem by utilizing a sample reweighting approach to introduce more diversity to the checkpoints. The authors of the paper also conduct a good amount of experiments and ablation to demonstrate the effectiveness of the proposed method. \n\nHere are some shortcomings/suggestions of the paper. Firstly, in my opinion, as an illustrative purpose, It might be worthwhile to see how big the gap is in terms of accuracy and diversity when compared to an explicit ensemble. In addition, it might be interesting to also give a more careful analysis (in addition to the ones present in section 5.3) on why the proposed method outperforms previously proposed checkpoint ensemble methods. Is it because of larger diversity or better individual model performance? I am also not entirely sure how relevant/helpful the theoretical analysis is. One of the main contributions of the paper seems to be about enhanced diversity with the proposed strategy, and yet there isn't any diversity involved in the section. It would be much more interesting if the authors of the paper can theoretically demonstrate the diversity aspect of the proposed method? Lastly, it seems that the proposed method also introduces additional hyper-parameters like the deviation rate. How sensitive is the performance of the algorithm to the choice of hyper-parameters, and how easy is it to tune for it? \n\nOverall I think this is a good submission and recommend accepting it.\n\n[1] Chang, Haw-Shiuan, Erik Learned-Miller, and Andrew McCallum. \"Active bias: Training more accurate neural networks by emphasizing high variance samples.\" Advances in Neural Information Processing Systems 30 (2017): 1002-1012.\n\n[2] Ren, Mengye, et al. \"Learning to reweight examples for robust deep learning.\" International Conference on Machine Learning. PMLR, 2018.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "Bn5RqI6zDyU", "reviewer": "Reviewer_v3ks", "summary": "The authors made a study on how to ensemble checkpoints with a boosting scheme, leading to faster convergence and more diversity. By adaptively reweight the training samples, Checkpoint-Boosted Neural Network (CBNN) can accelerate convergence and encourage diversity among individual ensemble members. The empirical evaluation demonstrated that under the same training budget, CBNN outperforms a number of baselines including Snapshot Ensemble. The authors also showed that it achieves great performance on imbalanced dataset because of CBNN's adaptability.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Snapshot Ensemble requires a modification to the learning rate scheme. This limits its application when cylinder learning rate scheme is not desirable. CBNN does not require any strange tweak to the current learning algorithm, so it has a much wider application. The authors provide convergence analysis, leading to a theoretically grounded algorithm. The baselines this paper compared to are competitive, including recent advances such as Snapshot Ensemble and FGE. CBNN outperforms these competitive baselines on a variety of network architectures and vision datasets in terms of the accuracy measure. Additionally, the diversity analysis section is also interesting as it provides more insights of the CBNN. It also demonsrates that the proposed reweighting scheme is helpful. Overall, the empirical evaluation is convincing and is consistent to the intuition of the proposed method. \n\nI also have a number of concerns about this paper. One of the major concern is that the paper is missing some ablation studies which are helpful to understand the proposed algorithm. For example, under the fixed training budget, how does the accuracy and diversity change if we change the training iterations per checkpoint t / or M. Another ablation study is dropping the sample reweighting during training but still keeping ensemble reweighting based on the error rate, how does this change the accuracy and diversity? This demonsrates the importance of sample reweighting which is the core component in the CBNN.\n\nThere are some other minor concers. For example, it might be better to report the accuracy and diversity (Figure 2a) of fully independent ensembles, which can be used as an upper bound. The gap also shows how much room left to improve CBNN. It is also benificial to report individual accuracy of each ensemble member (each checkpoint). This helps understand the diversity, too. \n\nOne of the best advantages of this algorithm is its wide applicability. The paper would be more convining if the authors can demonsrate its effectiveness in the trainsformer models in the NLP domain. Snapshot Ensemble and FGE are naturally short in these applications. Another potential experiments are uncertainty estimation, deep ensembles are powerful not only because of its improved accuracy but only the improved calibration / uncertainty quantification. The paper can be stronger if the authors can show CBNN has better calibration (expected calibration error,  ECE) performance than the single model.\n\nOverall, I consider this paper is slightly above boarderline at its current shape.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend to accept this paper.

In this paper, the authors proposed a boosting method to ensemble checkpoints during the training of neural networks called Checkpoint-Boosted Neural Network (CBNN) to improve the performance. In particular, a boosting scheme with both theoretical guarantee and empirical justification is proposed to accelerate model convergence and maximize the checkpoint diversity. In the post-rebuttal discussion, all the reviewers are agree that most concerns in the original reviewers are properly addressed. Reviewer Ca6wKai also raises the score. I will suggest the authors to take the suggestions from reviewers into account in the preparation of camera ready.
Decision: Accept (Poster)