Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed to ensemble the checkpoint models during the training to boost the performance of neural networks. To improve diversity over checkpoints, the proposed method changes the weights for each sample based on the mis-classification rate along the training process. When finally doing the ensemble over the checkpoints, a weight for each checkpoint based on mis-classification rate is also assigned. The experimental results show that comparing with existing checkpoint-based ensemble models and single model,  the proposed method has 1) better accuracy; 2) high diversity over different checkpoints; 3) low training cost. 
main_review: Overall, the paper is well-written and easy to follow. The proposed method is straightforward and should be novel. The experimental results look promising on both performance and training cost. 
But I have a few questions and concerns.

1. Is there a reason why ImageNet with Densenet and Resnet are omitted in Tab. 1?
2. As SSE is the second-best performance, I am wondering if the authors tried to adopts cyclic learning rate schedule as SSE does to improve CBNN? 
3. The theoretical analysis Theorem 1 and Theorem 2 seem trivial and the conclusion is a little bit vacuous. In particular, the conclusion is that a upper bound of the exponential loss is decreasing with more checkpoints. The upper bound can be very loose, which can lead to a vacuous conclusion. It would be better to calculate the tightness of the upper bound empirically. 
4. Although random labeling has the error rate (k-1)/k, the worst case error rate can be worse than this. Therefore, $\lambda_m > 0$ needs to be assumed.  

Overall, as the experimental results look promising, I think this paper would be interesting to the DL community. However, due the above concerns and lack of the limitation and societal impact discussion, I think the current version of the paper is not ready for publication. I am open to change the score based on the authors' rebuttal. 
limitations_and_societal_impact: I don't think the paper has explicitly mentioned limits of the methods. 
The paper showed better performance on imbalanced datasets in Sec. 5.4. Since imbalanceness is common in datasets with minority groups, it would be good if the authors can discuss the impact of their method in this aspect.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors made a study on how to ensemble checkpoints with a boosting scheme, leading to faster convergence and more diversity. By adaptively reweight the training samples, Checkpoint-Boosted Neural Network (CBNN) can accelerate convergence and encourage diversity among individual ensemble members. The empirical evaluation demonstrated that under the same training budget, CBNN outperforms a number of baselines including Snapshot Ensemble. The authors also showed that it achieves great performance on imbalanced dataset because of CBNN's adaptability.
main_review: Snapshot Ensemble requires a modification to the learning rate scheme. This limits its application when cylinder learning rate scheme is not desirable. CBNN does not require any strange tweak to the current learning algorithm, so it has a much wider application. The authors provide convergence analysis, leading to a theoretically grounded algorithm. The baselines this paper compared to are competitive, including recent advances such as Snapshot Ensemble and FGE. CBNN outperforms these competitive baselines on a variety of network architectures and vision datasets in terms of the accuracy measure. Additionally, the diversity analysis section is also interesting as it provides more insights of the CBNN. It also demonsrates that the proposed reweighting scheme is helpful. Overall, the empirical evaluation is convincing and is consistent to the intuition of the proposed method. 

I also have a number of concerns about this paper. One of the major concern is that the paper is missing some ablation studies which are helpful to understand the proposed algorithm. For example, under the fixed training budget, how does the accuracy and diversity change if we change the training iterations per checkpoint t / or M. Another ablation study is dropping the sample reweighting during training but still keeping ensemble reweighting based on the error rate, how does this change the accuracy and diversity? This demonsrates the importance of sample reweighting which is the core component in the CBNN.

There are some other minor concers. For example, it might be better to report the accuracy and diversity (Figure 2a) of fully independent ensembles, which can be used as an upper bound. The gap also shows how much room left to improve CBNN. It is also benificial to report individual accuracy of each ensemble member (each checkpoint). This helps understand the diversity, too. 

One of the best advantages of this algorithm is its wide applicability. The paper would be more convining if the authors can demonsrate its effectiveness in the trainsformer models in the NLP domain. Snapshot Ensemble and FGE are naturally short in these applications. Another potential experiments are uncertainty estimation, deep ensembles are powerful not only because of its improved accuracy but only the improved calibration / uncertainty quantification. The paper can be stronger if the authors can show CBNN has better calibration (expected calibration error,  ECE) performance than the single model.

Overall, I consider this paper is slightly above boarderline at its current shape.
limitations_and_societal_impact: The authors discussed the limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a novel method to obtain a diverse set of checkpoint models for efficient ensembles. Through the boosting inspired sampling weighting scheme, the proposed method not only facilitates the convergence of the model but also produces checkpoints with diverse predictions with an enhanced ensemble performance. The authors of the paper theoretically prove the convergence of the proposed method and empirically demonstrate that the method is capable of producing ensembles with good performance given a fixed computational budget.
main_review: All in all, this paper is well written and easy to follow. The proposed method seems technically sound. While the idea of sample reweighting is definitely not new in general in the context of deep learning (for example, see [1][2]), I personally find its connection to boosting and also the use of it as a way to promote model diversity among different checkpoints to be very interesting. Specifically, despite the reduction in training time computational cost, most of the previously proposed methods for "checkpoints ensembling" still perform poorly when compared to standard ensemble strategies likely due to low diversity among the models. The authors of the paper directly tackle this problem by utilizing a sample reweighting approach to introduce more diversity to the checkpoints. The authors of the paper also conduct a good amount of experiments and ablation to demonstrate the effectiveness of the proposed method. 

Here are some shortcomings/suggestions of the paper. Firstly, in my opinion, as an illustrative purpose, It might be worthwhile to see how big the gap is in terms of accuracy and diversity when compared to an explicit ensemble. In addition, it might be interesting to also give a more careful analysis (in addition to the ones present in section 5.3) on why the proposed method outperforms previously proposed checkpoint ensemble methods. Is it because of larger diversity or better individual model performance? I am also not entirely sure how relevant/helpful the theoretical analysis is. One of the main contributions of the paper seems to be about enhanced diversity with the proposed strategy, and yet there isn't any diversity involved in the section. It would be much more interesting if the authors of the paper can theoretically demonstrate the diversity aspect of the proposed method? Lastly, it seems that the proposed method also introduces additional hyper-parameters like the deviation rate. How sensitive is the performance of the algorithm to the choice of hyper-parameters, and how easy is it to tune for it? 

Overall I think this is a good submission and recommend accepting it.

[1] Chang, Haw-Shiuan, Erik Learned-Miller, and Andrew McCallum. "Active bias: Training more accurate neural networks by emphasizing high variance samples." Advances in Neural Information Processing Systems 30 (2017): 1002-1012.

[2] Ren, Mengye, et al. "Learning to reweight examples for robust deep learning." International Conference on Machine Learning. PMLR, 2018.  
limitations_and_societal_impact:  The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: I recommend to accept this paper.

In this paper, the authors proposed a boosting method to ensemble checkpoints during the training of neural networks called Checkpoint-Boosted Neural Network (CBNN) to improve the performance. In particular, a boosting scheme with both theoretical guarantee and empirical justification is proposed to accelerate model convergence and maximize the checkpoint diversity. In the post-rebuttal discussion, all the reviewers are agree that most concerns in the original reviewers are properly addressed. Reviewer Ca6wKai also raises the score. I will suggest the authors to take the suggestions from reviewers into account in the preparation of camera ready.
Decision: Accept (Poster)