Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
### Summary
The authors propose late-phase weights, a method of updating the weights near the end of training via a splitting and ensembling mechanism. They analyze the benefits in the noisy quadratic setting. The method improves validation performance on a range of image recognition tasks and on enwiki8.

### Comments
* The weight interaction functions $h$ should be more explicitly defined rather than just described in text.
* The paper is overall well written and flows smoothly.
* I think there should be more discussion on the choice of $T_0$. For example, in table 1, why does SGD perform worse when $T_0=0$? It would be good to get a sense of robustness to this hyperparameter.
* Good results on CIFAR. Late-phase weights are shown to boost performance over SGD and to be complementary with SWA. There are some benefits in the OOD setting as well.

### Recommendation / Justification
I vote to accept the paper. The idea is interesting, well-motivated, and seems straightforward to incorporate into existing pipelines. However, the improvements seems modest in some settings (e.g. ImageNet) and for the best performance, it seems like we should still stick to Deep Ensembles.

### Questions
* On the ImageNet experiments, what is the validation accuracy of the pre-trained model?
* Can you comment on the computaional and memory complexity of your algorithm versus vanilla SGD?
* In the comparisons between late phase weights and SGD, do both algorithms consume the same amount of data? If so, this would be good to mention.
* Could the entire network be treated as "late-phase weights"? Would this help performance?

### Minor comments
* I would consider alluding to possible choices of the weight interaction functions $h$ when it is first introduced at the start of 2.1.
* In Algorithm 1: How does the loss function consume three inputs? This is different from when it is initially described.  
* It's a bit unclear what is being compared in Figure 2.


(increased score from 6 to 7)

review_title: Official review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper proposes a method to improve solutions found by SGD by ensembling subsets of weights in late-phase. A family of low-dimensional late-phase methods are analyzed and shown to improve generalization in CIFAR-10/100, ImageNet and enwik8. Authors also analyze the method in more tractable noisy quadratic settings. 

Contribution of the authors is that rather obtaining ensemble they utilize efficient ensemble to guide SGD training and ultimately obtain a single model. 

Reason for score:

While the paper discusses efficient ways of utilizing late-phase weight ensemble and improving SGD training, the demonstrated benefit is not significant enough for practitioners to pursue the method. Without strong practical application potential, merit of the proposed method is weak since it does not obviously elucidate some aspects of neural network training. 

Pros:

The paper is clearly written and easy to understand the proposed method is. It is well structured that helps to improve the clarity. 

Proposed method tackles a significant problem in the standard ensemble method in which both training/inference computation can be quite costly. The paper’s method only ensembles subset of weights therefore added training cost is minimal and since inference is done on averaged weight, it becomes essentially a single model.

Among various late-phase schemes, BatchNorm late-phase seems to work well which is widely used among vision models so easily applicable. Also since late-phase can be applied post-pretraining, it can be used to improve pre-trained models.  

As far as I can tell various experimental conditions are very well controlled and thoughtfully designed.

Cons:
The idea of weight averaging is not so novel as duly noted by the authors.

Main question arises for the paper is whether the proposed method is worth the effort. While all experiments show that the proposed method improves the baseline somewhat, deep ensemble baselines remain strong. Also quoted difference between methods does not mean statistically significant effect (see Vincent Vanhoucke’s article on reporting significant figures https://towardsdatascience.com/digit-significance-in-machine-learning-dea05dd6b85b). According to this article, results reported in Table 1, CIFAR-10 in WRN, a significant figure with a 10k test set should be around 0.2% and differences between different methods are at best marginal. This can be applied to most tables and except for Deep Ensemble’s improvement other differences are not very significant.
	
I wonder as discussed by the authors, this is due to mostly the benefit of ensembles is
through incorporating different modes as argued in [Fort et al., 2020] rather than a single mode. I imagine a single mode ensemble could be beneficial when variance within the mode is large, however for models considered by the authors seem to have small model variance which minimizes effect of technique utilizing single mode. 

While \sigma_0 and T_0 are hyperparameters of the algorithm, no good way to determine it is explained. 

The role of section 3.1 is not clear. For one thing, the legend in Figure 1 is confusing where the role of non-integer K is mysterious to me. I would suggest clarifying what the message of the section would be in context of understanding late-phase weight models. 


Nits and additional feedback:

Anonymized link is neither there in the main paper or included as supplementary material. If the authors intended to include the code, this is a note that code can not be found to the reviewers.

For models that do not use BatchNorm, I believe most interest to practitioners would be using Transformer based models. I wonder if rank-1 late-phase or LayerNorm late-phase would show improvements in this case. 

Was “Late-phase classification layers” ever evaluated or discussed in the main paper? I find some discussion on the appendix but seem to be missing in the main text. 

---
I thank the authors for their hard work addressing issues raised by the reviewers.

Authors have answered many issues pointed out (by improved performance and showing robustness to hyperparameters) and I've increased my score from 5 to 6, and support accepting the paper. 

review_title: Official Blind Review #2

final_score: rating: 7: Good paper, accept
review_content: This work suggests a variant of ensembling that is more compute-efficient. Specifically, it involves forking an ensemble only in the late stage of training, and forming this ensemble via a "low-dimentional" family. That is, instead of maintaining independent networks, maintain only "low-rank"-style perturbations of the base network (for various instanciations of "low-rank").
The experimental results are somewhat limited, but appear to be competitive with current efficient-ensembling approaches like SWA/SWAG. The absolute improvement of this method is not very large (<0.3% on CIFAR, <0.2% on imagenet), and there is a large gap to Deep Ensembles. I weakly recommend acceptance, because the method appears promising for future work, and the experiments seem correct. 

There is also a theory section included, though I am generally unconvinced by results in such simple toy examples.
(such settings can usually be contrived to exhibit any desired behavior)

Weaknesses:
- The experimental section would be greatly strengthened by additional experiments for different models and settings. There are only 2 architectures tested on CIFAR-10, for example. It would also be informative to see the performance of these methods in "harder" settings -- for example, CIFAR-10 with fewer train samples.
- The OOD uncertainty results could be expanded. Uncertainty estimation and robustness are some of the most relevant practical uses of ensemble methods, so it is especially important to evaluate ensembles in this context. Currently aggregate results are shown in Table 4, but it would be good to explicitly see, for example: how the performance of this method degrades with increasing CIFAR-10C corruption severity, as opposed to Deep Ensembles. Also, reporting the Mean Corruption Error (mCE) for each dataset individually will allow standard comparison to prior methods.

Comments which do not affect the score:
It seems that starting the ensembling at a "late phase" in training is the main contribution of this work. This could be applied to any ensemble method, and you propose several explicit instantiations. It could help to focus the writing in terms of this contribution, and also to further investigate the role of T0 (the time at which ensembling starts).

---
Edit after rebuttal: Increased score from 6 to 7.

review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: To improve the generalization performance of SGD methods, 
this paper proposes to use an efficient ensemble-like approach 
which computes an average of an ensemble of SGD weights
when retrained from some late-phase of SGD dynamics. 
This idea is different to most recent ensemble-based approaches which 
aim to average the predictions of the models. 

The paper focuses on some specific layers of neural networks 
in order to apply the late-phase training. 
The batch normalization layers are shown to be 
simple and effective. Some other layers are also analyzed, 
including a recently introduced rank-1 multiplicative matrix 
weights idea for full-connected layers. 
Section 3 presents the numerical results and show that the generalization of SGD 
is more-or-less improved on various benchmarks.
Explanation of why the generalization is improved in relation with the flatness
of energy landscape is also discussed.  

I find that this approach is quite sensitive the choice of the hyper-parameters, 
such as the beginning of the late-phase T0, and the noise perturbation sigma0. 
It is written in Section 2.1 that in practice … sigma0>0 yields a set of models …
this results in improved final generalization. However, in the result of ImageNet in Section 3.3, 
the sigma0 equals to 0. Thus, it is not conclusive that sigma0>0 is better. 
As the improvement in Section 3.3 seems marginal compared to the baseline and the 
standard deviation, it thus does not fully support the effectiveness of the batch normalization layers. 
I would recommend using some other dataset or models, 
but with a more consistent set of hyper-parameters. 

In terms of writing, I would recommend to write out
the full algorithm of Alg. 1 or at least in the Appendix, 
including the variant of the SGD momentum and Adam. 
The SWA is also worth writing out clearly, which is not clear to the reader. 
Is the DeepEnsemble result in Table 1 from SGD or SWA? 
This is not clear from the text.

Overall, I think both the methodology and the writing need to be improved. 

##
The revisions made by the authors have addressed all my concerns. 

review_title: I think both the methodology and the writing need to be improved. 

Meta-review: This paper proposes to learn an ensemble of weights given a set of base weights from some point late in normal training. The authors apply this approach to a number of configurations and find modest performance improvements for normal test settings and larger improvements for out of distribution settings. While reviewers had some concerns about the size of the improvement relative to baselines, all reviewers agreed that the proposed method is interesting and will likely impact future work, especially given the new experiments provided by the authors. I recommend that the paper be accepted. 
Decision: Accept (Poster)