Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes the striking empirical phenomenon that, accounting for permutations, two independently trained networks of the same architecture will have no energy barrier on the linear path between them. Furthermore, the authors demonstrate an issue with naive interpolation for deeper networks where the activations experience a variance collapse on this linear path, creating an 'artificial' barrier on this connected path. To resolve this, the authors propose a simple renormalization procedure that re-establishes mode connectivity (albeit no longer linear). 
strength_and_weaknesses: Strengths:
- Renormalization without permutation significantly lowering the barrier is an interesting finding complementary to the concurrent Ainsworth et al. 2022 result.
- Renormalization technique is an improvement over Ainsworth et al. 2022, which requires the replacement of BatchNorm layers with LayerNorm layers to alleviate the barrier in Residual networks.

Weaknesses:
- The pattern of increasing test error corresponding to a larger barrier is an interesting finding. But there doesn't seem to be an intuition on why this is true. Without an explanation, it's unclear why this pattern is viewed as expected or "simple"

From the paper:
"the high barrier of ImageNet models stems in part simply from the relatively higher test error of said models"

Comment:
Could the authors elaborate on the connections between robustness and linear mode connectivity? Is there an expectation that linear mode connectivity will indicate robustness?  
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and refreshingly honest about the LMC phenomenon being shown empirically for smaller models while still needing further validation in the ImageNet setting where the energy barrier is still significant. The overall idea is simple and straightforward.

The authors do provide source code, so reproducibility is likely.
summary_of_the_review: The empirical validation of the LMC phenomenon is important, and this work does a good job of diagnosing the issues with LMC when applied in practice. The renormalization method appears novel and significantly improves the loss barrier along the bath between the two models.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the linear mode connectivity problem empirically and uncovers a new phenomenon called variance collapse (where activation variances of interpolated networks between two parent networks reduces significantly). Test accuracy barrier reduction is shown successfully for the interpolated networks (for CIFAR-10, barrier goes down to about 1.5% gap in test accuracy between parent models and the interpolated network).
strength_and_weaknesses: The paper has following strengths:

1. It is interesting to see the variance collapse phenomenon on the interpolated networks.

2. Barrier reduction for CIFAR-10 down to 1.5% is significantly better than existing techniques.

3. The paper is well-written and nicely explained. Related work is cited and discussed very well.


The paper has following weaknesses:

1. The correction method presented by the authors is trivial. It is well-known that tuning batchnorm can significantly improve accuracy (sometimes even in completely unsupervised, domain adaptation settings, e.g., see: TENT: https://openreview.net/forum?id=uXl3bZLkr3c). Hence, it is no surprise that fixing batch norm statistics would help in this case. While the proposed method achieves great results on CIFAR-10, we can see that on ImageNet (despite the proposed method being the current best), the gap in interpolation accuracy is very significant (making this method not too useful on large-scale datasets). I was wondering if instead of just initializing affine parameters and re-computing the running mean/variance statistics, the authors were to finetune (or learn) the BN stats similar to TENT above, would we see significant reduction in the ImageNet barrier (compared to their current results)? Having better numbers for large scale datasets like ImageNet would indeed make this work useful.

2. In the final sentence, the authors state “a better understanding of linear interpolation would result in improvements in applications like weight-space ensembling, checkpoint averaging, robust finetuning, etc.”. I think the current paper lacks the use of proposed ideas (e.g., fixing variance collapse, etc.) within a real application framework. It would make the paper significantly stronger if the authors could pick a couple of the above applications and show that fixing this subtle issue significantly improves the current solutions in those domains.

3. There are many figures in the paper that do not contribute much to other results. Example, training loss barrier, test loss barrier, etc. (Fig. 6ab, Fig. 7a), do not provide any new intuitive information over the test accuracy barrier results. I suggest to remove the redundant information and include more results on downstream applications and other experiments suggested above (e.g., TENT on linear interpolations, etc.).

4. Finally, I am a bit unclear on what happens when the model already has BNs (e.g., ResNets). Do you add a new BN layer in your correction method (even if a BN layer is already present)? Page 6, “We present the following practical…” paragraph seems to suggest that you add a new BN layer. If you do add an extra BN layer: When you do interpolation, wouldn’t you already set the affine parameters (of existing BN) to the respective interpolated means and interpolated variances from parent networks (similar to all other weights in the network)? In that case, I am not sure how the new BN layer adds anything new (other than recomputing the running mean and running variance statistics with training data, which is similar to other methods like TENT that also tune the BN layer (but they also do some more learning on the BN layers)). Also, when you do the permute invariance methods, do you consider the BN layers (for somehow permuting the affine parameters, running mean, running variances)?

clarity,_quality,_novelty_and_reproducibility: Paper is clear, lacks a bit of novelty since BN tuning methods already exist.
summary_of_the_review: The paper can be significantly stronger if it actually shows improved results on downstream applications and manages to reduce the ImageNet barrier further with more advanced BN tuning methods.

==== UPDATE AFTER REBUTTAL ====

I have read the author response and other reviews. I am raising my score to 6 since the authors have reasonably addressed my main concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: I thank the authors for their responses. I am especially happy about the added section 3.2. that tries to provide some theoretical reasons for the observed phenomenon of variance collapse. I am raising my score to 6.

The authors in this paper observe that interpolating between neural network weights decreases the variance of activations. By counteracting this phenomenon they show that much of the performance drop in interpolated models can be recovered. They show this across multiple architectures, normalisations and datasets.
strength_and_weaknesses: Strengths: The paper is very clear and well written. It is easy to follow and presents a concise observation leading to a simple modification to recover the performance in interpolated models. The experiments are straightforward and clear. They also do a good job in relating to existing research on this subject.

Weaknesses: It would be great if some more theoretical analysis were provided in an attempt to understand why variances collapse, how their modifications change the standard interpolation. While it is great to make a simple observation, propose a fix and demonstrate that it increases performance; one is left wondering, after reading the paper, what exactly we have learned from this scientific study.
clarity,_quality,_novelty_and_reproducibility: Random Notes:

It would be great if the paper were readable without prior knowledge of Entezari’s 2021 paper.

3 paragraph, introduction: please define ‘loss barrier’ when you first use the term

2.2. Please give a more detailed explanation of ‘unit-rescaling’, ‘positive homogeneity of ReLU’ and why SGD helps here.

Fig 2: I don’t find some of the filters from the middle again on the right, are these the right plots? Also, there seem to be some obvious mismatches, which one could change by hand. I know the best matching may not be interpretable, just wondering if there is a bug here?

Also, Fig2, I am wondering if for a CNN having a filter which is slightly shifted (e.g. phase shifted wavelet) may give a low correlation in filter shapes, but almost the same pattern of activations in the convolutional feature map? I.e., maybe there is some translation-invariant way of matching convolutional filters?

Which algorithm are you using for linear assignment? Please provide more technical details.

Just above eq. 2 there is a superfluous ‘the’

End of 2.3. How are the weights interpolated and permuted exactly? Please give more detail.

I am little confused about the ordering of tested models and datasets in different sections. Like, for instance, starting section 3, the standard choice would have seemed like LeNet on MNIST and ResNet on CIFAR10. Also, why are models split into different sections at all, like section 5.3??

Section 4, second paragraph: WHY? Can you give any motivation for this? I guess this relates to my general sense when reading the paper: It would be great to learn something about neural networks, rather than just observing that some statistic changed, fixing it with a hack and claiming victory because performance increased.

Here is a proposal for a little more understanding of what is happening: Take a 2D slice of the loss landscape (like in other model averaging papers, e.g., the ones about model soups) and visualise the different interpolation schemes: 1) regular interpolation should be a straight line; 2) adding the constraint that the mean and variance remain the same for the units should a slight deviation – possibly through regions with lower loss? This is just a first proposal for an experiment to dig deeper, build more intuition and try to actually understand what is going on.

Fig 6: too small, and the different losses seem redundant, maybe just pick one an put others in appendix?

Fig 6: what is this? Only permuted or also variance adjusted? Add clarification.

End of 5.2: OK, I appreciate that you add Ainsworth et al. as concurrent work. However, now you are comparing against them. If you do this, then please do it properly. Include it in your plots. Show if the variance trick also improves upon the better permutation scheme from their paper etc…

Fig 8: What is this? Top1/5 accuracy? Why are those two papers cited, is it their interpolation method?

5.4.: ‘have’ +a ‘maximum barrier’

What exactly is the statement you are trying to make with Fig. 9? Surely not a causal relationship? Also, I am not sure how much of an explanation that offers at the end of section 5.
summary_of_the_review: The paper is clear, simple and proposes an effective modification to interpolation variance collapse. There is some concurrent work, to which it would be great to provide a more detailed comparison. Moreover, it would be great if the paper would go beyond noticing that some statistics changed, modifying this and obtaining better performance. Ideally, after reading this paper, we would have a slight advance (even preliminary, anecdotal) about why variance collapses in interpolated models, how the modification changes the interpolation path and why this works.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper examines neural networks which are interpolations between two SGD solutions, and finds that, even after alleviating issues related to permutations (using existing algorithms), we get high loss barriers due to "variance collapse", i.e., a collapse in the variance of their activations, causing poor performance. An algorithm is suggested to repair this issue by rescaling the reactivations, which helps a lot with the final performance. 

The reviewers had some concerns, and this was initially a borderline paper. However, most concerns were addressed after a major revision of the paper (almost being a "too large" revision of the paper). The only remaining possible weakness seems to be the novelty of the method: it is very simple, and reminiscent of BN-tuning which was previously suggested (note to authors: previous papers already did this even without backpropagation [A], and without the pre-existence of a BN layer [B]). However, the specific method of initialization of the BN layers differs from previous papers. Anyway, I don't think this is much of an issue, as I think the main novelty here is that such a simple method is so useful for this different problem. 

[A] Sun et al., Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks, NeurIPS 2019

[B] Hubara et al., Accurate post training quantization with small calibration sets, ICML 2021
justification_for_why_not_higher_score: It's a simple and nice method, but in retrospect, it not is very surprising that it works. 
justification_for_why_not_lower_score: All reviewers vote to accept [6,6,8].
Decision: Accept: poster