Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides further evidence for the conjecture from Entezari et. al. 2021 that solutions found by SGD are linearly mode connected modulo permutations. They provide three methods to find these permutations, one of which (weight matching) is very fast to compute. Using these methods they show that high-width neural networks achieve low loss barriers for tasks like MNIST and CIFAR-10.
strength_and_weaknesses: Strengths:  
- This paper studies an important open conjecture from Entezari et. al. and provides substantial evidence for it, while stating caveats clearly (requirement of high width)     
- The proposed algorithms to find the permutations are interesting. I am surprised that the weight matching algorithm works at all, since it can speed up the compute required to solve the permutation problem by a large amount.    
- The paper is well-written   


Weaknesses:    
- I find the claims regarding the practical utility of merging models trained in a distributed way slightly exaggerated. You don’t see the same improvements in the combined model’s accuracy (Figure 9), and the combined model does much worse than ensembling. Moreover, even ensembling would be a weak baseline if you are trying to combine many models that have been trained on smaller splits of the data. Thus, it is hard for me to see how this method could be helpful in federated learning or distributed training. Nevertheless, I don’t think this particular claim is central to the paper and the results are interesting anyway.  

Neutral questions  
- Do the authors have any intuitions for why Figure 3b shows so many bumps? Perhaps the training of one of the models is slower? Maybe the right comparison would be interpolating models of approximately equal loss even if they are slightly shifted epochs.    
- What are the two different lines in each panel of Figure 2?   


clarity,_quality,_novelty_and_reproducibility: Clarity: The paper states the conjecture, algorithms and experimental results clearly.  
Originality: The results shown in the paper are novel.  
Reproducibility: I believe the authors provide sufficient information to reproduce the results in the paper.  
summary_of_the_review: I think this paper studies an important conjecture and clearly provides evidence in support of it. I recommend this paper for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes 3 methods for permuting the weights of a neural network $A$ to closely match a second neural network $B$. With this procedure, they show that often they achieve linear mode connectivity between the permuted model $A$ and model $B$. They further show that it is possible to merge independently trained models $A, B$ by averaging the weights of the permuted model $A$ and model $B$.
strength_and_weaknesses: ## Strengths

This is a genuinely exciting paper. I believe it will inspire further research into understanding the loss landscapes as well as merging models and related areas. I think the paper can potentially be quite impactful. 

As I will argue below, I think many of the pieces of this work have been explored before. To me, the main strength of this paper is that it provides a new perspective, or way of thinking about the loss landscapes, even if somewhat obvious in retrospect.

## Weaknesses

I like this paper, so the weaknesses that I point out are somewhat nit-picky.

**W1**: Many of the key pieces of this work have been considered before separately. I think some additional discussion of related work is needed.

**W2**: It appears that the linear mode connectivity results may be somewhat brittle.

**W3**: Some of the interpretations about the SGD inductive biases are not very well supported by the experiments in my opinion.

clarity,_quality,_novelty_and_reproducibility: ## Novelty and Related Work (W1)

Here I will describe some of the relevant methods, results and observations in prior work.

- The fact that the neural network optima found by SGD form a single connected basin has been well established in the literature, to the best of my knowledge first demonstrated by [1, 2] (cited by the paper). The observation has been extended in many ways, e.g. to multi-dimensional mode-connecting constructions by [5] and with a formal proof of mode connectivity [4]. In particular, [1] shows that modes without permutations can be connected by a polygonal chain with a single bend (two line segments). Relative to this observation, the present work shows that if we select an optimal permutation, we can connect the modes with one line segment instead of two.

- In Appendix A.7 of paper [1], the authors show that as the width of the model becomes larger, the mode connecting paths (without permutation) become closer to a line segment, and mode connectivity improves. This experiment is relevant to Section 5.3.

- The work [6] actually finds the optimal weight permutations for optimizing the mode connectivity, which is very closely related to Git-Rebasin. The one difference is that the authors of [6] do not claim linear mode connectivity. However, linear mode connectivity doesn't always hold, so the distinction between the two papers is not as obvious. In particular, both papers develop similar algorithms for finding optimal weight perturbations. I encourage the authors to discuss the differences with [6] in more detail.

- Merging models with weight perturbations has also been considered before in [7] with similar experiments to Figure 5.

Many of these works are cited in the paper, but very briefly. To me, Git-Rebasin combines many of these prior observations (and adds new ones!) into a single unifying picture which is highly valuable. At the same time, the novelty is not _as high_ as it may initially appear.

Other works that could be cited or discussed more:
- [4] shows that weight averaging can be used to merge close-by models and improve generalization. [8] actually more or less contradicts the Git-Rebasin hypothesis, as they argue that modes that are linearly connected also are similar functionally, and functionally distinct models cannot be linearly connected. [9] discusses another symmetry in neural network parameterization relevant to loss surface analysis.


## Experiments (W2)

While the main results are exciting, it appears that there are limitations to the linear mode connectivity observation, discussed in A.1. In particular, it seems like it is fairly easy to construct practically relevant neural network optima pairs for which linear mode connectivity will not hold after Git-Rebasin. The standard mode connectivity results of [1, 2] appear to be more robust.

Similarly, while Figure 5 shows improved test loss, compared to the models being merged, the test accuracy is not improved (Figure 9), and is significantly lower than the combined data accuracy. The improvement in the loss is not necessarily practically relevant, as it probably comes from reducing the confidence in predictions (leading to better calibration), which can be achieved in other ways (e.g. logit tempering [10]). 


## Interpretations (W3)

The authors state many times that the linear mode connectivity after Git-Rebasin is connected to the inductive biases of SGD. However, as far as I understand, there are no experiments that test this intepretation. The results is presented based on the observations that (1) there can be models where linear mode connectivity would not hold as shown in Section 4 and (2) linear mode connectivity improves during training. (1) appears to be somewhat irrelevant to SGD, as it is not shown that SGD would not find the solutions described in Section 4. (2) is not directly testing SGD either, and is confounded by the loss value; an alternative explanation is that the linear mode connectivity is more likely to hold for pairs of solutions with low loss.

If the authors want to push for the SGD inductive bias interpretation, I think they should consider experiments that e.g. try to apply Git-Rebasin to models found by non-SGD-like optimizers, e.g. full-batch training, or even L-BFGS. 
summary_of_the_review: Overall, really exciting paper. I pointed out some relatively minor weaknesses, so I currently vote for 8.

## References

[1] [_Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs_](https://arxiv.org/abs/1802.10026);
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson

[2] [_Essentially No Barriers in Neural Network Energy Landscape_](https://arxiv.org/abs/1803.00885);
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, Fred A. Hamprecht

[3] [_Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets_](https://arxiv.org/abs/1906.06247);
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, Rong Ge

[4] [_Averaging Weights Leads to Wider Optima and Better Generalization_](https://arxiv.org/abs/1803.05407);
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson

[5] [_Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling_](https://arxiv.org/abs/2102.13042);
Gregory W. Benton, Wesley J. Maddox, Sanae Lotfi, Andrew Gordon Wilson

[6] [_Optimizing Mode Connectivity via Neuron Alignment_](https://arxiv.org/abs/2009.02439);
N. Joseph Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, Rongjie Lai

[7] [_Model Fusion via Optimal Transport_](https://arxiv.org/abs/1910.05653);
Sidak Pal Singh, Martin Jaggi

[8] [_Linear Connectivity Reveals Generalization Strategies_](https://arxiv.org/abs/2205.12411);
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, Naomi Saphra

[9] [_Sharp Minima Can Generalize For Deep Nets_](https://arxiv.org/abs/1703.04933);
Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio

[10] [_On Calibration of Modern Neural Networks_](https://arxiv.org/abs/1706.04599);
Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper investigates the linear mode connectivity of neural networks trained with stochastic gradient descent, i.e., the possibility of smoothly interpolating between two (equivalence classes of) networks trained with different initialization parameters (i.e., one network is permuted to lie in the same loss basin as the other). To that end, the paper proposes three novel algorithms that compute a network permutation that matches another network's behavior (e.g., activations or weights). The approaches are based on regression between activations, approximate weight matching, and straight-through estimation. Finally, the paper conducts an empirical investigation of various architectures and datasets to demonstrate that the proposed methods successfully interpolate between two networks without increasing the loss.
strength_and_weaknesses: **Strengths**

The paper investigates an essential question in optimizing neural networks: Why do many local minima exist in the loss landscape with similar performance? The paper provides strong evidence for the conjecture that these minima all correspond to the same network modulo permutation symmetries, which provides a significant boost to the community's understanding of gradient descent landscapes. To that end, the paper introduces three novel and computationally efficient permutation selection methods that permute a model's activations or weights to match a target network's behavior. The methods are grounded in concepts from combinatorial optimization and are equipped with convergence guarantees. The paper opens up a variety of exciting research directions:
* Can the methods be extended for models with slightly different architectures?
* Does the method also work for models trained on fundamentally different datasets?
* Can we leverage these methods to combine different networks specializing in specific tasks to create a network capable of performing all tasks simultaneously?
* Can we leverage these methods to interpolate between two models in a subtractive manner, e.g., subtract a biased model from an accurate model without loss in accuracy?

**Weaknesses**

The paper claims that `neural network loss landscapes contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden units`. However, the paper only demonstrates that the proposed permutation selection methods induce linear mode connectivity for two given networks (as far as I can tell from the experiment description, the results have not been aggregated over multiple independent runs). As a result, the paper can only reasonably claim that `for some networks, the permutation symmetries allows projecting them into the same loss basin`. Thus, the claim either has to be reformulated, or the experiments have to be repeated for many network initialization parameters to provide evidence for the claim in its current form.

Moreover, some experimental details could be more precise (see clarity below). Finally, the paper posits a hypothesis on the width of models for ImageNet in section 5.1, which is relatively straightforward to test, but does not conduct a corresponding investigation.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is generally well-written and easy to follow. However, a few details should be clarified:
* Why does Algorithm 1 use a random permutation of layers instead of a uniform curriculum?
* Why does the straight-through estimator consider the loss at the midpoint of the (projected) network parameters instead of a minimax problem over all possible interpolation coefficients $\lambda$?
* Section 4 states that `SGD is implicitly biased towards solutions admitting LMC`. Why is that the case?
* How is the ensembling method in Figure 5 dependent on the interpolation coefficient $\lambda$?
* It is important to emphasize that any method that combines the models in section 5.4 should not rely on data (as in the case of weight matching).

Moreover, the interlude section is difficult to follow since all the results are presented in the Appendix. Therefore, I suggest using the additional page of the camera-ready version to move the result to the main paper.

**Quality**

The proposed methods are sound and efficient. Moreover, the experimental evaluation is exhaustive and showcases the effectiveness of the methods.

**Novelty**

To the best of my knowledge, the paper is the first to propose permutation selection methods to demonstrate linear mode connectivity of neural networks.

**Reproducibility**

The paper provides sufficient details (including code, logs, and model checkpoints) of the experimental setup to reproduce the results.

summary_of_the_review: Given the significance of the results, the potential impact on the community, and the quality of the paper, I strongly recommend acceptance of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) This paper addresses an important problem of understanding the relation between different learned models trained on the same training data (due to different initialization and SGD batches). Prior work had conjectured that these models are in fact "the same up to permutation". The current paper provides further support of this by using a matching method that aligns different multilayer models.
(b) The main interesting finding of this paper is linear-mode-connectivity (ie all solutions are the same up to permutation) for CIFAR datasets, as opposed to MNIST shown before.
(c) The paper suffers to some extent from over-claiming and lack of rigor (e.g., the author call a "counter-example" something which is not clearly a counter-example). The matching algorithm used is quite straight-forward, and arguably equivalent to transport-based methods. So while there is a real contribution here, the authors could have delivered it in a more scientific manner.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: The paper does make useful observations about the question of basins and permutation symmetries, and these would be of interest to the community. Towards the talk, the authors are encouraged to clearly explain their contribution in the context of prior work on this problem.
Decision: Accept: notable-top-5%