Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper combines neuron alignment with mode connectivity. Specifically, it applies paths neuron alignment to the calculation of mode-connecting and empirical results show that alignment helps in finding better curves.
Combining neuron alignment with mode connectivity seems to be a good idea, but the message the authors want to convey is somewhat vague. Some key details are not presented clearly, and some contents seem to be irrelevant. The following are some detailed comments and questions:
1. One main contribution of this paper is the observation that the observation that alignment helps in finding better curves. An observation is excellent if it brings significant performance improvements in practice, or if it brings deep insights in the understanding of the field. However, for the former, the improvement in the performance is not that much; for the latter, there is hardly any insight conveyed by this paper. Therefore, this observation itself is not strong enough.
2. One contribution of this paper is applying proximal alternating minimization (PAM) when optimizing the parameters and proving its convergence. Nonetheless, PAM is only used in one model (TinyTen) and does not bring any improvement in the performance. It seems that there is no point in applying PAM and the contents related to PAM are all somewhat irrelevant.
3. Usually sufficient details help in good understandings, but in this paper, some key details are unfortunately missing. For example, in Algorithm 2, details on the optimization step is not clear: what is the optimization method the authors use other than PAM? Also, no comments are addressed on Figure 7 to Figure 10, either in the main body or in the appendix. I would like to see more explanations details. If the source code is provided, it will be better.
In sum, the idea seems to be interesting, but the overall quality of the paper is still yet to be improved, and some key details need to be addressed more clearly before it can be accepted as a qualified submission.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper investigates the connection between symmetries in the neural network architectures and the loss landscape of the corresponding neural networks. In the previous works, there was shown that the two local minima of a neural network can be connected by a curve with the low validation/train loss along the curve. Despite the loss on the curve being close to the loss at the end points, there are segments of the curve on which the loss in higher than loss at the local minima. To overcome this problem, the authors proposed two-step procedure: 1. Align weights between two neural networks 2. Launch the path finding algorithm between aligned weights. In other words, the authors proposed to connect not original local minima but local minima that parametrize the same functions as the original ones, but have a different order of neurons. The authors also proposed PAM algorithm where they iteratively apply path finding algorithm and weights alignment algorithm.

Novelty and significance. The idea to combine the symmetrization of NN architectures with path finding algorithms is new to the best of my knowledge. Experimentally, the authors showed that ensembling the midpoint and endpoints of the curve found via path finding  algorithm coupled with the neural alignment algorithm delivers a better result than simple averaging of three independent networks. This is a new and significant result, since before the ensembling of points along the curve had the same quality as the ensemble of three independent networks or marginally better.  
The weak side of the paper is the PAM algorithm that occupies a significant part of the paper and does not deliver a significantly better result than the simple application of the neural alignment procedure before launching the path finding algorithm.

Clarity. Overall, the related work section contains all relevant references to the previous works to the best of my knowledge. The paper is well written, excluding the section Neuron Alignment that lacks notation description.  

The paper contains several typos and inaccuracies:
1. “However, We find its empirical performance is similar to standard curve finding, and we advocate the latter for practical use due to computational efficiency.” The word “We” should start with the lowercase letter.
2. In the sentence “The first question to address is how to effectively deal with the constraints in equation 6” the index i should be replaced with the index l.
3. Notation Π|Kl| introduced after equation 5.
4. In the section describing neuron alignment  algorithm Lie et al. [1] used a different notation. So I would recommend to further extend this section and add all necessary notations. Also, I would recommend to add a direct link to the paper where the problem is described in matrix form.
5. In the Algorithm 1 “Initialize P θ2 := [Wˆ 2 1 ,Wˆ 2 2 , ...,Wˆ 2 L ] as [W2 1 ,W2 2 , ...,W2 L ] for k ∈ {1, 2, ..., L − 1};”  k is not used anywhere in notation.
6. In Figure 3, “model 2 aligned ” sing is out of the plot box for ResNet-32 and VGG-16 architectures.
7. The appendix contains the sketch of the proof that is quite difficult to follow. I would recommend giving all necessary definitions as it is done in the [2] and extend the proof. 

Overall, it is quite an interesting paper but it contains some drawbacks.
[1] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E Hopcroft.  Convergent learning: Do different neural networks learn the same representations?  In ICLR, 2016

[2]  H́edy Attouch, J́erˆome Bolte, Patrick Redont, and Antoine Soubeyran.  Proximal alternating minimization and projection methods for nonconvex problems:  An approach based on the kurdyka-łojasiewicz inequality.Mathematics of Operations Research, 35(2):438–457, 2010

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Given two parameters theta_1 and theta_2 of the same architecture, the authors propose to learn a minimal loss curve between theta_1 and P theta_2, where P is a permutation matrix yielding another equivalent parameterization of the same network. The authors show that either by initializing P with neuron alignment or by optimizing P jointly with the curve, one can find a parameter-space with better accuracy and loss than by just learning a curve between theta_1 and theta_2. The authors also show that initializing P is sufficient to obtain these gains, avoiding the complexity of also optimizing for P. Furthermore, they show that ensembles across models from these curves have a very mild gain in accuracy to those of non-aligned curves.

The main qualm I have about this paper is about the significance of the contributions and the motivation.
At the core, the authors propose to find a curve between theta_1 and P theta_2 where P comes from aligning theta_1 and theta_2 as in (Li et al. 2016.). This is lacking almost any motivation, or discussion on what problem they are trying to solve. Are they trying to find ensembles with lower error? If that is the case, well the results are evidence of a negative result in this respect, which is okay but given how ad-hoc and poorly motivated the method is to that objective it's not much of a contribution. Are they trying to better understand theoretically the loss landscape of neural networks? I don't think there's any theoretical gain in that regard from this paper either. They show that the curves between aligned networks are better, but they don't show how this relates to anything else in the published literature or open questions in the theoretical deep learning field.

Regarding contribution 2, PAM is in the end shown to not converge to better curves than simply initializing with alignment. Also, doesn't the convergence result to a critical point also apply for standard descent methods? The convergence theorem doesn't seem to be much of a contribution in my opinion, either to the optimization or the deep learning community.

Regarding contribution 3, I agree that better curves can be learned faster, but why is this a meaningful contribution? What problem that the deep learning or the theoretical machine learning cares about does this help solve?

Regarding contribution 4, as the authors themselves admit, the improvement is very dim in comparison to non-aligned curves, and comparisons to other ensemble methods are not present.

I want to acknowledge that while the motivation and contributions are in my opinion dim, I find the experimental methodology of this paper very solid. All the claims are to my extent very well validated with experiments, and the experiments are in general well designed to validate those claims. My problem is sadly not with the validity of the claims but with their significance.
review_title: Official Blind Review #1

Meta-review: This paper studies the loss landscape of neural networks by taking into consideration the symmetries arising from the parametrisation. Specifically, given two models $\theta_1$, $\theta_2$, it attempts to connect $\theta_1$ with the equivalence of class of $\theta_2$ generated by weight permutations. 
Reviewers found several strengths in this work, from its intuitive and simple idea to the quality of the experimental setup. However, they also found important shortcomings in the current manuscript, chief among them the lack of significance of the results. As a result, this paper unfortunately cannot be accepted in its current form. The chairs encourage the authors to revise their work by taking the reviewer feedback into consideration. 
Decision: Reject