Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper studies a special type of weight symmetry in neural networks. I think studying the geometry of neural-nets is an interesting and important direction for understanding neural-nets, and along this direction, weight-space symmetry is an important subject. However, it seems to me this paper does not make enough contributions. Details are given below.
 
List of contributions of the paper:
 
1.	Propose an algorithm to find a (low-loss) path connecting arbitrary two partner local minima and passing through a permutation point, where a permutation point is defined as a weight setting where a pair of neurons (in the same layer) have the same fan-in and fan-out weights. 

2.	Theoretically prove that some permutation points are connected via paths with equal loss. (Proposition 1 and 2)

3.	Provide a lower bound for the number of permutation points and high-order permutation points (Proposition 3).

Cons: 

1.	The theory of this paper is a bit weak. There are three propositions.  Proposition 1 and Proposition 2 are about the equal-loss surface and theoretical existence of an equal-value path. They are kind of straightforward to prove. Prop. 3 is about counting the number of permutation points. It is a rather simple combinatorial problem, and the lower bound of the expression (an exponential bound) seems standard. 

2.	Simple proofs can sometimes provide nice insight, but it is not clear how the study of partner global minima can help improve the understanding of DNN. 
    --First, partner global minima are just a special case of critical points created by "neuron splitting", which has been comprehensively studied in  [FA2000] (Fukumizu and Amari, 2000). Note that Theorem 1 of this paper is also directly borrowed from [FA2000]. To me, this paper does not provide much additional theoretical insight on neuron splitting. 
    --Second, what is the significance of the simulation results? Prior works have shown the existence of low-cost path; this paper shows the existence of a low-cost path containing a permutation point. The major difference is that the new low-cost path is more special. Why is this finding interesting and useful? (noting that the proof of the existence of such a path seems to be much easier than proving the existence of such a path for two general global minima).
    

3.	It is not clear how the path-finding algorithm helps in practice.
a)	The algorithm is computationally expensive. It is a double-loop algorithm: in the outer loop, d is reduced by a tiny amount at each time; in the inner loop, for a fixed d, gradient descent is run for the entire DNN (with only one parameter excluded) until convergence. The total time is (# of d) * (original running time). Here, # of d’s depends on the grid: if the initial d is 10 and the grid size is 0.1, then (# of d) = 100. 
     Compared to the path-finding algorithm in Garipov et al., which only runs GD for one time, this algorithm is much more expensive, yet the benefit is unclear. 
b)	The motivation of the algorithm is not clear. Why choose to monotonically decrease the difference between the fan-in weights (of the chosen pair of neurons), but let all other weights freely optimized during the pathfinding algorithm? The paper does not provide a detailed explanation of this. 

4.	Minor issues
  -- The weight parameters created by Algorithm 1 may not be a critical point of the loss function, and thus not necessarily a saddle point. I think the claim “since the path connects two partner minima, there must be at least one saddle point on the path” is incorrect without extra assumptions. 
  --The paper mentioned "global minima" in the introduction; but in practical training, one does not always find global minima. Is "global minima" crucial for the theory and for the algorithm? 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper studies the permutation symmetry of deep neural networks. It was known that by reordering neurons and their connections in each layer, the input -> output map the neural network represents can be preserved. This corresponded to a set of unconnected equivalent points in the weight space. The authors study the weight-space connectivity of these points through midpoints they call “permutation points”. They demonstrate that such points are members of high-dimensional manifolds of equivalent points. After that, they look at empirical experiments and explicitly construct a path between two equivalent weight-space points on a toy task and MNIST.

I generally like the paper and its geometrical lens on the problem. I find the figures very helpful in understanding what is going on. However, there are a few points that I wasn’t entirely clear on. I will detail them below.

-- Point 1 --
Connecting equivalent minima vs connecting SGD-found minima. 

If I understood the paper correctly, the derivation connects two weight space points A and B whose weights and biases, once loaded to the neural network, would have the exact same answers on all inputs X i.e. f_A(X) == f_B(X), i.e. they are a pair of equivalent points. I understand that those are the ones we obtain by using the permutation symmetries.

However, some of the papers cited look at the low-loss paths between pairs of optima found by training with SGD from independent initializations, which in turn represent different functions. I.e. for two such optima C and D, the predictions on the val/test set are (sometimes) different, showing that the functions are not the same. I found the initial evidence in:

Large Scale Structure of Neural Network Loss Landscapes. Stanislav Fort and Stanislaw Jastrzebski. NeurIPS 2019. (https://arxiv.org/abs/1906.04724)

And also in much more detail in another OpenReview submission:

Deep Ensembles: A Loss Landscape Perspective. (https://openreview.net/forum?id=r1xZAkrFPr)

I found your results very compelling, however, the two problems seem to be quite different -- on one hand you are connecting a pair of minima that are in fact *identical* by construction. On the other hand the empirical work in literature (especially in the two papers I provided above) deals with pairs of minima that in fact do differ on the test set (at least).

Would you mind commenting on how the two approaches relate to each other? 

-- Point 2 --
Higher order connectivity

In Large Scale Structure of Neural Network Loss Landscapes. Stanislav Fort and Stanislaw Jastrzebski. NeurIPS 2019. (https://arxiv.org/abs/1906.04724), the authors look at higher-order connectivity between SGD-found optima (e.g. connecting 3 optima on a 2-manifold, 4 optima on a 3-manifold etc.). They also have a particularly simple path-finding algorithm. This seems relevant to the approach you are presenting, although the points in Point 1 still stand.

-- Point 3 --
Previous work on connecting two optima using layer-wise weight merging

Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets. Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, Rong Ge. (https://arxiv.org/abs/1906.06247)

They prove that a low-loss path between 2 optima exists provided you can apply a p_keep = 0.5 dropout on each of the optima without incurring a significant loss punishment for it. This paper seems very related to your approach. Would you mind commenting on the differences?

-- Conclusion -- 
I like the paper and the idea in general. I appreciate the geometrical lens the authors took. My main point of confusion relates to the connection between this work and the low-loss connectivity between inequivalent optima found in literature, which (at least to me) seems to be the more interesting of the two connectivities.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper presented a method for studying the landscape of the loss function w.r.t. parameters in a neural network from the perspective of weight-space symmetry. The detailed method includes constructing/optimising a low-loss path in between two parameter vectors (incoming connections from the previous layer) of two neurons respectively, and then set the output weight vectors (outgoing connections to the next layer) to be the same without changing the output of the current layer. 

The empirical results show that the proposed optimisation scheme for finding the path is indeed low-loss, which implies that there exists numerous critical points in between two equivalent local minima (which are global minima in over-parametrised models).

My major concern is that scope of the study is very limited, since only permutation is considered here. I am not an expert in this field, however, I could come up with examples which could easily generalise the method of study to rotations since rotation matrices include permutation matrices. 

We can look into the loss landscape of 2D matrix factorisation in this form UV^T=W, and it is obvious that any rotation of U on the RHS of it is an optimal solution as long as the same rotation is applied to V. The perspective from this optimisation problem is that it gives us a continuous plateau and it includes permutation of the dimensions. 

For neural networks with only one hidden layer. For example, consider a neural network in this form y=Uf(Vx) where U and V are parameter matrices and f() is a monotonic squashing function. It is easy to show that, when U is timed with a matrix R, where RR^T= I, as in U'= UR, there exist a V' and \alpha that gives R^Tf(Vx) = \alpha f(V'x) so that Uf(Vx) = URR^Tf(Vx) = U'f(V'x) for the hyperbolic tangent function. In the case where ReLU activation function is used as f(), then as long as the rotation doesn't produce negative entries, V' exists. In addition, when f is ReLU, isotropic scaling of the outputs from the first layer also gives rise to equivalent optima.  

Compared to the proposed study, the aforementioned way of studying the neural networks naturally gives continuous plateaus w.r.t. U in the loss landscape, and, by studying the discontinuity of the landscape w.r.t. V, more understanding could be unveiled. 
review_title: Official Blind Review #3

Meta-review: After communicating with each reviewer about the rebuttal, there seems to be a consensus that the paper contains a number of interesting ideas, but the motivation for the paper and the relationship to the literature needs to be expanded.  The reviewers have not changed their scores, and so there is not currently enough support to accept this paper.
Decision: Reject