Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper shows that for ReLU networks satisfying certain conditions, the weights and biases leading to the exact functional form are the ones obtained by neuron permutations and rescalings, and that no other reparametrizations preserving the function exist. The authors provide an explicit algorithm applying these symmetries.

Disclaimer: My knowledge in this particular subfield is limit and I therefore cannot assess the novelty of the approach.

I find the topic very interesting and the authors’ approach reasonable. I appreciate that they clearly qualify the assumptions used. The apparent tensions between the intuition that many different reparametrizations can exist and their result suggesting that in fact only very few weight space points lead to the same function is also discussed in the conclusion, which I really appreciate.

It would be interesting to study the regime where the functions are not exactly the same. In particular, this is very relevant because we only care about answers on a discrete set of points (train set / test set), and on top of that we only care about the argmax of the logits, rather than the actual detailed answer. I believe such a result would be significantly stronger and more relevant to the practical applications of DNNs, however, I understand that it might be more difficult to obtain.

Overall, I enjoyed this paper and I think it deals with an important problem.

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper proves that, modulo permutation and scaling, ReLU networks with non-increasing widths are uniquely characterized by the function they induce (excepting some degenerate cases).  This result is not apriori obvious and is of interest.

Authors are commended for balancing brevity, intelligibility, and precision.  However, it is not clear which elements of the proof technique are inapplicable to leaky ReLUs.  It would be helpful to include a brief discussion on this.

I recommend acceptance, since I didn't find any proof errors and the contribution is clear.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 
In this paper, the authors studied the equivalence class of ReLU networks with non-increasing weights, and proved that permutation and scaling are the only function preserving weight transformations. The proof technique is novel, and provides some insights in the geometry space of the loss surface. I think the proof technique could have its general implications to some other research, however, the direct value of this paper is not very clear.

1)	The paper starts with the discussions on the redundancy introduced by over-parametrization, as one of its motivations. However, what is discussed in this paper is actually far distant from over-parametrization. The redundancy in over-parametrized networks and the redundancy in ReLU networks are different concepts and the connection between them is not well established. The over-parametrization is talking about the smooth information flow brought by wide intermediate layers, however, the equivalence class in this paper is more about the mathematical properties of the activation functions and the topological connections of the neural networks. I feel that the authors have some confusing understanding on these two concepts.

2)	The theory was established regarding restrictive types of ReLU networks (feedforward, with non-increasing width). However, many widely used networks are not of this kind. Without extensions to other shapes of the networks, convolutional and recurrent structures, and many kind of normalization transformations (e.g., batch norm and layer norm), the practical value of this paper is limited. 

3)	Some important references are missing. The following paper has in-depth theoretical analysis on ReLU networks, and characterizes the redundancy brought by positive-scale invariant properties of ReLU networks. It is strange that the authors did not cite it.  It would be necessary for the authors to discuss their additional technical contributions given this ICLR 2019 paper.
-  Q. Meng, et al. G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space, ICLR 2019

4)	The practical implication of the theories in this paper is not clearly discussed. What if we know there are only these two kinds of redundancies? What kind of new algorithms and practices can be inspired by such theoretical understandings.


** I read the author rebuttal. Different people may have different criteria on evaluating a paper and my criterion is not only about "what", but more importantly about "so what". I still think the authors should think harder about the implications of their work, either theoretically or practically. Furthermore, I understand that some published papers also use restricted settings to ease their proofs, however, I personally do not think this is well justified. I think for top conference like ICLR, more solid and less restrictive works are preferred.  I could at most adjust my score to weak reject.
review_title: Official Blind Review #1

Meta-review: This work proves that the weights of feed-forward ReLU networks are determined, up to a specified set of symmetries, by the functions they define. Reviewers found the paper easy to read and the proof technically sound. There was some debate over the motivation for the paper, Reviewer 1 argues that there is no practical significance for the result, a point that the authors do not deny. I appreciate the concerns raised by Reviewer 1, theorists in machine learning should think carefully about the motivation for their work. However, while there is no clear practical significance of this work, I believe there is value to accepting it. Because the considered question concerns a sufficiently fundamental property of neural networks, and the proof is both easy to read and provides insights into a well studied class of models, I believe many researchers will find value in reading this paper.
Decision: Accept (Poster)