Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This work analyzes the continuous-time dynamics of gradient descent when training two-layer ReLU networks (one input, one output, thus only one layer of ReLU units). The work is interesting in the sense that it does not involve some unrealistic assumptions used by previous works with similar goal. Most importantly, this work does not assume independence between input and activations, and it does not rely on noise injection (which can simplify the analysis). Nonetheless, removing these simplifying assumptions comes at the expense of limiting the analysis to:
1. Only one layer of nonlinear units
2. Discarding the bias term in ReLU while keeping the input Gaussian (thus constant input trick cannot be used to simulate the bias term).
3. Imposing strong assumption on the representation on the input/output via (bias-less) ReLU networks: existence of orthonormal bases to represent this relationships.

Having that said, as far as I can tell, the paper presents original analysis in this new setting, which is interesting and valuable. For example, by exploiting the symmetry in the problem under the assumption 3 I listed above, the authors are able to reduce the high-dimensional dynamics of the gradient descent to a bivariate dynamics (instead of dealing with original size of the parameters). Such reduction to 2D allows the author to rigorously analyze the behavior of the dynamics (e.g. convergence to a saddle point in symmetric case, or to the optimum in non-symmetric case).

Clarification Needed: first paragraph of page 2. Near the end of the paragraph you say "Initialization can be arbitrarily close to origin", but at the beginning of the same paragraph you state "initialized randomly with standard deviation of order 1/sqrt(d)". Aren't these inconsistent?

Some minor comments about the draft:
1. In section 1, 2nd paragraph: "We assume x is Gaussian and thus the network is bias free". Do you mean "zero-mean" Gaussian then?
2. "standard deviation" is spelled "standard derivation" multiple times in the paper.
3. Page 6, last paragraph, first line: Corollary 4.1 should be Corollary 4.2

review_title: Optimization of a ReLU network under new assumptions

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a convergence analysis of some two-layer NNs with ReLUs. It is not the first such analysis, but maybe it is novel on the assumptions used in the analysis, and the focus on ReLU nonlinearity that is pretty popular in practice. 

The paper is quite hard to read, with many English mistakes and typos. Nevertheless, the analysis seems to be generally correct. The novelty and the key insights are however not always well motivated or presented. And the argument that the work uses realistic assumptions (Gaussian inputs for example) as opposed to other works, is quite debatable actually. 

Overall, the paper looks like a correct analysis work, but its form is really suboptimal in terms of writing/presentation, and the novelty and relevance of the results are not always very clear, unfortunately. The main results and intuition should be more clearly presented, and details could be moved to appendices for example - that could only help to improve the visibility and impact of these interesting results. 
review_title: Potentially new analysis, but hard to read

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the author analyzes the convergence dynamics of a single layer non-linear network under Gaussian iid input assumptions. The first half of the paper, dealing with a single hidden node, was somewhat clear, although I have some specific questions below. The second half, dealing with multiple hidden nodes, was very difficult for me to understand, and the final "punchline" is quite unclear. I think the author should focus on intuition and hide detailed derivations and symbols in an appendix. 

In terms of significance, it is very hard for me to be sure how generalizable these results are: the Gaussian assumption is a very strong one, and so is the assumption of iid inputs. Real-world feature inputs are highly correlated and are probably not Gaussian. Such assumptions are not made (as far as I can tell) in recent papers analyzing the convergence of deep networks e.g. Kawaguchi, NIPS 2016. Although the author says the no assumption is made on the independence of activations, this assumption is shifted to the input instead. I think this means that the activations are combinations of iid random variables, and are probably Gaussian like, right? So I'm not sure where this leaves us.

Specific comments:

1. Please use D_w instead of D to show that D is a function of w, and not a constant. This gets particularly confusing when switching to D(w) and D(e) in Section 3. In general, notation in the paper is hard to follow and should be clearly introduced.

2. Section 3, statement that says "when the neuron is cut off at sample l, then (D^(t))_u" what is the relationship between l and u? Also, this is another example of notational inconsistency that causes problems to the reader.

3. Section 3.1, what is F(e, w) and why is D(e) introduced? This was unclear to me.

4. Theorem 3.3 suggests that (if \epsilon is > 0), then to have the maximal probability of convergence, \epsilon should be very close to 0, which means that the ball B_r has radius r -> 0? This seems contradictory from Figure 2. 

5. Section 4 was really unclear and I still do not understand what the symmetry group really represents. Is there an intuitive explanation why this is important?

6. Figure 5: what is a_j ?

I encourage the author to rewrite this paper for clarity. In it's present form, it would be very difficult to understand the takeaways from the paper.
review_title: Hard to read paper; unclear conclusions.

final_score: rating: null
review_content: The assumption of Gaussian input X seems rather strong. Do you have a sense of how the analysis would change if the input features have non-Gaussian or multi-modal distributions (as is most likely the case with real-world input features)?  Are there any other implicit or explicit assumptions made in the paper?
review_title: Gaussian input assumption

Meta-review: The paper analyzes the dynamics of learning under Gaussian input using dynamical systems theory. As two of the reviewers have pointed out, the paper is hard to read, and not written in a way which is accessible to the wider ICLR community. Hence, I cannot recommend its acceptance to the main conference. However, I recommend acceptance to the workshop track, since it has nice technical contributions that can lead to interesting interactions. I encourage the authors to make it more accessible for a future conference.
Decision: Invite to Workshop Track