Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper considers the problem of understanding the impact of deep neural networks (DNN) model architecture on the convergence rate of gradient descent dynamics. To achieve this goal, the paper follows the recent trend of continuous-time perspective of optimization, and proposes to model gradient descent via the gradient flow, which is a first-order ODE. The induced loss dynamics is then also following a first-order ODE with a coefficient matrix H (that depends on the solution trajectory, and hence non-constant). The paper then claims that the convergence rate is characterized by the minimum eigenvalue of H, and analyzes this H through a straightforward path-based formula obtained by chain rules. In particular, the authors try to explain the effect of width, depth and number of paths on the convergence rate, and validated these through a few numerical experiments.

Admittedly, the idea of this paper is interesting. However, I think the novelty and rigorousness of this paper is not convincing, as explained in more details below.

1. On the novelty side, characterizing the convergence via the H matrix is not new, and most of the discussions in Section 4 have appeared in exactly the same form in the previous works [13] (two-layer) and [25] (general), which are also cited in this paper. In addition, the path formula is also very closely related (if not completely the same as) with the expansion of G matrix in Section 4 of [25] (where G is the H in this paper), which decomposes the G (or H) matrix into summation over layers. These facts largely lower the contribution of this paper on a high level. 

2. On the rigorousness side, the paper is not very consistent in the notation. 
1) The notation is not very consistent. The authors use H in the notation section to denote the matrix, use X_i to denote the data input, but use \bf{H} and x_i (lower cased) subsequently. 
2) In Section 4, the authors immediately start with the continuous-time perspective, without even mentioning the gradient dynamics or some related ansatz. The authors may want to mention that they use the ansatz w_k=W(kh), where W is a smooth curve, and take h to 0 to obtain the ODE models, as is done in [1].
3) The notation list at the beginning of Section 4.3 is too long and not clear. In particular, l(p) is not even defined before appearing in (14), and \sigma_s seems to be overridden by the notation activation(w_s, X_i) and does not appear later in the path gradient, which is weird. Shouldn't there be \sigma_s inside the formula of (14)?

3. Again on the rigorousness side, the paper is very non-rigorous when stating the claims and theorems.
1) The authors claim that "This result will hold for other l_p losses", but indeed what holds is different for l_p losses (the rate is scaled by p/2).
2) Section 4.2 does not make any sense. The authors should either directly cite the corresponding content in [25], which are much clearer, or directly invoke the standard linear ODE theory and use the matrix exponential and Taylor expansion to make the explanations.
3) The paper seems to use a very informal argument that H stays close to its initial value to derive all the theory, which is only empirically checked in the appendix. But given the proportion of the theory part of this paper, I think the paper should either clearly state the assumption as H being constant, or follow the manner of [25] to prove that H stays close to some fixed matrix and use this to prove the other theorems rigorously. Otherwise, the theory part is both hard to understand and verify. 
4) Proposition 5.1 should clearly state whether the statement is in the expectation sense, or high probability sense, or something else. 
5) The explanation in Section 5.2 is rather unclear. In particular, I don't understand why "eigenvalues of H(g) are pushed in the direction of g" implies that "the updates prefer the direction of u", and why this then further implies something related to the momentum acceleration. The authors should provide a rigorous statement here.
6) The authors claim at the bottom of page 1 that they show adding a new layer leads to H(t) being decomposed into an adaptive learning rate term and a momentum term. But the adaptive learning rate part is not showing anywhere later in the paper. 

Minor comments:
1) In equation (5), there should be a minus sign on the right-hand side.
2) In Section 5.2, there is no "Equation 5.2". It should be something else.
3) The authors may want to add citations to [2] (which is a concurrent work with [25] on essentially the same topic) and [3] (which is a predecessor work of neural ODE).

[1] Su, Weijie, Stephen Boyd, and Emmanuel Candes. "A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights." Advances in Neural Information Processing Systems. 2014.
[2] Allen-Zhu, Zeyuan, Yuanzhi Li, and Zhao Song. "A convergence theory for deep learning via over-parameterization." arXiv preprint arXiv:1811.03962 (2018).
[3] Lu, Yiping, et al. "Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations." arXiv preprint arXiv:1710.10121 (2017).

################## post rebuttal ##################
After reading the authors' rebuttal, I decide to raise my rating to 3 (weak reject).
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper studies the training dynamics of a neural network model as a dynamical system.  The authors proposed a path-based approach to compute the derivatives that would appear in the H matrix which governs the learning dynamics.  They further utilized this formulation to (1) simplify the analysis of convergence rate of 2-layer neural networks w.r.t. width; and (2) presented an argument for the similarity between added depth in the network and momentum-based optimization; and (3) also argued about the importance of the number of paths for fast convergence.

The dynamical systems view of the learning process is quite new to me, even though I’m aware of a few recent papers exploring this.  I found the paper to be mostly clear and not too hard to follow, and the dynamical systems perspective interesting.  The path gradient is quite intuitive, but I’m a bit surprised there’s no prior work (at least not discussed in this paper) studying the relationship between gradients and the paths in the network.

It is a bit hard for me to judge the significance of this work because of my lack of context.  The main implications of the path gradients were presented in section 5.  The first part shows that using their theorem 4.1 can simplify the derivation of the linear relationship between convergence rate and the width of 2-layer nets.  This is a simplification but the original derivation is not complicated either.  The second part tried to draw a relationship between added depth in a network and momentum-based optimization, which I found to be a bit hand-wavy.  In particular, I found the argument that, du/dt prefers the direction of u itself which carries information about past directions therefore this is similar to momentum, to be not convincing.  The third implication argues that we need H to be full-rank for fast convergence, and in order to achieve this we want to increase the number of paths in the network, which is interesting.

There are a few other things that could be clarified:
- the terminology in section 4.3 is a bit confusing, w_s seems to be associated with an edge in the graph, but is also referred to as a node, “we denote the activation value at node w_s with activation(w_s, X_i) ”.
- on page 6 there is a reference to Equation 5.2 which does not exist
- the results in Figure 4a is counter-intuitive - is this showing wider networks actually converge slower?  Isn’t this against the argument of this paper?
- it is unclear how the convergence rate lambda values are computed in Figure 4b, the curves in Figure 4a clearly doesn’t follow an exponential decay pattern.

Overall I found this paper presented some interesting ideas, but may need a bit more work to be ready to be published.  Happy to change my judgement however, if other more experienced reviewers can comment better on the significance of this work.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents a simple and intuitive interpretation of the dynamics of gradient descent between labels and predictions rewritten in terms of all possible paths from inputs to outputs in FC networks. The basic starting point is from the gradient flow (gradient descent in continuous time) by Du et al 2018 [13] on the difference y-u of labels y predictions u (eq (6) or (7) in the paper) instead of the standard gradient flow dW/dt = - ∂L(W)/∂w. The coefficient matrix H of the system is known to determine the convergence properties, and this can be rewritten with respect to path-wise sums of gradients through the chain rule (Theorem 4.1).  This provides some intuitive interpretations for several facts discussed in the community: 1) the linear convergence rate is more intuitively obtained compared to the naive derivations (Remark 5.2 with comparison to [13]). 2) the 'depth' of FC network affects the convergence like momentum (section 5.2). 3) the number of paths compared to the number of nodes has a predominant impact on the convergence (section 5.3). The paper also demonstrates several experiments to understand the impact of depth or paths on the convergence.

Though the paper's contribution is a quite simple path decomposition using chain rules for the coefficient matrix H of the gradient flow, it indeed provides several intuitive understandings on the impact of paths onto the gradient-descent convergence. All implications are basically confirming already known things in different (path-based) words, and the impact or novelty is rather small, but nevertheless, it would be informative. 

One concern is about the description in section 4.1 and 4.2 for preliminaries results. I would suggest that emphasizing the fact 4.1 as an already discussed fact makes easy for readers to follow the results and focus more on the paper's contribution. The cited paper by Du et al [13] was published at the last year's ICLR, and the contents in 4.1 and 4.2 would be due to it including the formulation as gradient flow (gradient descent in continuous time). Given that it is already known that H is the determining factor for the convergence, then the contribution and usefulness of the proposed new representation of eq (15) should be discussed. At the first glance, the use of gradient flow w.r.t prediction u instead of parameter w might be (misleadingly) seen as the idea of this paper. The sentences in the abstract can also mislead readers to this. Also check the paper by Du et al in ICML 2019 'Gradient descent finds global minimum of deep neural networks'. 

Two minor points: 
1) parentheses for citing references and eq numbers are quite confusing. Please check the style format for citations.
2) in Figure 2, pre(w_0) is written as "presum(w_0)".

review_title: Official Blind Review #3

Meta-review: This paper focuses on understanding the role of model architecture on convergence behavior and in particular on the speed of training. The authors study the gradient flow of training via studying an ODE's coefficient matrix H. They study the effect of H in terms of possible paths in the network. The reviewers all agreed that characterizing the behavior in terms of path is nice. However, they had concerns about novelty with respect to existing work on NTK. Other comments by reviewers include (1) poor literature review (2) subpar exposition and (3) hand-wavy and rack of rigor in some results. While some of these concerns were alleviated during the discussion. Reviewers were not fully satisfied.  I general agree with the overall assessment of the reviewers. The paper has some interesting ideas but suffers from lack of clarity and rigor. Therefore, I can not recommend acceptance in the current form.
Decision: Reject