Title: MODEL ARCHITECTURE CONTROLS GRADIENT DE-SCENT DYNAMICS: A COMBINATORIAL PATH-BASED FORMULA Anonymous

Abstract: Recently, there has been a growing interest in automatically exploring neural network architecture design space with the goal of finding an architecture that improves performance (characterized as improved accuracy, speed of training, or resource requirements). However, our theoretical understanding of how model architecture affects performance or accuracy is limited. In this paper, we study the impact of model architecture on the speed of training in the context of gradient descent optimization. We use the ideas from prior work that shows gradient descent can be modeled as a first-order ODE and use ODE's coefficient matrix H to characterize the convergence rate. We introduce a simple analysis technique that enumerates H in terms of all possible "paths" in the network. We show that changes in model architecture parameters reflect as changes in the number of paths and the properties of each path, which jointly control the speed of convergence. We believe our analysis technique is useful in reasoning about more complex model architecture modifications.

1. INTRODUCTION:

Gradient descent and its variants are the cornerstones of deep learning. The theoretical properties of gradient descent have been widely studied in the literature; study of the convergence bounds and guarantees [6, 12, 15, 13] , the characterization of the local geometry of stationary points [21, 11, 10, 19, 25, 30, 24, 23, 13] , exploration of better algorithms to optimize the descent process [28, 16, 26, 33, 7 ] are just a few example of active research areas in this domain. Another major research area is the exploration of network architecture and its impact on performance. In recent years, network architecture search and design have shown major performance boosts in various aspects of deep learning [36, 35, 18, 8, 34] .

Recent theoretical results are trying to relate model architecture and gradient descent properties. It has been shown that over-parametrization in width guarantees convergence in deep neural network [15] . It has also been shown that increasing depth has an impact similar to adding momentum optimization and adaptive learning rate to the objective function [4] .

The gradient descent process can be described by a system of first-order differential equations in the continuous-time limit, in the form of ẋ(t) = H(t)x(t). In this paper, we study the properties of the coefficient matrix H(t), which governs the dynamics of the gradient descent. Specifically, we formulate H(t) in terms of all possible paths in the network. This representation is powerful as it enables us to analyze the gradient behavior symbolically and abstracts away the unnecessary bookkeeping of derivatives and weights. Specifically, we use this representation to explore the followings. (1) We study the impact of width on convergence. Using a simple path counting argument, we show that H(t) is a sum of m i.i.d. terms where m is dictated by the width of the network. This has been implied in the work of Simon Du, et al. [12] for the special case of a 2-layer RELU-activated network. We contrast the ease of arriving at such conclusions using our representation against methods used in previous work [12, 15, 13, 6] . (2) We study the impact of depth on convergence. While prior work [3] established the relationship between momentum and depth, we show a direct relationship between depth-induced momentum and the scale of the output value. We also discuss how depth-induced momentum is different than explicit momentum.

(3) We argue why the number of paths is more important than the number of nodes in a network.

2. RELATED WORK:

Related work can be categorized into three groups which we will explain in detail below. In general, the common theme across previous works is to show that if some conditions, such as bound on models' width or depth or assumptions on geometrical properties of the loss landscape are met, gradient descent or its variance converges to a global minimum.

Landscape Analysis Researchers have extensively studied the properties of error surface in linear neural network ( [10, 21] ) and have proved convergence to a global minimum for non-convex optimizations if error surface shows certain properties, such as all saddle points to be strict (i.e. there exists a negative curvature)( [11, 19, 25, 30, 24, 23, 13] ). However, it has been shown that the strict saddle property is not guaranteed for deep neural networks.

Width Analysis A number of recent work have characterized minimum width that guarantees convergence in terms of model architecture parameters and input data. We on the other hand study how convergence rate varies with width. Examples of finding width bounds include [15] where it has been shown that for a 2-layer ReLU-activated neural network with squared loss, as long as every hidden layer is wide enough, gradient descent converges to a global minimum at a linear rate. In [14, 1] , it is extended to fully connected linear networks and ResNets. Du et. al. [12] has also generalized this work to an L-layer fully connected linear neural network and shows the convergence rate as a function of depth, output dimension and least eigenvalue of the Gram matrix for large enough hidden layers. Ardalani et. al. [2] have shown empirically that for a wide range of RNN applications, increasing width will reduce the number of steps to minimum validation loss. Neural tangent kernel [22] studies convergence in the infinite width limit.

Depth Analysis Bartlett et al. [6] have proven for a deep linear neural network with isotropic input and identity initialization that the number of steps to -proximity of the best answer is polynomial in the number of layers. Neural ODE studies the convergence in the infinite depth limit [27, 9, 20, 17] . Work that motivated our depth analysis are [3, 4] . They have shown that over-parametrization accelerate training with these extra restrictions: (1) linear neural network with (2) fully-connected layers and (3) "balanced" initialization where (4) dimensions of hidden layers are at least the minimum of the input and output dimensions. We contribute in the following ways. We characterize depthinduced momentum in value space with an inequality and compare with explicit gradient descent momentum. We observe that momentum is directly impacted by value of the sub-network. We lift the structure and activation restrictions.

3. NOTATIONS AND CONVENTIONS:

Matrix entries are denoted with row and column indices, for instance H = (H ij ) is a matrix with entries H ij . When H is diagonalizable, we denote by λ min (H) its smallest eigenvalue. For a matrix W , W (i, :) and W (:, j) denote the row vector in the i-th row and the column vector in the j-th column, respectively. Write w∈W for summing over all elements of W . We use pairs {X i , y i }, 1 ≤ i ≤ N to represent labeled data where X i is input and y i is the output. We write x i,k for the k-th component of X i . We denote loss by L. We work with feed-forward neural networks, which we sometimes denote abstractly by a function f , or f (w, x) as a function of input x and weights w. We denote network weights individually by w i . Predictions are denoted by u i = f (w, X i ) 1 ≤ i ≤ N . The prediction on all inputs is denoted by the vector u. l p loss is defined as (y -u i ) p for even integers p ≥ 2. We use angle brackets to denote inner products, i.e. -→ v , -→ w = i v i w i represents inner product of two vectors. i.i.d. random variables stands for independent and identically distributed random variables. ODE stands for ordinary differential equations. NN stands for neural networks. GD stands for gradient descent.

4. BACKGROUND:

We build upon prior work [15, 12, 14, 3, 31] which take gradient descent to a continuous time limit and derive the Gram-matrix H governing gradient descent dynamics. In this section, we give a quick derivation for Equation (4) for completeness.

Convergence rate can be characterized as the rate of change in loss during the training process. If steps are infinitesimally small, the rate of change in loss can be characterized as:

EQUATION

Where u i (t) = f (w(t), X i ) is the prediction on input X i at time t. For simplicity we focus on the dynamics of du i dt , which can be expanded as follows:

EQUATION

We use the definition of gradient descent and chain rule to derive the equation above. Gradient descent at infinitesimal small step is defined by:

EQUATION

Observe that -→ ∂L ∂u is a column vector. Hence, if we define H ij as:

EQUATION

Then we can simplify equation ( 2) in a vector form as:

EQUATION

Assuming L is l 2 loss, equation ( 5) simplifies to the following. This result will hold up to a constant for other l p losses (see Appendix A).

EQUATION

EQUATION

Notice that this puts yu in a system of differential equations whose single variable analogue is:

EQUATION

The above equation's solution is:

EQUATION

when h < 0, e h < 1 and let e h = 1 -ξ. ξ governs the convergence rate in Equation ( 9). In the actual system of equations ( 7), ξ will be determined by the minimum eigenvalue of H(t), denoted λ min (H(t)).

5. MAIN IDEA: CHARACTERIZING H IN TERMS OF NETWORK PATHS:

In order to state the path decomposition formula, we first establish some notations.

• We denote the neural network abstractly with a function f (X).

• For any neural network, we view the underlying network as a directed graph where edges are in the backward propagation direction.

• We define path p, as a series of connected nodes and directed edges. We define an output path as a path that starts with the output node. We denote the length of p by l(p).

• We index the nodes as follows. Index the output node by 0, the one after by 1, and so on. We denote the j-th node in the i-th path by p i j .

• We allow different activation functions at nodes. We denote the activation function at p s by σ s .

• An example path that ends in an input node may look like this: • The post-subnetwork at w s is defined as the network formed by the subgraph rooted at the right vertex of w s . The pre-subnetwork at w s is defined as the union of all output paths ending in w s , denoted by pre(w s ).

• For every input X i , we denote the activation value at the node after w s with f [w s ](X i ). Note that this is equivalent to the value at the right vertex of w s once X i propagated through the post-subnetwork at w s .

• We denote the local gradient wrt. input at node s along the path p with τ i ps .

• Given the pre-subnetwork at w s , we write the explicit formula for ∂ui ∂ws :

EQUATION

where path_gradient p (w s , X i ) is the gradient value along the path p at the left vertex of w s , and can be expanded as follows using the chain rule:

EQUATION

we refer to path_gradient p (w s , X i ) as PG p (w s , X i ) for brevity from now on.

Theorem 5.1. For a network f ,

EQUATION

which easily follows by substituting Equation (10) in Equation ( 4). In the rest of the paper, we will make many arguments most of which rely on the number of terms in this equation. Number of terms in Equation 12is itself controlled by the number of paths in the pre-subnetworks at each weight. Following examples provide some intuition.

Example 5.2. In this example, our network has a structure as illustrated in Figure 2a . The output node is denoted with value one, indicating that the activation function at output layer is an identity function. As shown in Figure 2b , there are only one path in each weight's pre-subnetwork. Therefore, there are only three terms in Equation 12, where each summand is of form PG p (w, X i )PG p (w, X j ).

We refer to these terms as symmetrical pairs since both multipliers are representing the same path.

Example 5.3. A more complicated application of the formula can be found in Figure 2d . There is one path in each of the following weights' presubnetwork: w 0 ,w 1 ,w 2 ,w 3 , and w 4 , and three paths at w 5 's presubnetwork. Therefore, Equation 12would have 5 + 3 2 = 14 summands, 8 of which are terms with symmetrical pairs and 6 are cross-terms induced by the paths in w 5 's pre-subnetwork. In general, cross-terms will appear only if there are two paths that share a final edge which can only happen if there are three or more number of layers in the network. 6 ARCHITECTURE IMPLICATIONS

6.1. CONVERGENCE RATE SCALES LINEARLY WITH WIDTH:

In this Section, we show that for sufficiently wide neural networks, convergence rate has a linear relationship with the network's width. Results are drawn theoretically at time 0 assuming random i.i.d initialization. For these results to generalize across time, we assume that H stays close to its initial value. In the Appendices D and E, we show empirical results that suggest this assumption is true.

For the extreme case of infinitely-wide networks, it has been shown that H is constant throughout training [22] . Others [5, 15] have also shown that for sufficiently-wide neural networks weights remain close to initialization, and so our argument holds true across training.

Proposition 6.1. For a 2-layer neural network with hidden dimension m, the expected convergence rate scales linearly with m.

To demonstrate the simplicity and expressiveness of our approach, next we will discuss proofs with and without our approach.

Brute-force approach A 2-layer fully-connected linear neural network can be modeled as:

EQUATION

where V ∈ R 1,m and W ∈ R m,d are the weights of the second and first layer, respectively and the input X is a d-dimensional column vector. We begin the analysis with the simple case of a 2-layer linear network where the network's width is one, i.e. m = 1 and u(X) = v 0 • W (:, 1), X .

H ij can be decomposed into two terms, partial derivatives with respect to weights of the first layer and partial derivatives with respect to weights of the second layer.

EQUATION

For a more general case where the width is m, we have:

EQUATION

Adding non-linearity would make this equation even more complex, which is the approach used in previous literature together with induction:

EQUATION

) See Appendix B for details of how we derive these equations. Since we assume all weights (w lk , v 0 ) are independent and identically distributed, the summands within Equations ( 15) and ( 16) are also independent and identically distributed. This indicates that H ij is composed of m i.i.d components, hence convergence rate scales linearly with m.

Our approach: Using Theorem 5.1 More generally, by applying Theorem 5.1, it is immediate to see every pre-subnetwork will be a path in a 2-layer fully connected network. There are two types of paths; m length 1 paths and m length 2 paths. The weights are independent and follow the same distribution, hence, a 2-layer network's H matrix always decomposes as m i.i.d sums. Hence convergence rate scales linearly with m. This proves Proposition 6.1. See Section 7 for empirical analysis that motivated this study. We show for a 2-layer linear and non-linear networks that convergence rate has linear relationship with width.

6.2. DEPTH-INDUCED FIRST-ORDER MOMENTUM:

In this section, we use our formula to explain how depth has a momentum-like effect. Take a network f . Construct a new network g by adding a multiplication layer on top, i.e. g(X) = µf (X) with an extra trainable parameter µ. We first present an inequality (with the proof left to Appendix F), then explain why the inequality implies momentum-like behavior and finally how they are different. Proposition

EQUATION

where cos denotes cosine between two vectors. Remark 6.3 (Momentum interpreted as directional preference). In physics, momentum is defined as the tendency of an object to follow its current velocity of motion. In gradient based optimization, momentum can be defined as in Equation ( 1) of [32] ,

EQUATION

where v's are the updates, is the learning rate and m 0 is the momentum coefficient. Quantitatively, cos(v t+1 , v t ) increases with m 0 . A bigger m 0 implies a stronger preference in v t+1 for the direction of v t . For network depth, Proposition 6.2 describes the stronger directional preference of the deeper network g in the direction of the current value vector. In the depth-induced momentum experiment below, we show that this directional preferences can have similar effects on convergence rates as momentum in gradient based optimization. Remark 6.4 (Difference to momentum). The key difference between depth-induced momentum and gradient descent momentum is the following. The depth-induced momentum is a condition between u and u, whereas gradient descent momentum is a second order condition between u and u (see [29] Eq. ( 4) and ( 10)). 2 , depth-induced momentum will be more pronounced when u can increase. In Appendix G, we present an experiment that shows that depth-induced momentum is more pronounced when output scale is larger.

6.3. ON THE IMPORTANCE OF NUMBER OF PATHS FOR CONVERGENCE:

Here, we will provide some heuristic arguments, based on our formula, on why over-parametrization through increasing the number of paths can be more effective than increasing the number of nodes (parameters) and conclude that increasing depth is more effective than increasing width for accelerating the training process. We first introduce an important terminology which we will use below. Definition 6.6. Given a collection of vectors {g i }, the Gram matrix G is defined as a matrix whose ij-th entry is g i , g j under the Euclidean inner product. Remark 6.7. Based on the definition above it is easy to see that Gram matrix is symmetric and positive-semidefinite and its rank is exactly the dimension of the subspace spanned by {g i }. Suppose H = H 1 + • • • + H n are all Gram matrices. Then H is positive definite if any of the H i is positive definite. This is because of supperadditivity of the minimum eigenvalue function, i.e. λ min (H) ≥ λ min (H 1 ) +

• • • + λ min (H n ).

Strongly-Gram vs. Weakly-Gram Equation( 12) in Theorem 5.1 can be expanded further as follows and the terms can be classified into two groups. The first group will include all the symmetrical pairs and the second group will include all the cross-terms.

EQUATION

)

EQUATION

We refer to the first group as strongly Gram, and denote it by H S ij (f ), and refer to the second group as weakly Gram and denote it by H W ij (f ), i.e. we have: Remark 6.8 (Gramness of the strongly-Gram component). Suppose there are q paths in the network, p 1 , p 2 , . . . , p q . Then Equation (19) implies that the strongly Gram component is Gramian with vectors

EQUATION

Remark 6.9 (Insignificance of the weakly Gram component). We claim that the weakly Gram component is insignificant, see Appendix C for heuristics and empirical evidence.

Remark 6.10 (Number paths vs. number of nodes). The Gram matrix H S should ideally be of full rank for good convergence. So we want the subspace spanned by {g i } to have dimension at least N . N is the dimension of H and also the number of sample points. On the other hand, as shown in Equation 21 g i 's are embedded as vectors in a q-dimensional vector space where q is the number of paths.Therefore, to ensure H is full-rank, q should satisfy q ≥ N . The bigger q is, the more likely they are linearly independent and make the matrix full rank. In deep learning, N is usually large, therefore, we need to have networks whose number of paths is comparable to N for best convergence.

Remark 6.11 (Depth vs. width). Adding a new parameter at a new layer can increase the number of paths more effectively than adding it at an existing layer. In other words, making models deeper increases the number of paths faster than making models wider.

7. EXPERIMENTS:

In this section, we provide experimental results that demonstrate the impact of model structure on convergence.

Setup We employ a teacher-student framework to put our ideas to test. Our teacher model is a two-layer neural network with hidden dimension 100, input dimension 50 and output dimension 1. We assume a Gaussian distribution both for the inputs and ground-truth weights. The student models are also two-layer neural networks with variable hidden dimensions, and l 2 loss. The convergence rate at step t is defined as 1 -Losst+1 Losst . 0 2 0 0 0 0 4 0 0 0 0 6 0 0 0 0 8 0 0 0 0 1 0 0 0 0 0 1 2 0 0 0 0 1 4 0 0 0 0 #Steps (#Gradient Updates) Two-Layer Linear Neural Network As shown in Figure 3a , training curve in log scales look like straight lines. As shown in Figure 3b , convergence rate (λ) grows linearly with width.

Two-Layer Non-Linear Neural Network As shown in Figure 4a , training curve for models with the sigmoid non-linearity begins in a steep region where loss drops very quickly for a short period of time and ends in a steady region where loss drops very slowly. We are interested in characterizing the curve in the steady region. As shown in Figure 4b , convergence rate (λ) grows linearly with width.

Depth induced momentum Figure 5 compares the convergence rate curves for 2-layer, 3-layer and 4-layer NN using GD, against a 2-layer network with a momentum optimizer. The deeper networks are constructed from 2-layer networks with additional scalar multiplication layers. As discusses in Section 4.5, one can observe depth-induced momentum in convergence rates, similar to using momentum optimizer with a 2-layer NN.

8. CONCLUSION:

In this paper, we provide a path-based combinatorial formula for the coefficient matrix H which governs the gradient descent dynamics. We advocate analyzing H with this path-based approach by showing the insights we could derive from this.

Figure 5 : Depth induced momentum.

A l p LOSS

We argue very briefly that without loss of generality, we can assume that we work with l 2 loss instead of other L p norms. For l p loss, ∂L ∂u j = p • (y j -u j ) p-1

So for l 2 loss, Equation 5becomes

EQUATION

Recall the argument for decay rate under Equation 8. The single variable analogue of the system is:

EQUATION

the solution looks like:

EQUATION

and decays like:

EQUATION

EQUATION

When h is small, by Taylor expansion

EQUATION

Instead of using the closed form solution, this ODE could equivalently be solved using the series method and we would be getting the Taylor expansion of equation ( 25) on the right and equation ( 28) remains the same. Note that the series method (more fundamentally, Taylor expansion involving only one matrix) works for both ODEs and systems with convergent assumptions on the coefficient matrix H. ηH is small by assumption, so we have convergent Taylor expansions, so we continue to argue for l p loss as if we are in the single variable case.

For general l p loss, we have:

EQUATION

EQUATION

and the decay rate would be h p , scaled down by p, so all the convergence rate statements we make for the rest of the paper for l 2 remains true if scaled by p 2 .

B 2 LAYER LINEAR NETWORK:

The detailed brute-force calculation of H ij for a 2-layer fully connected linear network with m = 1 is as follows. Recall m is the width of the hidden layer. For a general m, we have:

EQUATION

EQUATION

C WEAKLY GRAM MATRIX IS EMPIRICALLY INSIGNIFICANT:

In terms of network architecture, the weakly Gram term would only appear when two paths share a final edge. That means the network needs to have at least 3 layers. The number of pairs could grow quadratically with width. Heuristically, each term E[PG p1 (w, X i ) • PG p2 (w, X j )] tend to be 0 because of independent weights in PG and the weights having expectation 0. Hence, we focus on the strongly Gram part for the rest of the section. As in Equation ( 19), the number of terms in weakly Gram component grows quadratically in the width of the hidden layer. One might expect this results in a quadratic growth in convergence, given all the terms are i.i.d. random variables and strictly positive. In order to test this hypothesis, we exploit a teacher-student framework as outlined in Section 7, however the student models are 3-layer RELU-activated networks to ensure the number of terms in weakly Gram component is non-zero. We choose RELU activation function to ensure that the expected value of the activation to be greater than zero. Our experimental results shows that the convergence rate relationship with width is linear (see Figure 6 ) and not quadratic. This implies that weakly Gram components have expectation zero.

D JUSTIFY H STAYS CLOSE TO INITIALIZATION:

Weights and H are indeed time-dependent and follow their respective stochastic process. However, theoretical results in the overparametrized regime suggest that every weight vector remains close to initialization (Lemma 5.3 [5] and Lemma 3.3 [15] ). We also observe that the mean and variance stay very close to initialization, N (0, 0.1) (Figure 7 ). In Figure 8 , we ran the Kolmogorov-Smirnov test to compare the weight distribution against the initialization distribution. As shown, the p-values are overall stable and far from significance levels to invalidate the null hypothesis (H 0 : trained weights follow N (0, 0.1)). Mean of Weights, Linear Network 

E JUSTIFY IID ASSUMPTION FOR WEIGHTS:

The Turning Point Test is commonly used to check iid-ness. Figure 9 shows the p-value through the training process. As shown, p-values do not exhibit any major regime shifts and the null hypothesis (the weights are iid samples) stays far from rejectable with any commonly used significance levels (0.1, 0.05 . . . ). iid-ness can also be seen as a consequence of closeness to initialization in response 1.

By staying close to initialization, statistical properties are preserved and the weights remain an iid sample of the original distribution.

F PROOF OF PROPOSITION 6.2

Proof. By definition of f and g, in g, there's a new path of length 1 whose edge is µ and all paths in f are annexed by µ at the front. Hence H ij for the new network g can be computed in terms of H ij The number after "std" represents the output's standard deviation in the teacher network, equivalently scale as the mean is 0. As we can observe from the graph, momentum acceleration is not present for small standard deviation (0.06 and 0.12) and the behavior is more similar to a shallow 2-layer network of the same width. When standard deviation is raised to 0.24, then we observe acceleration of the convergence rate. In all of these experiments, µ is initialized to 1 and empirically stays close to 1 to counteract the effect of µ 2 in equation ( 34).

:



Figures and Tables:

Figure 1: Illustration of a multilayer perceptron

Figure 2: Path Decomposition Examples.

Sensitivity to output scale). Since u , u =

Figure 3: Training Curve Characterization for Two-layer Linear Neural Network.

Figure 4: Training Curve Characterization for Two-layer Non-Linear Neural Network.

Figure 6: Convergence Rate for a 3-layer RELU-activated Network.

Figure 7: Mean and std across the training process. (X-axis is the number of steps)

Figure 9: Turning point test p-values

Figure 10: Momentum's dependence on output scale.

Peer review (iclr2020):
This paper considers the problem of understanding the impact of deep neural networks (DNN) model architecture on the convergence rate of gradient descent dynamics. To achieve this goal, the paper follows the recent trend of continuous-time perspective of optimization, and proposes to model gradient descent via the gradient flow, which is a first-order ODE. The induced loss dynamics is then also following a first-order ODE with a coefficient matrix H (that depends on the solution trajectory, and hence non-constant). The paper then claims that the convergence rate is characterized by the minimum eigenvalue of H, and analyzes this H through a straightforward path-based formula obtained by chain rules. In particular, the authors try to explain the effect of width, depth and number of paths on the convergence rate, and validated these through a few numerical experiments. Admittedly, the idea of this paper is interesting. However, I think the novelty and rigorousness of this paper is not convincing, as explained in more details below. 1. On the novelty side, characterizing the convergence via the H matrix is not new, and most of the discussions in Section 4 have appeared in exactly the same form in the previous works [13] (two-layer) and [25] (general), which are also cited in this paper. In addition, the path formula is also very closely related (if not completely the same as) with the expansion of G matrix in Section 4 of [25] (where G is the H in this paper), which decomposes the G (or H) matrix into summation over layers. These facts largely lower the contribution of this paper on a high level. 2. On the rigorousness side, the paper is not very consistent in the notation. 1) The notation is not very consistent. The authors use H in the notation section to denote the matrix, use X_i to denote the data input, but use _x0008_f{H} and x_i (lower cased) subsequently. 2) In Section 4, the authors immediately start with the continuous-time perspective, without even mentioning the gradient dynamics or some related ansatz. The authors may want to mention that they use the ansatz w_k=W(kh), where W is a smooth curve, and take h to 0 to obtain the ODE models, as is done in [1]. 3) The notation list at the beginning of Section 4.3 is too long and not clear. In particular, l(p) is not even defined before appearing in (14), and sigma_s seems to be overridden by the notation activation(w_s, X_i) and does not appear later in the path gradient, which is weird. Shouldn*t there be sigma_s inside the formula of (14)? 3. Again on the rigorousness side, the paper is very non-rigorous when stating the claims and theorems. 1) The authors claim that *This result will hold for other l_p losses*, but indeed what holds is different for l_p losses (the rate is scaled by p/2). 2) Section 4.2 does not make any sense. The authors should either directly cite the corresponding content in [25], which are much clearer, or directly invoke the standard linear ODE theory and use the matrix exponential and Taylor expansion to make the explanations. 3) The paper seems to use a very informal argument that H stays close to its initial value to derive all the theory, which is only empirically checked in the appendix. But given the proportion of the theory part of this paper, I think the paper should either clearly state the assumption as H being constant, or follow the manner of [25] to prove that H stays close to some fixed matrix and use this to prove the other theorems rigorously. Otherwise, the theory part is both hard to understand and verify. 4) Proposition 5.1 should clearly state whether the statement is in the expectation sense, or high probability sense, or something else. 5) The explanation in Section 5.2 is rather unclear. In particular, I don*t understand why *eigenvalues of H(g) are pushed in the direction of g* implies that *the updates prefer the direction of u*, and why this then further implies something related to the momentum acceleration. The authors should provide a rigorous statement here. 6) The authors claim at the bottom of page 1 that they show adding a new layer leads to H(t) being decomposed into an adaptive learning rate term and a momentum term. But the adaptive learning rate part is not showing anywhere later in the paper. Minor comments: 1) In equation (5), there should be a minus sign on the right-hand side. 2) In Section 5.2, there is no *Equation 5.2*. It should be something else. 3) The authors may want to add citations to [2] (which is a concurrent work with [25] on essentially the same topic) and [3] (which is a predecessor work of neural ODE). [1] Su, Weijie, Stephen Boyd, and Emmanuel Candes. *A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights.* Advances in Neural Information Processing Systems. 2014. [2] Allen-Zhu, Zeyuan, Yuanzhi Li, and Zhao Song. *A convergence theory for deep learning via over-parameterization.* arXiv preprint arXiv:1811.03962 (2018). [3] Lu, Yiping, et al. *Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations.* arXiv preprint arXiv:1710.10121 (2017). ################## post rebuttal ################## After reading the authors* rebuttal, I decide to raise my rating to 3 (weak reject).