Title: Symmetry-Breaking Convergence Analysis of Certain Two-layered Neural Networks with ReLU nonlinearity
Authors: Yuandong Tian
Venue: ICLR 2017
Keywords: Theory
Deep learning
Optimization

Abstract: In this paper, we use dynamical system to analyze the nonlinear weight dynamics of two-layered bias-free networks in the form of $g(x; w) = \sum_{j=1}^K \sigma(w_j \cdot x)$, where $\sigma(\cdot)$ is ReLU nonlinearity. We assume that the input $x$ follow Gaussian distribution. The network is trained using gradient descent to mimic the output of a teacher network of the same size with fixed parameters $w*$ using $l_2$ loss. We first show that when $K = 1$, the nonlinear dynamics can be written in close form, and converges to $w*$ with at least $(1-\epsilon)/2$ probability, if random weight initializations of proper standard derivation ($\sim 1/\sqrt{d}$) is used, verifying empirical practice. For networks with many ReLU nodes ($K \ge 2$), we apply our close form dynamics and prove that when the teacher parameters $\{w*_j\}_{j=1}^K$ forms orthonormal bases, (1) a symmetric weight initialization yields a convergence to a saddle point and (2) a certain symmetry-breaking weight initialization yields global convergence to $w*$ without local minima. To our knowledge, this is the first proof that shows global convergence in nonlinear neural network without unrealistic assumptions on the independence of ReLU activations. In addition, we also give a concise gradient update formulation for a multilayer ReLU network when it follows a teacher of the same size with $l_2$ loss. Simulations verify our theoretical analysis.

TL;DR: In this paper, we use dynamical system to analyze the nonlinear weight dynamics of two-layered bias-free ReLU networks.

Review discussion:
The assumption of Gaussian input X seems rather strong. Do you have a sense of how the analysis would change if the input features have non-Gaussian or multi-modal distributions (as is most likely the case with real-world input features)?  Are there any other implicit or explicit assumptions made in the paper?
Thanks the reviewer for the comments!
The i.i.d Gaussian assumption is extensively used in theoretical analysis in machine learning and statistics, and is in general not a strong assumption at all, in particular when a close-form solution is to be derived. As mentioned in the introduction, previous analysis often uses independence assumptions of ReLU activations for the same input, independence assumption about different path from the input to the target in deep network, or assuming more parameters than the number of parameters, etc, which are potentially stronger than our assumptions in my opinion. We emphasize that considering the difficulty in analyzing nonlinear and non-convex multilayer models, making reasonable assumptions is often the key to sensible conclusions.
In this paper, the Gaussian input assumption is mainly used for obtaining a close-form solution for weight update in one ReLU node (Eqn. 10-11), which reveals interesting nonlinear structure of ReLU node, used in more complicated case of multiple hidden nodes. Intuitively, this assumption is used to "smooth out" the non-differentiable nature of ReLU for easier analysis. For general (multimodal) distributions, since they can be represented as a mixture of Gaussians, similar analysis can also be applied (by rewriting Eqn. 21 as a summation of conditional expectations, and by considering bias and variance terms in the integral computation), although the corresponding close form could be quite complicated.
Furthermore, we also show empirically (Sec. 5, Fig. 3c), other zero-mean distributions, e.g., uniform distribution, also follows Eqn. 10 in high-dimensional case.
We made all our assumptions explicit in the theorems and corollaries.
For example, In Corollary 4.2, we explicitly specify that (1) the network is bias-free two-layered with i.i.d Gaussian input and upper weights fixed to be one, (2) the weights of teacher's network is orthonormal and (3) the student network starts with initialization (x, y, y, ...) in the coordinates of teacher's weights, with x > y but could be arbitrarily small. Other theorems follow similar patterns.
In this paper, the author analyzes the convergence dynamics of a single layer non-linear network under Gaussian iid input assumptions. The first half of the paper, dealing with a single hidden node, was somewhat clear, although I have some specific questions below. The second half, dealing with multiple hidden nodes, was very difficult for me to understand, and the final "punchline" is quite unclear. I think the author should focus on intuition and hide detailed derivations and symbols in an appendix.
In terms of significance, it is very hard for me to be sure how generalizable these results are: the Gaussian assumption is a very strong one, and so is the assumption of iid inputs. Real-world feature inputs are highly correlated and are probably not Gaussian. Such assumptions are not made (as far as I can tell) in recent papers analyzing the convergence of deep networks e.g. Kawaguchi, NIPS 2016.
Although the author says the no assumption is made on the independence of activations, this assumption is shifted to the input instead. I think this means that the activations are combinations of iid random variables, and are probably Gaussian like, right?
So I'm not sure where this leaves us.
Specific comments:
1. Please use D_w instead of D to show that D is a function of w, and not a constant. This gets particularly confusing when switching to D(w) and D(e) in Section 3. In general, notation in the paper is hard to follow and should be clearly introduced.
2. Section 3, statement that says "when the neuron is cut off at sample l, then (D^(t))_u" what is the relationship between l and u?
Also, this is another example of notational inconsistency that causes problems to the reader.
3. Section 3.1, what is F(e, w) and why is D(e) introduced? This was unclear to me.
4. Theorem 3.3 suggests that (if \epsilon is > 0), then to have the maximal probability of convergence, \epsilon should be very close to 0, which means that the ball B_r has radius r -> 0? This seems contradictory from Figure 2.
5. Section 4 was really unclear and I still do not understand what the symmetry group really represents. Is there an intuitive explanation why this is important?
6. Figure 5: what is a_j ?
I encourage the author to rewrite this paper for clarity. In it's present form, it would be very difficult to understand the takeaways from the paper.
Thank the reviewer for the comments.
All the activations are obviously not Gaussians since they are activations of ReLU and hence positive. I think what the reviewer suggests is that all the activations seem to be independent of each other since the inputs are i.i.d Gaussian. This is definitely not the case. Given the same input, they are highly related to each other. E.g., for two dimensional case, if w1 = [1, 0] and w2 = [-1, 0], then obviously their responses are perfectly negatively correlated. In general, the responses of w1 and w2 are uncorrelated only if w1 and w2 are orthogonal. During optimization, in general wi and wj are not orthogonal until they converge to wi* and wj*, which are assumed to be orthogonal (Section 4).
Answer to other questions:
1. Yes all D are dependent on w.
2. u and l is the same. Sorry for the notation issue here.
3. F(e, w) is introduced to simplify the notation of gradient (as shown in the Appendix), since the gradient can be represented by the subtraction of two terms, F(e, w) and F(e, w*), where e is the normalized vector e of the current w, e = w/|w|. The gating matrix D does not depend on the magnitude of w, but only its direction.
Therefore using the notation of D(e) is better.
4. Theorem 3.3 is not inconsistent. It says as long as r <= O(sqrt(d)), then with 1/2-eps probability the sampling in the ball of radius r will lead to convergence to w*.
So (1) if r is smaller than O(sqrt(d)), then the condition of the theorem still holds. That's the reason for "initialization can be arbitrarily close to the origin", and (2) Usually we just use largest r that still satisfies the condition, to avoid any issue of numerical instability.
So r ~ sqrt(d).
5. Basically Section 4 assumes that there is a symmetry among {w1, w2, ..., wK} in which w_i is a dimension-wise shifting of w_j by j - i.
For example, w1 = [1, 2, 3], w2 = [3, 1, 2] and w3 = [2, 3, 1].
Because of this symmetry, the original dynamics of d*K (K weights) variables now becomes the dynamics of K variables (1 weights). The conclusion in Section 4 is based on this symmetry.
6. a_j is the weight of each ReLU node. In the previous analysis (Section 4), a_j is 1.
This work analyzes the continuous-time dynamics of gradient descent when training two-layer ReLU networks (one input, one output, thus only one layer of ReLU units). The work is interesting in the sense that it does not involve some unrealistic assumptions used by previous works with similar goal. Most importantly, this work does not assume independence between input and activations, and it does not rely on noise injection (which can simplify the analysis).
Nonetheless, removing these simplifying assumptions comes at the expense of limiting the analysis to:
1. Only one layer of nonlinear units
2. Discarding the bias term in ReLU while keeping the input Gaussian (thus constant input trick cannot be used to simulate the bias term).
3. Imposing strong assumption on the representation on the input/output via (bias-less) ReLU networks: existence of orthonormal bases to represent this relationships.
Having that said, as far as I can tell, the paper presents original analysis in this new setting, which is interesting and valuable.
For example, by exploiting the symmetry in the problem under the assumption 3 I listed above, the authors are able to reduce the high-dimensional dynamics of the gradient descent to a bivariate dynamics (instead of dealing with original size of the parameters). Such reduction to 2D allows the author to rigorously analyze the behavior of the dynamics (e.g. convergence to a saddle point in symmetric case, or to the optimum in non-symmetric case).
Clarification Needed: first paragraph of page 2. Near the end of the paragraph you say "Initialization can be arbitrarily close to origin", but at the beginning of the same paragraph you state "initialized randomly with standard deviation of order 1/sqrt(d)". Aren't these inconsistent?
Some minor comments about the draft:
1. In section 1, 2nd paragraph: "We assume x is Gaussian and thus the network is bias free". Do you mean "zero-mean" Gaussian then?
2. "standard deviation" is spelled "standard derivation" multiple times in the paper.
3. Page 6, last paragraph, first line: Corollary 4.1 should be Corollary 4.2
Thanks reviewer for the comments! I really appreciate it.
The two paragraphs of page 2 are not inconsistent.
Theorem 3.3 gives the relationship: as long as r <= O(sqrt(d)), then with 1/2-eps probability the sampling in the ball of radius r will lead to convergence to w*.
So (1) if r is smaller than O(sqrt(d)), then the condition of the theorem still holds. That's the reason for "initialization can be arbitrarily close to the origin",
and (2) Usually we just use largest r that still satisfies the condition, to avoid any issue of numerical instability.
So r ~ sqrt(d).
I will make it more clear in the next revision. Thanks again.
Response to the minor comments:
1. Yes. the Gaussians are zero-mean.
2. Sorry for the spelling error and issues in theorem numbering, I will fix it.
The paper proposes a convergence analysis of some two-layer NNs with ReLUs. It is not the first such analysis, but maybe it is novel on the assumptions used in the analysis, and the focus on ReLU nonlinearity that is pretty popular in practice.
The paper is quite hard to read, with many English mistakes and typos.
Nevertheless, the analysis seems to be generally correct. The novelty and the key insights are however not always well motivated or presented. And the argument that the work uses realistic assumptions (Gaussian inputs for example) as opposed to other works, is quite debatable actually.
Overall, the paper looks like a correct analysis work, but its form is really suboptimal in terms of writing/presentation, and the novelty and relevance of the results are not always very clear, unfortunately. The main results and intuition should be more clearly presented, and details could be moved to appendices for example - that could only help to improve the visibility and impact of these interesting results.
We thanks the reviewers for their comments.
All reviewers agree that the paper propose a novel analysis with a different kind of assumption than independent assumption on the activations. Reviewer2 summarizes our contributions in details, and pointed out that the analysis is original, interesting and valuable.
We indeed assume that the input is drawn from zero-mean Gaussian. The technical motivation of the Gaussian assumption is to derive an analytic form of Eqn. 10 and Eqn. 11 (for expected gradient). The i.i.d Gaussian assumption gives a convenient tool for such analysis. Similar assumptions have been extensively applied to many branches of mathematics.
In addition, the underlying intuition that "the population gradient is smooth with a linear term and a nonlinear term dependent on the angle between w and w*" is not restricted to i.i.d Gaussian input, but can be applied to general zero-mean distributions as long as the input space is properly covered (p(x) > 0 everywhere). In Fig. 3(d), we also show that for zero-mean uniform distribution, the analytic form of Eqn. 10 is still empirically correct.
Reviewer1 doubt that under this assumption, the hidden activation might also be Gaussian. This is not true.
First, all activations are obviously not Gaussians since they are activations of ReLU and hence positive. I think what the reviewer suggests is that all the activations seem to be independent of each other since the inputs are i.i.d Gaussian. This is definitely not true as well. Given the same input, they are highly related to each other. E.g., for two dimensional case, if w1 = [1, 0] and w2 = [-1, 0], then obviously their responses are perfectly negatively correlated. In general, the responses of w1 and w2 are uncorrelated only if w1 and w2 are orthogonal. During optimization, in general wi and wj are not orthogonal at all, until they converge to wi* and wj*, which are assumed to be orthogonal (Section 4).
Reviewer1 and Reviewer3 mentioned that the motivation and notation are not generally clear in the paper, especially the second part of the analysis. The motivation of the second part is to show that the analytic form of expected (or population gradient) can be used to analyze multiple hidden units (K>=2) of two-layered network. In that setting, under the additional assumption that (1) teachers' weights are orthonormal and (2) we start from symmetric initialization w, then it is possible to prove the convergence to w*, even if the dynamics is highly nonlinear.
Furthermore, the convergence result is not local, since the initialization w could be made in the epsilon-ball around origin with arbitrarily small epsilon. As mentioned in the introduction, to our knowledge, such result is very novel. We will make sure that the notation is clearly written, and intuitions are given first in the next version.
Answer to questions from Reviewer1:
1. Yes all D are dependent on w.
2. u and l is the same. Sorry for the notation issue here.
3. F(e, w) is introduced to simplify the notation of gradient (as shown in the Appendix), since the gradient can be represented by the subtraction of two terms, F(e, w) and F(e, w*), where e is the normalized vector e of the current w, e = w/|w|. The gating matrix D does not depend on the magnitude of w, but only its direction.
Therefore using the notation of D(e) is better.
4. (Common question from Reviewer1 and Reviewer2). Theorem 3.3 is not inconsistent. It says as long as r <= O(sqrt(d)), then with 1/2-eps probability the sampling in the ball of radius r will lead to convergence to w*.
So (1) if r is smaller than O(sqrt(d)), then the condition of the theorem still holds. That's the reason for "initialization can be arbitrarily close to the origin", and (2) Usually we just use largest r that still satisfies the condition, to avoid any issue of numerical instability.
So r ~ sqrt(d).
5. Basically Section 4 assumes that there is a symmetry among {w1, w2, ..., wK} in which w_i is a dimension-wise shifting of w_j by j - i.
For example, w1 = [1, 2, 3], w2 = [3, 1, 2] and w3 = [2, 3, 1].
Because of this symmetry, the original dynamics of d*K (K weights) variables now becomes the dynamics of K variables (1 weights). The conclusion in Section 4 is based on this symmetry.
6. a_j is the weight of each ReLU node. In the previous analysis (Section 4), a_j is 1.

Meta-review:
role: PC
data: comment: The paper analyzes the dynamics of learning under Gaussian input using dynamical systems theory. As two of the reviewers have pointed out, the paper is hard to read, and not written in a way which is accessible to the wider ICLR community. Hence, I cannot recommend its acceptance to the main conference. However, I recommend acceptance to the workshop track, since it has nice technical contributions that can lead to interesting interactions. I encourage the authors to make it more accessible for a future conference.

Decision: Invite to Workshop Track