Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper studies the role of initialization for training deep linear neural networks. The authors specifically consider the orthogonal initialization, and prove that with the orthogonal initialization proposed in equation (4), the gradient descent can achieve zero training error in a linear convergence rate. The improvement of the orthogonal initialization lies at the dependence of the layer width $m$, which is independent of the network depth $L$.

The problem considered in this work is very interesting since there are lots of empirical studies show that good initialization can benefit the training of deep neural networks. However, my main concern about this work is its novelty, especially for the proof techniques used in the current paper. It seems that most of the proofs are similar to the previous work Du & Hu (2019), and the main reason that it can remove the dependence of $L$ seems to be Lemma 4.2, which can be derived using the orthogonal property of the initialization. In this sense, there is not too much contribution for the current paper given the previous work Du & Hu (2019). Are there any other significant changes need to be made in the proofs to get the main results? If the authors can provide the convergence guarantees of the stochastic gradient descent, the contributions would be strong. 

For the proof of Theorem 5.1, is it a straightforward extension of the proofs in Shamir (2018)? What is the main challenge when prove the general $d$ case?


Minor comments:
For the last equation in (4), why you need $W_L(0)W_L(0)^\top=mI_{d_y}$ instead of $W_L(0)^\top W_L(0)=d_yI$ as you used in the later proofs?

There is no experiment to verify the theory

Update:
I thank the authors for their response, I would like to keep my score.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper studies the convergence of deep linear networks under orthogonal initialization. Most of the problem setup, analysis techniques and proof roadmap are adapted from Du & Hu (2019). The difference is the orthogonal initialization compared with the Gaussian initialization in Du & Hu (2019). Since the product of orthogonal weight matrices is identity, the new initialization can remove the dependency of the number of nodes $m$ on the depth $L$ of the network. The authors also proved a lower bound of the loss function trained by Gaussian initialization within certain iterations. This justifies the disadvantage of Gaussian initialization.

My biggest concern is that this paper seems to be very similar to Du & Hu (2019) in many places. The whole Sections 3 & 4 are almost the same as Sections 3,4,5 & 7 in their paper. The contribution in this paper is too incremental given previous work. 

What is the advantage of using orthogonal matrices in (4) compared with just using identity matrices as initialization? Can we prove the same result with just identity initialization? In this case, what would we lose by restricting us to this special case?

In the proof of Lemma 4.2, it seems that only the randomness of the last layer $W_L(0)$ is used. Why do we need all the layers to be uniformly sampled?

It should be explained in more details that $W_L(0)$ is drawn from a uniform distribution over orthogonal matrices in $d_y\times d_y$ space. Then $1/\sqrt{m} W_L(0)\cdot z /\|z\|^2$ is not distributed on the whole space of $d_y$-sphere. The argument in the proof of Lemma 4.2 thus needs more justification.

It would be interesting to see an empirical comparison of the proposed initialization and the Gaussian initialization. Due to the lower bound proved in this paper, the experiments are expected to show distinct difference between these two.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper rigorously proves that if a deep linear network is initialized with random orthogonal weights and trained with gradient descent, its width required for convergence  does not depend on its depth. To compare, when weights in deep linear networks are initialized with Gaussian initialization, the minimal width required for convergence will depend on the depth of the network. This proof explains why orthogonal weight initialization can help to train networks efficiently, especially for those very deep ones.

The theoretical contribution of this paper is very important. Orthogonal initialization is found to be useful in deep network training. Although the theory in this paper is developed for linear networks, it still has important guidance meaning in practices in more areas of deep learning. The derivations are correct to my best knowledge. And the paper is well-written and easy to read.

Minor points:
- typo in the last equation in (4)

=======================
Update: Despite the similarity with a previous paper, I still think the theoretical results and empirical observations important and thus I will keep my score.
review_title: Official Blind Review #2

Meta-review: The paper shows that initializing the parameters of a deep linear network from the orthogonal group speeds up learning, whereas sampling the parameters from a Gaussian may be harmful.

The result of this paper can be interesting to the deep learning community. The main concern the reviewers raised is the huge overlap with the paper by Du & Hu (2019). It would have been nice to actually see whether the results for linear networks empirically also hold for nonlinear networks. 
Decision: Accept (Poster)