Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper analyzes the statistics of activation norms and Jacobian norms for randomly-initialized ReLU networks in the presence (and absence) of various types of residual connections. Whereas the variance of the gradient norm grows with depth for vanilla networks, it can be depth-independent for residual networks when using the proper initialization.

The main theoretical results stem from a norm propagation duality that allows computations of certain statistics using simplified architectures. As far as I know, this is a novel way of analyzing multi-pathway architectures, and the method and the implied results will be of interest to deep-learning theorists.

There are two types of empirical results, one to verify the theoretical predictions, and the other to see if those predictions translate into improved generalization performance on some real-world tasks. Figures 2 and 3 show good agreement with theory. I am less convinced by Fig 4. Unless I missed something, I thought the implication from the theoretical analysis was that multi-pathway models should behave better than single-pathway models at large depth, and for single-pathway models concatenated ReLU may have slightly better large-depth behavior than standard ReLU. Doesn't this suggest that for large depth DenseNet ~= ResNet > CR > ReLU? Is the good performance of CR supposed to be understood as supporting evidence for the theoretical analysis? Or is the point more like "the theory suggests CR might be interesting, and indeed it performs well, but we leave the details of generalization performance to a future study"? It would be important to specify the desired interpretation here.

But I think regardless of the interpretation, the empirical results are not strong enough to stand on their own as support for CR, especially since this type of architecture has been introduced and studied previously (c.f. also the looks-linear setup from [1]). Therefore my evaluation is mostly based on the theoretical contributions, which I think are themselves are probably sufficient for publication.

One main area for improvement is in the related work. A significant line of work is omitted that studies higher-order statistics of Jacobian matrices in the large-depth, infinite-width regime [2-8], and while this is at infinite width, it work for any non-linearity, so is an important point for comparison. Other kinds of initialization schemes are also relevant for comparison, e.g. [9-10].

[1] Balduzzi, David, et al. "The shattered gradients problem: If resnets are the answer, then what is the question?." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
[2] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice." Advances in neural information processing systems. 2017.
[3] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. "The emergence of spectral universality in deep networks." International Conference on Artificial Intelligence and Statistics. 2018.
[4] Hayase, Tomohiro. "Almost Surely Asymptotic Freeness for Jacobian Spectrum of Deep Network." arXiv preprint arXiv:1908.03901 (2019).
[5] Tarnowski, Wojciech, et al. "Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function." The 22nd International Conference on Artificial Intelligence and Statistics. 2019.
[6] Burkholz, Rebekka, and Alina Dubatovka. "Initialization of ReLUs for Dynamical Isometry." arXiv preprint arXiv:1806.06362 (2018).
[7] Xiao, Lechao, et al. "Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks." International Conference on Machine Learning. 2018.
[8] Chen, Minmin, Jeffrey Pennington, and Samuel Schoenholz. "Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks." International Conference on Machine Learning. 2018.
[9]Sutskever, Ilya, et al. "On the importance of initialization and momentum in deep learning." International conference on machine learning. 2013.
[10] Le, Quoc V., Navdeep Jaitly, and Geoffrey E. Hinton. "A simple way to initialize recurrent networks of rectified linear units." arXiv preprint arXiv:1504.00941 (2015).
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper studies the mean and variance of the gradient norm at each layer for vanilla feedforward, ResNet and DenseNet, respectively, at the initialization step, which is related with Hanin & Ronick 2018 studying the mean and variance of forward activations. They show that ResNet and DenseNet preserve the variance of the layer gradient norm through depths. In comparison, for the vanilla feedforward network, although the mean of the gradient norm is preserved if  is properly initialized, the variance of the layer gradient norm increases over depths, which may explode or decay the gradient at deeper layers. 

The result presented in the paper is interesting, and the theory and empirical verification have a good match. I  recommend the acceptance after the following points are well addressed.

1. The mean and variance of layer activation norm and gradient norm have been studied in the mean field literatures for example [1] and the paper does not have a good comparison with them.
2. The experiment of  CONCATENATIVE RELU is not convincing given the small tasks intertwined performance.

Minor presentation flaws:
1. The sentence in Abstract "This depth invariant result is surprising in light of the literature results that state that the norm of the layer’s activations grows exponentially with the specific layer’s depth." is not clear itself because of no relevant reference.
2. There are many typos such as "weather", "for of" in the paper. Please carefully correct them.
3. The values on the y-axis of Figure 2 and Figure3 are not correctly shown.

[1] Greg Yang and Samuel Schoenholz 2017. Mean Field Residual Networks: On the Edge of Chaos


#####after rebuttal period, I found a fatal error in the paper#####

In (13), the network output $y_t^L$ is decomposed into two parts $y^{L,k}_t$ and $\hat{y}^{L,k}_t$, where $y^{L,k}_t$ is composed of paths that go through weight matrix $W^k$ and $\hat{y}^{L,k}_t$ is composed of paths that skip weight matrix $W^k$.

In (14), the paper derives $J^k:=\frac{\partial y_t^L}{\partial W^k} = \frac{\partial y^{L,k}_t}{\partial W^k}$. However, in fact the other part $\hat{y}^{L,k}_t$ also has gradient with respect to  $W^k$ even if the path $\gamma_t$ does not go through $W^k$ because the activation $z_{\gamma_t}$ is a function of  $W^k$ .

With this error, all the following claims in the paper are wrong. Thus I vote to reject this paper.  

One can do experiment to verify  that "Thm. 1 also reveals a surprising property of the gradients in general Relu networks. That is, when the weights are sampled from the same distribution in each layer, the gradient’s magnitude for each layer are equal in expectation, and depend only on the output statistics." is wrong for ResNet.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper studies the effects of residual and dense net type connections on the moments of per-layer gradients at random initialization. In particular, using duality, bounds on the variance of the square norm of Jacobian (with respect to the randomness of random initialization) are derived for vanilla networks. By noticing that connections that bypass a specific layer does not affect the expected square norm of the gradients of a particular layer, the paper continue to characterize the bounds for residual networks and densely connected networks that have special skip connections. In particular, with properly chosen initialization scales for each layer, the architectures with skip connections can be initialized so that the gradient norm does not explode with increasing depth.

In the second part of the paper, the topic shifted to a bit unrelated topic, comparing the effects of ReLU vs concatenated ReLU blocks on the norm of layer activations.

The results in this paper seem to be useful for guidance on choosing proper initialization schemes for different architectures at different depths. I hope the paper could spend more efforts discussing the potential impact of their results. Moreover, some preliminary empirical studies could also help:

- how well do the bounds still hold after the optimization starts to run?
- does the proposed initialization scheme actually allow training of very deep neural networks?

Minor comments:

- please add x and y labels to the plots
- please add grid (as in Fig 4) to Fig 2 and 3. Currently it is especially hard to decide the slope for curves in Fig 2b.
review_title: Official Blind Review #1

Meta-review: This paper studies the statistics of activation norms and Jacobian norms for randomly-initialized ReLU networks in the presence (and absence) of various types of residual connections. Whereas the variance of the gradient norm grows with depth for vanilla networks, it can be depth-independent for residual networks when using the proper initialization.

Reviewers were positive about the setup, but also pointed out important shortcomings on the current manuscript, especially related to the lack of significance of the measured gradient norm statistics with regards to generalisation, and with some techinical aspects of the derivations. For these reasons, the AC believes this paper will strongly benefit from an extra iteration. 
Decision: Reject