Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: -
review_title: -

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a mathematical analysis of how information is propagated through deep feed-forward neural networks, with novel analysis addressing the problem of vanishing and exploding gradients in the backward pass of backpropagation and the use of the dropout algorithm. The paper is clear and well-written, the analysis is thorough, and the experimental results showing agreement with the model are very nice. 
review_title: Excellent analysis 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper expands a recent mean-field approximation of deep random neural networks to study depth-dependent information propagation, its phase-dependence and the influence of drop-out. The paper is extremely well written, the mathematical analysis is thorough and numerical experiments are included that underscore the theoretical results. Overall the paper stands out as one of the few papers that thoroughly analyses training and performance of deep nets.
review_title: An important and thorough contribution to the theoretical analysis of deep neural networks

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: I'm not familiar enough with mean-field techniques to judge the soundness of Eq 2, but I'm willing to roll with it.

Minor point on presentation: Speaking of the "evolution" of x_{i;a} as it travels through the network could give some readers helpful intuition, but for me it was confusing because x_{*;a} is the immutable input vector, and it's the just-introduced z and y variables that represent its so-called evolution, no?

In interpreting this analysis - A network may be trainable if information does not pass through it, if the training steps, by whatever reason, perturb the weights so that information starts to pass through it (without subsequently perturbing the weights to stop information from passing through it.) Perhaps this could be clarified by a definition of “training algorithm”?

Comments on central claims:
Previous work on initializing neural networks to promote information flow (e.g. Glorot & Bengio, http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf) concluded (1) that the number of units in the next layer and the previous layer should both figure into the variance of the elements of the weight matrices, and (2) that they should be drawn from a Uniform distribution rather than a Gaussian. Could the authors comment on the merit of that initialization strategy in light of this analysis?

Comments on evaluation:
Why does the dashed line in the result figures correspond to twice the depth scale instead of just the depth scale? What is the significance of 14000 steps of SGD on MNIST? Does it represent convergence of SGD? Why are all the best SGD models well above the depth scale? Why is there a little dark area precisely under the peak in Figure 5(a) and (c)? That’s interesting - initializations that propagate error best seem untrainable at the depths traditionally used - but only with SGD not RMSProp?

The accuracy of the trained models on CIFAR-10 and MNIST are not reported - it seems important to the overall argument of the paper that the sorts of networks underlying Figures 5 and 6 are the same as the ones that people would consider state-of-art within the model class (fully connected, sigmoidal nonlinearities, etc.).

review_title: interesting analysis - empirical results could be clarified

Meta-review: This is one of the two top papers in my stack. In total the reviews are a little bit on the light side in terms of level of detail and there are some concerns regarding how useful the results are from a practical point of view. However, I am confident that the paper should be accepted. 
Decision: Accept (Poster)