Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the regularization effects involved in the stochastic depth training of ResNets. Specifically, by applying the second-order Taylor approximation to the loss function and the intermediate features, the authors show that the stochastic depth training involves a regularization term that promotes flatness and acts like Gaussian noise to the features. Based on this theoretical insight, the authors suggest a strategy for setting the survival probabilities of sub-networks depending on the computational budget, which corresponds to controlling the regularization effect. This strategy developed upon the new insight in the paper is validated by empirical experiments. 

main_review: Stochastic depth is a simple yet effective regularization technique for training very deep ResNets. However, it has been unclear how the stochastic depth induces regularization effects and how to set dropout rates of sub-networks. In this regard, the authors present valuable theoretical analysis that shows a flatness promoting regularization term and layer-wise Gaussian noise effects involved in the stochastic depth training. I think that this novel insight can help to develop effective strategies for stochastic depth training and its novel variants, and as an example, the authors give a strategy for setting dropout rates depending on the computational budget. Considering the prevalence of ResNet in the machine learning community, I believe that this work will be of interest to both researchers and practitioners. 

The paper is well-written, and the structure is easy-to-follow and interesting. However, there are few minor typos here and there (such as missing legend in Fig 1 (b), crossentropy (L159), The empirical loss given by (L160), hessian (footnote in p.5)). 


* Minor commnet: The authors use the Taylor approximation around the drop ratio of 1, which is highly unlikely in practice and can make higher-order terms non-negligible. I’m wondering if the authors can explain how the small drop rates can impact the regularization term in Eq 6 and Gaussian noise $\gamma_{\alpha, L}(x)$. 

limitations_and_societal_impact: The authors addressed the limitation in the paper, and I agree with the authors' claim that this work will have no direct societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper provides theoretical analysis for stochastic depth from a regularization perspective. The paper further illustrates the different regularization effects of stochastic depth based on perturbation analysis and signal propagation. The analysis leads to principled guidelines for choosing the survival rates.

main_review: The paper is well written and easy to follow.

The paper discusses interesting regularization effects of stochastic depth such as flatness and information discrepancy.

There is no detail about \phi in Line 284. It is important to provide detailed discussion about how to choose the linear mapping \phi and how it impacts the performance.

Line 314: it is not clear why normal behavior is more accurate for deeper layers under uniform mode with large budget. Does this implies SD only mimics Gaussian Noise Injection for deeper layers?

Section 4 title seems incomplete.

Theorem 4 is not provided in the paper.

limitations_and_societal_impact: Limitations that are not addressed and suggestions are included in the above main review. The paper really needs proofreading and grammar check.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper focuses on understanding the regularization effect of stochastic depth in residual neural networks. The authors give 3 main results: 

(a) Stochastic depth helps to mitigate (to some extent) the gradient explosion phenomena observed at initialization in the Vanilla ResNet model, 

(b) Stochastic depth mimics Gaussian Noise Injection (GNI) in deeper models by implicitly adding some non-isotropic Gaussian noise at each layer during training. Thereby, they show that Stochastic depth has the same regularization effects as GNI.

(c) They propose SenseMode, a principled method to choose the survival rates of each layer for training. SenseMode performs better than the existing methods in certain cases.

The authors corroborate all their propositions in an experimental study.
main_review: 

The main paper needs a lot of work before it can be published. Here are my questions and few suggestions.

(1) How do you define the linear mapping $\varphi$ in SenseMode? Is the definition arbitrary? Or do you sort the values and match them to equally spaced numbers between $[p_{\min}, 1]$? How does this kind of mapping solve the issue in the small budget regime when the regularization contains the term $p_{l} (1 - p_{l})$ (which is a quadratic function)? 

Also, it will be great to have similar experiments for Vanilla ResNet in table 2. Because all the propositions and experiments involved Vanilla ResNet and SenseMode was built on the structure of the Vanilla ResNet model.

(2) How does proposition 1 change when you have batch norm in the network? Will you still observe exponential growth of $\tilde{q}_{\ell}(x, z)$? Or is it just going to be a linear growth? Is the $1/\sqrt{L}$ factor still necessary with the batch norm? Because I see in [1], the authors also consider Vanilla ResNet while proposing the $1/\sqrt{L}$ factor? It will be interesting to see if explosion still exists with batch-norm in experiments and stochastic depth still reduces exponential growth. Otherwise, I don't see a point in Proposition 1.

(3) How do I read the plot in Figure 5? Can you please give more details in the experimental section about "Gaussian Noise Injection"? For example, what neurons do you consider for the left plot in Figure 5? Do you take the entire set of neurons to do the left plot and all the neurons per layer for the right plot?
 
(4) Multiple issues in notation and presentation:

(a) What is $G_{l}$? The authors need to specify the function clearly. Is it the function that connects $y_l$ to $y_{out}$? It's not the way to define a function, as has been mentioned in the paper.

(b) Use of both $l$ and $\ell$ is frustrating. For example, the reader has to differentiate the $l$ and $\ell$ used in $H_{\ell}$ and $\zeta_{l}$ (equation 5).

(c) Clearly demarcate Vanilla ResNet and Stable ResNet110. It will be good to have some diagram differentiating the two. As far as I have understood (and reviewed), Vanilla ResNet contains only linear layers and ReLU in their residual blocks. In contrast, Stable ResNet110 contains the sequence BatchNorm-ReLU -Convolution-BatchNorm in their residual blocks.  It's essential to clearly demarcate this since experiments in Figure 2, Figure 3, Figure 4, Figure 5 are done with Vanilla ResNet, and the experiments in Table 2 are done with Stable Resnet110. Also, at some places, you have mentioned Vanilla Stable Resnet. Would you please demarcate the terms properly?

(d) The caption in table 1 says that the table reports $\tilde{q}_l(x, z)$ at initialization. However, I don't see exponential numbers as claimed by proposition 1. Did you mean to write the growth rate (as mentioned in the text)?

(5) Plots lack proper labels and captions:

(a) Figure 1(b) doesn't have a label. What do the two lines in the graph refer to? 

(b) What are the numbers in Table 2(a) and (b)? Are they error or accuracy of the models in the task?

(c) The x and y labels in all the plots are unreadable. I had to guess every time I read some plot (from a printed pdf).




1: S. Hayou, E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau. Stable resnet. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pages 324–1332, 2021a.



***After Rebuttal***
I have increased my score to 6. The paper has many clarity issues and I would appreciate if the authors could incorporate changes to address the concerns raised by all the reviewers.
limitations_and_societal_impact: I have mentioned different issues that exist in the main paper in my main review. The authors discuss the remaining limitations pretty well. As the authors have mentioned, the paper focuses on the general analysis of a regularization method. Hence I don't see a potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: 
The authors study stochastic depth from an theoretical point of view. In particular, they provide insights into the effective depth at each iteration (Lemma 1), discuss the relationship of stochastic depth with exploding gradients (Proposition 1), and relate stochastic depth with explicit regularization (Section 4.1) and Gaussian noise injection (Section 4.2). 
main_review: Their results are novel and provide first insights into stochastic depth from a mathematical perspective. One could argue that the results require many assumptions (such as the abundant, not always completely obvious approximations) and only scratch at the surface of stochastic depth (for example, what do these regularizers mean?). But a complete analytical understanding of stochastic depth, in my opinion, is currently out of reach, and the authors provide a first, reasonable step toward it.

I have not read the proofs in the Appendix.

There is some work on layer-wise regularization that might be connected especially to Section 4.1:

Hebiri, M. and Lederer, J., 2020. Layer sparsity in neural networks. arXiv preprint arXiv:2006.15604.


limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper gives a theoretical analysis of the stochastic depth (SD) technique for ResNet. It is shown that SD mitigates the gradient explosion phenomenon. Moreover, it is shown that SD behaves like a Gaussian noise injection to the features in the internal layers and thus it works as a regularization. Finally, the authors proposed a method called SenseMode that determines the mode under a small computational budget limitation.   

The analysis given in this paper is interesting. It shed light to understand what SD is essentially doing. The proposed SenseMode is also useful.  
On the other hand, as the reviewers pointed out, there are some issues in its writing, i.e., typos, confusing notations and unclear descriptions. In particular, the definition of the linear mapping $\varphi$ in the definition of SenseMode should be explicitly given in the final version. I guess everybody would be confused at this point.    

In summary, this paper has sufficient novelty and gives a good insight for SD, which is beneficial to the community. I think this paper can be accepted by NeurIPS.
Decision: Accept (Poster)