Regularization in ResNet with Stochastic Depth
Authors: Soufiane Hayou
Fadhel Ayed
Venue: NeurIPS 2021 Poster (2021)
Keywords: ResNet
Stochastic Depth
Regularization
Theory

Abstract: Regularization plays a major role in modern deep learning. From classic techniques such as L1, L2 penalties to other noise-based methods such as Dropout, regularization often yields better generalization properties by avoiding overfitting. Recently, Stochastic Depth (SD) has emerged as an alternative regularization technique for residual neural networks (ResNets) and has proven to boost the performance of ResNet on many tasks [Huang et al., 2016]. Despite the recent success of SD, little is known about this technique from a theoretical perspective. This paper provides a hybrid analysis combining perturbation analysis and signal propagation to shed light on different regularization effects of SD. Our analysis allows us to derive principled guidelines for choosing the survival rates used for training with SD.

Regularization in ResNet with Stochastic Depth
Soufiane Hayou'
Fadhel Ayed"
Department of Statisties University of Oxford United Kingdom
Huawei Technologies France
Abstract
Regularization plays major role in modern deep learning. From classic techniques such as L1, Lo penalties to other noise- basee methods such as Dropout, regulariza- tion often yields better generalization properties by avoiding overfitting. Recently, Stochastic Depth (SD) has emerged as alternative regularization technique for residual neural networks (ResNets) and has proven to boost the performance ResNet many tasks [Huang et al., 2016]. Despite the recent success of SD, little is known about this technique from a theoretical perspective. This paper provides hybrid analysis combining perturbation analysis and signal propagation to shed light on different regularization effects of SD, Our analysis allows us to derive principled guidelines for choosing the survival rates used for training with SD.
Introduction
Stochastic Depth (SD) is well-established regularization method that was first introduced by Huang et al. [2016]. It is similar in principle to Dropout [Hinton et al., 2012, Srivastava et al., 2014] and DropConnect [Wan et al., 2013]. It belongs to the family of noise-based regularization techniques, which includes other methods such as noise injection in data [Webb, 1994, Bishop, 1995] and noise injection throughout the network [Camuto al., 2020J. While Dropout, resp. DropConnect consists of removing some neurons, resp. weights, at each iteration, SD randomly drops full layers, and only updates the weights of the resulting subnetwork at each training iteration. As result of this mechanism, SD can be exclusively used with residual neural networks (ResNets).
There exists stream of papers in the literature the regularization effect of Dropout for linear models [Wager et al., 2013, Mianjy and Arora, 2019, Helmbold and Long, 2015, Cavazza et al., 2017]. Recent work by Wei al. [2020] extended this analysis to deep neural networks using second-order perturbation analysis. It disentangled the explicit regularization of Dropout on the loss function and the implicit regularization regularization effect induced by adding Gaussian Noise to the activations and empirically illustrated the benefits of this regularization scheme. However, to the best of our knowledge, no analytical study of SD exists in the literature. This paper aims to fill this gap by studying the regularization effect of SD from an analytical point of view; this allows us to derive principled guidelines on the choice of the survival probabilities for network layers. Concretely, our contributions are four-fold:
the gradient. Similarly, Camuto et al. [2020] studied the explicit
We show that SD acts as an explicit regularizer on the loss function by penalizing a notion of information discrepancy between keeping and removing certain layers.
We prove that the uniform mode, defined as the choice of constant survival probabilities, is related to maximum regularization using SD.
We study the large depth behaviour of SD and show that in this limit, SD mimics Gaussian Noise Injection by implicitly adding data-adaptive Gaussian noise to the pre-activations.
'qqull contribution. Correspondence to: <soufiane, hat.fiiinnaayouyaaaoo...o hww.uyyaaoo.o..oooo fr; fadhel ayedthuavei.com
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

By defining the training budget L as the desired average depth, we show the existence of two different regimes: small budget and large budget regimes. We introduce a new algorithm called SenseMode to compute the survival rates under a fixed training budget and provide series of experiments that validates our Budget hypothesis introduced in Section 5.
2 Stochastic Depth Neural Networks
Stochastic depth neural networks were first introduced by Huang et al. [2016]. They are standard residual neural networks with random depth. In practice, each block the residual network is multiplied by random Bernoulli variable 6y (1 is the block's index) that is equal to 1 with some survival probability pl and otherwise. The mask is re-sampled after each training iteration, making the gradient act solely on the subnetwork composed of blocks with ót 1.
We consider slightly different version where we apply the binary mask the pre-activations instead of the activations. We define a depth L stochastic depth ResNet by
(2;; 8) Vo(r, Wo),
y (a; 6) 1x; 6) + ( Wi), 1SISL, Yout (2; 6) Vout (1;; 6), W Wout),
(I)
where Wi are the weights in the 1th layer, V is a mapping that defines the nature of the layer, are the pre-activations, and 6 = (Si) is a vector of Bernoulli variables with survival parameters (t.)) 8 is re-sampled at each iteration. For the sake of simplification, we consider constant width ResNet and we further denote by N the width, i.e. for all 7 E [I- - 1], yt E IR-V The output function of the network is given by s(yout) where is some convenient mapping for the learning task, e.g. the Softmax mapping for classification tasks. We denote by the dimension of the network output, i.e. s(your) RR which is also the dimension of yout. For our theoretical analysis, we consider a Vanilla model with residual blocks composed of a Fully Connected linear layer
Vi(a W) WΦ(x),
where (bx) is the activation function. The weights are initialized with He init (He al., 2015], e.g. for ReLU, W!J (0, 2/N).
There are no principled guidelines on choosing the survival probabilities. However, the original paper by Huang et al. [2016] proposes two alternatives that appear to make empirical consensus: the uniform and linear modes, described by
Uniform: pt PL.
Linear: pt = 1- 4 PL),
where we conveniently parameterize both alternatives using PL-
Training budget L. We define the training budget L to be the desired average depth of the sub- networks with SD. The user typically fixes this budget, e.g., a small budget can be necessary when training is conducted on small capacity devices.
Depth of the subnetwork. Given the mode p (pt)aSIEL after each iteration, the subnetwork
has a depth Ls = Σl=1 & with an average Lp := Lo|Ls] = E/= Pt. Given budget L, there is a infinite number of modes p such that Lp = L. In the next lemma, we provide probabilistic bounds on La using standard concentration inequalities. We also show that with a fixed budget L, the uniform mode is linked to maximal variability.
Lemma (Concentration of L8). For any 3 E (0, 1), we have that with probability at least 1 - B,
[log(2/3)
[T. - Tppl S Up -1
(2)
Upp
where Lp E[Ls)= -t Up Var(Lss) = Et- Pi(1 pt). and u(1) (1+ log(1 + 1) L.
Moreover, for given average depth L. = L, the upperbound in Eq. (2) is maximal for the uniform choice of survival probabilities 4 :..
...)) 4.

---

Lemma shows that with high probability, the depth of the subnetwork that we obtain with SD is within an & error of Up upu=1 (log(2/8) from the average depth Lp. Given a fixed Up
Distabutton the Effective depth the subtwwookk
0.10
0.08
LinAer
Fieg seggooryy
0.00
budget L. this segment is maximized for the uniform mode Fig. Ia highlights this result. This was expected since the variance of the depth Ls is also maximized by the uniform mode. The uniform mode corresponds to max- imum entropy of the random depth, which would intuitively results in maximum regularization. We depict this behaviour in more details in Section 4.
p = (L/L...., L).
0.04
0.02
0.00
(a) Distributions of Ls for Resnet100 with average survival rate L/L = 0.5 for the uniform and linear modes. Training time
SD vs Dropout. From a computational point of view, SD has the advantage of reducing the effective depth dur- ing training. Depending on the chosen budget, the subnet- works might be significantly shallower than the entire network (Lemma 1). This depth compression can be effectively lever- aged for computational training time gain (Fig. lb). It is not the case with Dropout. Indeed, assuming that the choice of dropout probabilities is such that we keep the same number of parameters on average compared to SD, we still have to multi- ply matrices L times during the forward/backward propagation. It is not straightforward to leverage the sparsity obtained by Dropout for computational gain. In practice, Dropout requires an additional step of sampling the mask for every neuron, re- sulting in longer training times than without Dropout (Fig. lb).
120
Time (min)
110
100
$O
90
Dropout
0.8
1.0
Average survival rate
(b) Training time of Dropout and SD on CIFARIO with ResNet56 for 100 epochs.
Figure 1
However, there is trade-off between how small the budget is and the performance of the trained model with SD (Section 6).
3 Effect of Stochastic Depth at initialization
Empirical evidence strongly suggests that Stochastic Depth allows training deeper models [Huang et al., 2016]. Intuitively, at each iteration, SD updates only the parameters of subnetwork with average depth L, = Ei < L, which could potentially alleviate any exploding/vanishing gradient issue. This phenomenon is often faced when training ultra deep neural networks. To formalize this intuition, we consider the model's asymptotic properties at initialization in the infinite-widnn limit N + +00. This regime has been the focus of several theoretical studies [Neal, 1995, Poole et al., 2016, Schoenholz et al., 2017, Yang, 2020, Xiao et al., 2018, Hayou et al., 2019, 2020, 2021a] since it allows to derive analytically the distributions of different quantities of untrained neural networks. Specifically, randomly initialized ResNets, as well as other commonly-used architectures such as Fully connected Feedforward networks, convolutional networks and LSTMs, are equivalent to Gaussian Processes in the infinite- width limit. An important ingredient in this theory is the Gradient Independence assumption. Let us formally state this assumption first.
Assumption (Gradient Independence). In the infinite width limit, we assume that the weights W used for back prokppropggation are an iid version of the weights W usedfor forward propagation.
Assumption is ubiquitous in the literature on the signal propagation in deep neural networks. has been used to derive theoretical results on signal propagation in randomly initialized deep neural network [Schoenholz et al., 2017, Poole et al., 2016, Yang and Schocnholz, 2017, Hayou et al., 2021b,a] and is also key tool in the derivation of the Neural Tangent Kernel [Jacot et al., 2018, Arora et al., 2019, Hayou et al., 2020]. Recently, it has been shown by Yang [2020] that Assumption I yields the exact computation for the gradient covariance in the infinite width limit. See Appendix A0.3 for a detailed discussion about this assumption. Throughout the paper, we provide numerical results that substantiate the theoretical results that derive using this assumption. We show that Assumption yields an excellent match between theoretical results and numerical experiments.
Leveraging this assumption, Yang and Schoenholz [2017]. Hayou et al. [2021a] proved that ResNet suffers from exploding gradient at initialization. We show in the next proposition that SD helps mitigate the exploding gradient behaviour at initialization in infinite width ResNets.
3

---

Growth rate of gradient magnitude
Growth rate grodient magnitude
Growth rate of gradient megritude
1014
standard untorm, _0_007
untiomm550
1g
10
Incerr30 30,0.7
11 APl
IP Pa
1P,,477
119, .11
10
13'
15
10
10
(a) L/L = 0.5, standard ResNet. (b) L/L = 0.7, standard ResNet. (c) L/L = 0.7, stable ResNet.
Figure 2: Empirical illustration of Proposition 1 ((a) and (b)) and Stable Resnet (c). Comparison of the growth rate of the gradient magnitude (z, 2) at initialization for Vanilla ResNet50 with width 512. The y-axis of figures (a) and (b) are in log scalc. The y-axis of figure (c) is in lincar scale. The expectation is computed using 500 Monte-Carlo (MC) samples.
Proposition 1. Let é ReLU and [(x, E(youe (z; 8), 2) for 22 E Rd x R', where ((z, 2') is some differentiable loss function. Let (2, 2) Ew.
VV vT LTP where the numerator and denomi-
nator are respectively the norms of the gradients with respect to the inputs of the 8th and 1 layers. Then, in the infinite width limit, under Assumption for all I E [L] and (x, z) E Rd x RR, we have
With Stochastic Depth, (1, 2) I (1 ps),
Table 1:
Gradient magnitude growth rate with Vanilla Resnet50 with width 512 and training budget L.LL = 0.7. Empirical vs. The- orctical value (between parenthe- sis) of q (z. z) initialization, for standard (no SD), uniform and lin- ear modes. The expectation is per- formed using 500 MC samples.
Without Stochastic Depth (i.e. 6 1), (x,, 24-1
Proposition 1 indicates that with or without SD, the gradient plodes exponentially at initialization as it backpropagates through the network. However, with SD, the exponential growth is charac- terized by the mode p. Intuitively. if we choose Pt < 1 for some layer 1, then the contribution of this layer in the exponential growth is negligible since 1 + x 1. From a practical point of view, the choice of pt < means that the 7th layer is hardly present in any subnetwork during training, thus making its contribution to the gradient negligible on average (w.r.t 8). For ResNet with
Standard
Uniform
Linear
2,001
1,705 (1.7)
1.64
2,001 (2)
1,708 (1.7)
1633 (1.629)
20 2,001 (2)
1,007 (1.7)
1.569 (1.573)
L = 50 and uniform mode p (1/2,.... 1/2). SDD reduces the 1.999 (2)
1,716 (1.7)
(1.516)
1,739 (1.7)
(L.45)
gradient exploding by six orders of magnitude. Fig. 2a and Fig. 2b illustrates the exponential growth of the gradient for the uniform and linear modes, as compared to the growth of the gradient without SD, We compare the empirical//teoreeicee growth rates of the magnitude of the gradient in Table 1; the results show a good match between our theoretical result (under Assumption 1) and the empirical ones. Further analysis can found in Section A4.
Stable ResNet. Hayou et al. [2021a] have shown that introducing the scaling factor 1/ 1/VT. in front of the residual blocks is sufficient to avoid the exploding gradient at initialization, as illustrated in Figure 2c, The hidden layers Stable ResNet (with SD) are given by,
or VL
yn (z; 8) = y 1== 8)
v 1 8), WT), 1515L.
The intuition behind the choice of the scaling factor 1/VL comes for the variance of yt- At initial- ization, with standard ResNet (Eq. (1)), we have Var[w]] = Varly- + @(1), which implies that Var Var[yr] (I). With Stable ResNet (Eq. (3)), this becomes Var[y] = Var (y- + @(1/L), L.). resulting in Var = @(1) (See Hayou et al. [2021a] for more details). In the rest of the paper, we restrict our analysis to Stable ResNet; this will help isolate the regularization effect of SD in the limit of large depth without any variance/gradient exploding issue.
Nevertheless, the natural connection between SD and Dropout, coupled with the line of work on the regularization effect induced by the latter [Wager et al., 2013, Mianjy and Arora, 2019, Helmbold and Long, 2015, Cavazza et al., 2017, Wei et al., 2020], would indicate that the benefits of SD are not limited to controlling the magnitude of the gradient. Using a second order Taylor expansion, Wei et al. [2020] have shown that Dropout induces an explicit regularization on the loss function. Intuitively, one should expect a similar effect with SD. In the next section, we elucidate the explicit regularization effect of SD on the loss function, and we shed light on another regularization effect of SD that occurs in the large depth limit.

---

Regularization effect of Stochastic Depth
4.1 Explicit regularization on the loss function
Consider a dataset D = A' x T consisting of n (input, target) pairs {( ((a,,t))ss hssgg with (I,, E R4 x R Let E IR4 x IR - R be a smooth loss function, e.g. quadratic loss, cross-entropy loss etc. Define the model loss for single sample (ar, t) by
L(W, 5) ( 6), ((W, = Es (x;
where W =Wi)ostgr The empirical loss given by L(W) = Σ Eg [f(your 6), [:)]
To isolate the regularization effect of SD on the loss function, we use a second order approximation of the loss function around 6 1, this allows us to marginalize out the mask d. The full derivation
is provided in Appendix A2. Let (2;;;; ō) = Wr(Wi, 11-1 8)) be the activations. For some pair
(ar, () E D, we obtain
L
C(W x) R L(W x) +
m(1 - PW) 2), Σ
(4)
21
1=1
where L(W, .x) x E(oout (z; p), t) (more precisely, L(W, .z) is the second order Taylor approxi- mation of (yoou (2: p). around p 12), and gi (. x) = 2 (2; V
21 [
Gil(m((;;) (2)
with G2 is the function defined by (2; 1) = Gi(m--1(:;; 1) + VT. (z;
The first term L(W 2x) in Eq. (4) is the loss function of the average network (i.e. replacing ō with its mean p). Thus, Eq. (4) shows that training with SDD entails training the average network with an explicit regularization term that implicitly depends on the weights W.
SDD enforces flatness. The presence of the hessian in the penalization term provides a geometric interpretation of the regularization induced by SD: enforces notion of flatness determined by the hessian of the loss function with respect to the hidden activations by the weights, thus leading flatter minima. Recent works by [Keskar et al., 2016, Jastrzebski et al., 2018, 8, Yao et al., 2018) showed empirically that flat minima yield better generalization compared to minima with large second derivatives of the loss. Wei et al. [2020] have shown that similar behaviour occurs in networks with Dropout.
This flatness is naturally inherited
Let (1 v (r (z; 1)) be the Jacobian of the output layer with respect to the hidden layer y with 5 = 1, and (xx) VLl(2) l (=u((>;)) the hessian of the loss function f. The hessian matrix inside the penalization terms gi [W, ..) can be decomposed as in [LeCun et al., 2012, Sagun et al., 2017]
V [L Gr](nn 1)) =(x))" (z)))(()) + +1((z),
where depends on the hessian of the network output. I is generally non-PSD, and therefore cannot be seen as regularizer. Moreover, it has been shown empirically that the first term generally dominates and drives the regularization effect [Sagun et al., 2017, Wei et al., 2020, Camuto et al., 2020]. Thus, we restrict our analysis to the regularization effect induced by the first term, and consider the new version of gi defined by
gv W, .) = ((x W)T H()) W) = Tr (H()) (1z, W)G(Z, W)").
(5)
where (zx, W = J[(() 2 1). The quality of this approximation is discussed in Appendix A4.
Information discrepancy. The vector Çi represents a measure of the information discrepancy between keeping and removing the 1th layer. Indeed, & measures the sensitivity of the model output to the thh layer,
Yout (z; 1) yout (z; 11) 2 V (oy 8-1 G( W),
where 1, is the vector of I's everywhere with in the 1th coordinate.
With this in mind, the regularization term 9i in Eq. (5) is most significant when the information discrepancy is well-aligned with the hessian of the loss function, i.e. SD penalizes mostly the layers with information discrepancy that violates the flatness, confirming our intuition above.
Note that we could obtain Eq. (4) using the Taylor expansion around d = However, in this case, the Hessian will depend on p. which complicates the analysis of the role of p in the regularization term
5

---

Quadratic loss With the quadratic loss f(2, = ||2 2'1 the hessian Hz(x) = 2/ is isotropic, Le. does not favorite any direction over the others. Intuitively, we expect the penalization to be similar for all the layers. In this case, we have gi (W, = 2 (1)(())) l2, and the loss is given by
((W) = ((W)+
Σ 1=1
P1
(6)
2L
where (W) Σ (G( W)1B is the regularization term marginalized over inputs X.
Eq. (6) shows that the mode p has a direct impact on the regularization term induced by SD. The latter tends to penalize mostly the layers with survival probability Pt close to 50%. The mode = (1/2.... ..2....,,,/. 1/2) is therefore a universal maximizer of the regularization term, given fixed weights W. However, given training budget L, the mode p that maximizes the regularization term 2T Ei= pr(1 (W)) depends the values of gi (W). We show this in the next lemma.
Lemma 2 (Max regularization). Consider the empirical loss & given by Eq. (6r some fixed weights W (e.g. W could be the weights at any training step of SGD). Then, given a training budget L, the C9: (W)-)). where C is normalizing constant, that has the same sign as L 2L. The global maximum is obtained for pt 1/2.
regularization is maximal for pi min (1, max(0,
Distribution of the penalization coefficients
Lemma shows that under fixed budget, the mode p' that maximizes the regularization induced by SD is generally layer- dependent (ff uniform). However, we show that at initialization, on average (w.r.t W). p' is uniform.
Theorem 1 (p" is uniform at initialization). Assume & = ReLU and W are initialized with N0, )) Then, in the infinite width limit, under Assumption 1, for all E [1: L}, we have
Ew (gu (W) = Ew (ewl9((W))
20
25
30
As a result, given budget I., the average regularization term 2T Σi= pr(1 Ew) [g)()g)))))) is maximal for the uniform mode p" (L/L....LLII) L/L).
Distribution of gi (W) across the layers initialization for Vanilla ResNet50 with width 512.
Figure 3:
The proof of Theorem is based on some results from the signal propagation theory in deep neural network. We provide an overview of this theory in Appendix A0. Theorem shows that, given training budget I. and a randomly initialized ResNet with //0, 2/N) and N large, the average (w.r.t W) maximal regularization at initialization is almost achieved by the uniform mode. This is because the coefficients w [9x are equal under Assumption 1, which we highlight in Fig. 3. As a result, we would intuitively expect that the uniform mode performs best when the budget L is large, e.g. L is large and L R L, since in this case, at each iteration, we update the weights of an overparameterized subnetwork, which would require more regularization compared to the small budget regime. We formalize this intuition in Section 5.
In the next section, show that SD is linked to another regularization effect that only occurs in the large depth limit; in this limit, we show that S'D mimics Gaussian Noise Injection methods by adding Gaussian noise to the pre-activations.
4.2 Stochastic Depth mimics Gaussian noise injection
Recent work by Camuto et al. [2020] studied the regularization effect of Gaussian Noise Injection (GNT) on the loss function and showed that adding isotropic Gaussian noise to the activations z improves generalization by acting as a regularizer on the loss. The authors suggested adding a zero mean Gaussian noise parameterized by its variance. At training time t, this translates to replacing zf by z + of, where 2/ is the value of the activations in the 1th layer at training time t, and of is a parameter that controls the noise level. Empirically, adding this noise tends to boost the performance by making the model robust to over-fitting. Using similar perturbation analysis as in the previous section, we show that when the depth is large, S'D mimics GNI by implicitly adding non-isotropic dala-adaptive Gaussian noise to the pre-activations yt at each training iteration. We bring to the reader's attention that the following analysis holds throughout the training (it is not limited to the initialization), and does not require the infinite-widte regime.

---

Consider an arbitrary neuron Vog in the (aL) h layer for some fixed a E (0. 1). VoL (z, 6) can be approximated using first order Taylor expansion around ô We obtain similarly,
al
(2, = (o)
m (2,, V. G(((((;)))
VL
where Gj defined by Var 1) (yn(( 1)), "n = ój - Pt. and Far(o) = (x, 1
VI. Ei (pi - 1) (21, Vu Gi 1))) = Vaa (I,
Let (x) - Em m (2. v () 1))} With SD, VaL(zi 6) can therefore be seen as perturbed version of VaL (z: (the pre-actinn pre-activation of the average network) with noise Yo, L (z). The scaling factor 1/VL ensures that Jt.,. remains bounded (in &2 norm) as L grows. Without this scaling, the variance of 7 will generally explode. The term T captures the randomness of the binary mask o, which up to factor a, resembles to the scaled mean in Central Limit (ecoorem(CTT) and can be written (x) vā x VaI X1, where (z) m (2i, V (z;
Ideally, we would like to apply CLT to conclude on the Gaussianity of Ya. (2)) ) in the large depth limit. However, the random variables X; are generally not i.i.d (they have different variances) and they also depend on L. Thus, standard CLT argument fails. Fortunately, there is a more general form of CLT known Lindeberg's 's CLT which use in the proof of the next theorem.
Theorem 2. Let x E Rd, (z) = Tn (2) where (2,)(2)) = (Ei, v ()(((; 1)), and of (z) = Vara [x), = pt(1 PI),, for E [L]. Assume that
1. There exists a E 10.) such that for all L, and I E [L], E (a, 1 a).
maNvell] (x) O.
2. lim
El (z)
Mode: unitorm, p 0.5
0.25
Seath
3. v0, = limo-+00 ())
Septh 100
exists and is finite.
Seath 200
paplooo
Then, (x)
D2 N10, (x)))))). L+000
0.15
0.10
Fig. 4 provides empirical verification of the second condition of Theorem 2 across all training epochs. There is clear downtrend as the depth increases; this trend is consistent throughout training, which supports the validity of the second condition in Theorem 2 at all training times. Theorem 2 shows that training a ResNet with SD involves implicitly adding the noise "fa, L (z) to v This noise becomes asymptotically normally distributed confirming that SD implicitly injects input-dependent Gaussian noise in this limit. Camuto et al. [2020] studied GNI in the context of inputtindeeendent noise and concluded on the benefit of such methods on the overall performance of the trained network. We empirically confirm the results of Theorem in Section 6 using different statistical normality tests.
0.05
Epoch
Figure 4: (Theorem 2) Assumption 2 as function the depth L and epoch.
Similarly, we study the implicil regularization effect of SD induced on the gradient in Appendix A3, and show that under some assumptions, SD acts implicitly on the gradient by adding Gaussian noise in the large depth limit.
The Budget Hypothesis
We have seen in Section 4 that given budget L, the uniform mode is linked to maximal regularization with SD at initialization (Theorem 1). Intuitively, for fixed weights W, the magnitude of standard regularization methods such as Il-I|: [|-I|2 correlates with the number of parameters; the larger the model, the bigger the penalization term. Hence, in our case, we would require the regularization term to correlate (in magnitude) with the number of parameters, equivalently, the number of trainable layers. Assuming L >>>>> 1, and given a fixed budget L, the number of trainable layers at each training
'The limiting variance Va, oo (z) depends on the input æ, suggesting that Yo, (.) might converge in distribution a Gaussian process in the limit of large depth, under stronger assumptions. We leave this for future work.
7

---

iteration is close to L (Lemma 1). Hence, the magnitude of the regularization term should depend on how large/small the budget L is, as compared to
Small budget regime (L/L= < 1). In this regime, the effective depth Ls of the subnetwork is small compared to L. As the ratio L gets smaller, the sampled subnetworks become shallower, suggesting that the regularization need not be maximal in this case, and therefore p should not be uniform in accordance with Theorem 1. Another way to look at this is through the bias-variance trade-off principle. Indeed, T./T. T. - O, the variance of the model decreases (and the bias increases), suggesting less regularization is needed. The increase in bias inevitably causes a deterioration of the performance; we call this the for small budget regimes, we introduce a new Information Discrepancy based algorithm (Section 4). We call this algorithm Sensitivity Mode or briefly SenseMode. This algorithm works in two steps:
trade-off. To derive more sensible choice of p
1. Compute the sensitivity (S) of the loss w.r.t the layer at initialization using the approximation,
S = 1W: 11) = V V& ((W:)) 6-1-
St is measure of the sensitivity of the loss to keeping/removing the 1th layer.
2. Use a mapping to map S to the mode, - (SS), where is a linear mapping from the range of S to [pmin: 1J and pmin is the minimum survival rate (fixed by the user). & is the linear mapping from the range of S to the segment [Pmmny l]. In other words,
Pt Pmin + a x St. where the constant alpha is chosen in order satisfy the budget constraint: ΣIPI=
Large budget regime (T./L. ~ 1). In this regime, the effective depth La of the subnetworks is close I., and thus, we are in the overparameterized regime where maximal regularization could boost the performance of the model by avoiding over-fitting. Thus, we anticipate the uniform mode to perform better than other alternatives in this case. We are now ready to formally state our hypothesis,
Budget hypothesis. Assuming L >> 1, the uniform mode outperforms SenseMode in the large budget regime, while SenseMode outperforms the uniform mode in the small budget regime.
We empirically validate the Budget hypothesis and the Budget-performance trade-off in Section 6.
6 Experiments
The objective of this section is two-fold: we empirically verify the theoretical analysis developed in sections 3 and 4 with Vanilla ResNet model on toy regression task; we also empirically validate the Budget Hypothesis on the benchmark datasets CIFAR-10 10 and CIFAR-100 [Krizhevsky et al., 2009]. Notebooks and code to reproduce all experiments, plots and tables presented are available in the supplementary material. We perform comparisons at constant training budgets.
Implementation details: Vanilla Stable ResNet is composed of identical residual blocks each formed of Linear layer followed by ReLU. ResNet1 follows [He et al., 2016, Huang et al., 2016]; comprises three groups of residual blocks; each block consists of a sequence Convolution- -ew.--ooo---L--Cooooouuee Battuioonn-Bahhooo.... We use the adjective "Stable" (Stable Vanilla ResNet, Stable ResNetl 10) to indicate that we scale the blocks using factor 1/VL L as described in Section 3. We build on an open-source implementation of standard ResNet'. The toy regression task consists of estimating the function fa+ x + sin(8" r), where the inputs x and parameter B are in R256 sampled from a standard Gaussian. CIFAR-10, CIFAR-100 contain 32-by-32 color images, representing respectively 10 and 100 classes of natural scene objects. We present here our main conclusions. Further implementation details and other insightful results are in the Appendix A4.
Gaussian Noise Injection: We proceed by empirically verifying the Gaussian behavior of the neurons as described in Theorem 4. For each input x, we sample 200 masks and the corresponding y( 8). We then compute the p-value pV, of the Shapiro- Wilk test of normality [Shapiro and Wilk, 1965]. In Fig. 5 we represent the distribution of the p-values (pez x E X)). We can see that the Gaussian behavior holds throughout training (left). On the right part of the Figure,
that the Normal behavior becomes accurate after approximately 20 layers. In the Appendix we report further experiments with different modes, survival rates, and different test of normality to verify both Theorem 2 and the critical assumption 2.
classification pk -ytorrch
8

---

Shapiro vavales
Shepiro test valuues for uniform
0.8
0.8
vellues
0.4
0..4
8.0
300
30
40
50
epochs
depth
Figure 5: Empirical verification of Theorem 2 on Vanilla ResNet100 with width 128 with average survival probability L/L = 0.7 and uniform mode. Distribution of the p- p-values for Shapiro's 's normality test as function of the training epoch (left) and depth of the network (right). The tests are performed for the final output neuron (2(x)) (x) (left) and for arbitrary neuron per layer (right).
Empirical verification of the Budget Hypothesis: We compare the three modes: Uniform, Linear, and SenseMode two benchmark datasets using a grid survival proportions. The val- ues and standard deviations reported are obtained using four runs. For SenseMode, we use the simple rule Pt & [Sr]. where S is the sensitivity (see section 5). We report in Table the results for Stable ResNet110. Results with Stable ResNet56 are reported in Appendix A4. The empirical results are coherent with the Budget Hypoth- esis. When the training budget is large, i.c. L/L 2 0.5, the Uniform mode outperforms the others. We note nev- ertheless that when L/L 2 0.9, the Linear and Uniform models have similar performance. This seems reasonable as the Uniform and Linear probabilities become very close for such survival proportions. When the budget is low, i.e.
Table 2: Comparison of the modes of selec- tion of the survival probabilities with fixed budget with Stable ResNet1
L/L
Uniform
SenseMode
Linear
0.1
17.2 + 0.3
15.4 0.4
0.2
10.3 + 0.4
+ 0.5
L/L < 0.5, the SenseMooe outperforms the uniform one
0.3
7.7 0.2
7,0 0.3
0.4
7.4
7.3
(the linear mode cannot used with budgets < L/2 when L >> 1, since Σ pt/L. L.> > 1/2-1/(2L) 1/2), thus
0.5
6.8 +0.1
7.3
9.1 + 0.1
0.6
6.3 10.2
6.9
7.5 0.2
confirming the Budget hypothesis. Table also shows a clear Budget-pe perggee-prfoommmceeeeee trade-off.
0.7
5.9
7.3
0.2
0.8
5.7
6.2
6.1 0.2
0.9
5.7 1.1
6.2 0.2
6.0 0.2
6.37 0.12
7 Related work
CIFARIO
The regularization effect of Dropout in the context of linear models has been the topic of stream of papers [Wager et al., 2013, Mianjy and Arora, 2019, Helmbold and Long,
L/L
Uniform
SenseMode
Linear
0.1
$5.2
51.3
0.6
0.2
38.3
0.3
36.4 0.4
2015, Cavazza al., 2017]. This analysis has been recently
0.3
31.4 0.2
30.1
0.5
extended to neural networks by Wei et al. [2020] where au-
0.4
30.9
1:2
28.5
0.4
thors used a similar approach to ours to depict the explicit
+.34+
29.5
0.5
36.5 0.4
0.6
26.5
0.4
29.9
0.6
30.9 :: 0.4
and implicit regularization effects of Dropout. To the best
0.7
25.8 0.1
29.5 0.3
27.3 0.3
of our knowledge, our paper is the first to provide analyt-
0.8
25.5
30.0 0.3
25.7 0.2
ical results for the regularization effect of SD, and study the large depth behaviour SD, showing that the latter mimics Gaussian Noise Injection [Camuto et al., 2020). A further analysis of the implicit regularization effect of S'D is provided in Appendix A3.
25.5
28.3 0.2
25.5 + 0.2
26.5 0.2
(b) CIFARI00
Limitations and extensions
In this work, we provided an analytical study of the regularization effect of SD using a second order Taylor approximation of the loss function. Although the remaining higher order terms usually dominated by the second order approximation (See the quality the second order approximation in Appendix A4), they might also be responsible for other regularization effects. This is common limitation in the literature on moise-based regularization in deep neural networks [Camuto et al., 2020, Wei et al., 2020]. Further research is needed to isolate the effect of higher order terms.
We also believe that SenseMode opens an exciting research direction knowing that the low budget regime has not been much explored yet in the literature. We believe that can probably get even better results with more elaborate maps such that p = PSS). Another interesting extension of algorithmic nature the dynamic use of SenseMode throughout training. We are currently investigating this topic which we leave for future work.

---

References
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pages 646-661. Springer, 2016.
Geoffrey Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint urXiv:/207.05500 2012.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, lly Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitring. The journal of machine learning research, 15(1):1929-1958 2014.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropeonnect. In International conference on machine learning, pages 1058-1066, 2013.
A.R. Webb. Functional approximation by feed-forward networks: least-squares approach to generalization. IEEE Transactions on Neural Networks, 5(3):363-371, 1994.
Chris M. Bishop. Training with noise is equivalent to tikhonov regularization. Neural Computation, 1(1):088-116, 1995.
A. Camuto, M. Willetts, U. Simsekli, S. J. Roberts, and C. C. Holmes. Explicit regularisation in gaussian noise injections. In Advances in Neural Information Processing Systems, volume 33, pages 16603-16614, 2020.
Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. In Advances in neural information processing systems, pages 351-359, 2013.
Poorya Mianjy and Raman Arora. On dropout and nuclear norm regularization. arXiv preprint arXiv:1905.//887, 2019.
David P Helmbold and Philip M Long. On the inductive bias of dropout. The Journal of Machine Learning Research, 1644533403-3444 2015.
Jacopo Cavazza, Pietro Morerio, Benjamin Haeffele, Connor Lane, Vittorio Murino, and René Vidal. Dropout as a low-rank regularizer for matrix factorization. arXiv preprint arXiv:1710.05092. 2017.
C. Wei, S. Kakade, and T. Ma. The implicit and explicit regularization effects of dropout. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 10181-10192. PMLR, 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-leved perfor- mance imagenet classification. In 2015.
R.M. Neal. Bayesian Learning for Neural Networks, volume 118. Springer Science & Business Media, 1995.
B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli. Exponential expressivity in deep neural networks through transient chaos. In Advances in Neural Information Processing Systems, 2016.
S.S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations, 2017.
G. Yang. Tensor programs ii:: Neural matrix laws. arXiv preprint arXiv:2009.1068) 2020.
I. Xiao, Y. Bahri, J. Sohl-Dickstein, S. S. Schoenholz, and P. Pennington. Dynamical isometry and mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. JCMI. 2018. 2018.
S. Hayou, A. Doucet, and J. Rousseau. On the impact of the activation function on deep neural networks training. In International Conference Machine Learning, 2019.
10

---

S. Hayou, A. Doucet, and J. Rousseau. Mean-fin field behaviour of neural tangent kernel for deep neural networks. arXiv preprint arXiv: 1905. 3654, 2020.
Hayou, E. Clerico, B. He, G. Deligiannidis,, A. Doucet, and Rousseau. Stable resnet. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pages 1324 1032, 2021a.
G. Yang and S. Schoenholz. Mean field residual networks: On the edge of chaos. In Advances in Neural Information Processing Systems, pages 7103-7114, 2017.
S. Hayou, J.F. Ton, Doucet, and Y.W. Teh. Robust pruning at initialization. In International Conference on Learning Representations, 2021b.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances Neural Information Processing Systems, 2018.
Arora, S.S. Du, W. Hu, Z. Li, R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large- batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: NXX::1669 2016.
Stanislaw Jastrzebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey, On the relation between the sharpest directions of dnn loss and the sgd step length. arXiv preprint / 1, 2018.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian- analysis of large batch training and robustness to adversaries. In Advances in Neural Information P Processing Systems, pages 4949-4959, 2018.
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-48. Springer, 2012.
Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametriied neural networks. arXiv preprint urXiv:1706.0444 1706.04454. 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Samuel Sanford Shapiro and Martin B Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591-611 1965.
J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, Novak, Sohl-Dickstein, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems. 2019.
A.G. Matthews, Hron, M. Rowland, R.E. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018.
T. Lillicrap. D. Cownden, D. Tweed, and C. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7(13276), 2016.
A. Neelakantan, L. Vilnis, Quoe V. Le, I. Sutskever, L. Kaiser, K. Kurach, and J. Martens. Adding gradient noise improves learning for very deep networks. arXiv prePrint 1511.06807, 2015.
RALPH B. D' Agosstno. Transformation normality of the null distribution of gl. Biometrika, 57(3): 679-681, 12 1970. ISSN 0006-3444. doi: 10. 20.0099/hiomme//7.3..799 URL https://d://d org/ 10. 10.1099/bbi htte://iommee///eeeeeeeeee omet/m7...... 30...//..m.m77...77.....


Reviews:
[{"review_id": "p6TCkJHQ4Dm", "reviewer": "Reviewer_Z7Sb", "summary": "The paper provides theoretical analysis for stochastic depth from a regularization perspective. The paper further illustrates the different regularization effects of stochastic depth based on perturbation analysis and signal propagation. The analysis leads to principled guidelines for choosing the survival rates.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow.\n\nThe paper discusses interesting regularization effects of stochastic depth such as flatness and information discrepancy.\n\nThere is no detail about \\phi in Line 284. It is important to provide detailed discussion about how to choose the linear mapping \\phi and how it impacts the performance.\n\nLine 314: it is not clear why normal behavior is more accurate for deeper layers under uniform mode with large budget. Does this implies SD only mimics Gaussian Noise Injection for deeper layers?\n\nSection 4 title seems incomplete.\n\nTheorem 4 is not provided in the paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations that are not addressed and suggestions are included in the above main review. The paper really needs proofreading and grammar check.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "klNFCXY3gFQ", "reviewer": "Reviewer_AyjF", "summary": "The paper focuses on understanding the regularization effect of stochastic depth in residual neural networks. The authors give 3 main results: \n\n(a) Stochastic depth helps to mitigate (to some extent) the gradient explosion phenomena observed at initialization in the Vanilla ResNet model, \n\n(b) Stochastic depth mimics Gaussian Noise Injection (GNI) in deeper models by implicitly adding some non-isotropic Gaussian noise at each layer during training. Thereby, they show that Stochastic depth has the same regularization effects as GNI.\n\n(c) They propose SenseMode, a principled method to choose the survival rates of each layer for training. SenseMode performs better than the existing methods in certain cases.\n\nThe authors corroborate all their propositions in an experimental study.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nThe main paper needs a lot of work before it can be published. Here are my questions and few suggestions.\n\n(1) How do you define the linear mapping $\\varphi$ in SenseMode? Is the definition arbitrary? Or do you sort the values and match them to equally spaced numbers between $[p_{\\min}, 1]$? How does this kind of mapping solve the issue in the small budget regime when the regularization contains the term $p_{l} (1 - p_{l})$ (which is a quadratic function)? \n\nAlso, it will be great to have similar experiments for Vanilla ResNet in table 2. Because all the propositions and experiments involved Vanilla ResNet and SenseMode was built on the structure of the Vanilla ResNet model.\n\n(2) How does proposition 1 change when you have batch norm in the network? Will you still observe exponential growth of $\\tilde{q}_{\\ell}(x, z)$? Or is it just going to be a linear growth? Is the $1/\\sqrt{L}$ factor still necessary with the batch norm? Because I see in [1], the authors also consider Vanilla ResNet while proposing the $1/\\sqrt{L}$ factor? It will be interesting to see if explosion still exists with batch-norm in experiments and stochastic depth still reduces exponential growth. Otherwise, I don't see a point in Proposition 1.\n\n(3) How do I read the plot in Figure 5? Can you please give more details in the experimental section about \"Gaussian Noise Injection\"? For example, what neurons do you consider for the left plot in Figure 5? Do you take the entire set of neurons to do the left plot and all the neurons per layer for the right plot?\n \n(4) Multiple issues in notation and presentation:\n\n(a) What is $G_{l}$? The authors need to specify the function clearly. Is it the function that connects $y_l$ to $y_{out}$? It's not the way to define a function, as has been mentioned in the paper.\n\n(b) Use of both $l$ and $\\ell$ is frustrating. For example, the reader has to differentiate the $l$ and $\\ell$ used in $H_{\\ell}$ and $\\zeta_{l}$ (equation 5).\n\n(c) Clearly demarcate Vanilla ResNet and Stable ResNet110. It will be good to have some diagram differentiating the two. As far as I have understood (and reviewed), Vanilla ResNet contains only linear layers and ReLU in their residual blocks. In contrast, Stable ResNet110 contains the sequence BatchNorm-ReLU -Convolution-BatchNorm in their residual blocks.  It's essential to clearly demarcate this since experiments in Figure 2, Figure 3, Figure 4, Figure 5 are done with Vanilla ResNet, and the experiments in Table 2 are done with Stable Resnet110. Also, at some places, you have mentioned Vanilla Stable Resnet. Would you please demarcate the terms properly?\n\n(d) The caption in table 1 says that the table reports $\\tilde{q}_l(x, z)$ at initialization. However, I don't see exponential numbers as claimed by proposition 1. Did you mean to write the growth rate (as mentioned in the text)?\n\n(5) Plots lack proper labels and captions:\n\n(a) Figure 1(b) doesn't have a label. What do the two lines in the graph refer to? \n\n(b) What are the numbers in Table 2(a) and (b)? Are they error or accuracy of the models in the task?\n\n(c) The x and y labels in all the plots are unreadable. I had to guess every time I read some plot (from a printed pdf).\n\n\n\n\n1: S. Hayou, E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau. Stable resnet. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pages 324–1332, 2021a.\n\n\n\n***After Rebuttal***\nI have increased my score to 6. The paper has many clarity issues and I would appreciate if the authors could incorporate changes to address the concerns raised by all the reviewers.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have mentioned different issues that exist in the main paper in my main review. The authors discuss the remaining limitations pretty well. As the authors have mentioned, the paper focuses on the general analysis of a regularization method. Hence I don't see a potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 - 5 hours"}, "extra_text": {}}, {"review_id": "VZqhG-cZiwd", "reviewer": "Reviewer_eU7J", "summary": "This paper studies the regularization effects involved in the stochastic depth training of ResNets. Specifically, by applying the second-order Taylor approximation to the loss function and the intermediate features, the authors show that the stochastic depth training involves a regularization term that promotes flatness and acts like Gaussian noise to the features. Based on this theoretical insight, the authors suggest a strategy for setting the survival probabilities of sub-networks depending on the computational budget, which corresponds to controlling the regularization effect. This strategy developed upon the new insight in the paper is validated by empirical experiments. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Stochastic depth is a simple yet effective regularization technique for training very deep ResNets. However, it has been unclear how the stochastic depth induces regularization effects and how to set dropout rates of sub-networks. In this regard, the authors present valuable theoretical analysis that shows a flatness promoting regularization term and layer-wise Gaussian noise effects involved in the stochastic depth training. I think that this novel insight can help to develop effective strategies for stochastic depth training and its novel variants, and as an example, the authors give a strategy for setting dropout rates depending on the computational budget. Considering the prevalence of ResNet in the machine learning community, I believe that this work will be of interest to both researchers and practitioners. \n\nThe paper is well-written, and the structure is easy-to-follow and interesting. However, there are few minor typos here and there (such as missing legend in Fig 1 (b), crossentropy (L159), The empirical loss given by (L160), hessian (footnote in p.5)). \n\n\n* Minor commnet: The authors use the Taylor approximation around the drop ratio of 1, which is highly unlikely in practice and can make higher-order terms non-negligible. I’m wondering if the authors can explain how the small drop rates can impact the regularization term in Eq 6 and Gaussian noise $\\gamma_{\\alpha, L}(x)$. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitation in the paper, and I agree with the authors' claim that this work will have no direct societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20"}, "extra_text": {}}, {"review_id": "313xAYOVxh2", "reviewer": "Reviewer_j3iT", "summary": "\nThe authors study stochastic depth from an theoretical point of view. In particular, they provide insights into the effective depth at each iteration (Lemma 1), discuss the relationship of stochastic depth with exploding gradients (Proposition 1), and relate stochastic depth with explicit regularization (Section 4.1) and Gaussian noise injection (Section 4.2). ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Their results are novel and provide first insights into stochastic depth from a mathematical perspective. One could argue that the results require many assumptions (such as the abundant, not always completely obvious approximations) and only scratch at the surface of stochastic depth (for example, what do these regularizers mean?). But a complete analytical understanding of stochastic depth, in my opinion, is currently out of reach, and the authors provide a first, reasonable step toward it.\n\nI have not read the proofs in the Appendix.\n\nThere is some work on layer-wise regularization that might be connected especially to Section 4.1:\n\nHebiri, M. and Lederer, J., 2020. Layer sparsity in neural networks. arXiv preprint arXiv:2006.15604.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper gives a theoretical analysis of the stochastic depth (SD) technique for ResNet. It is shown that SD mitigates the gradient explosion phenomenon. Moreover, it is shown that SD behaves like a Gaussian noise injection to the features in the internal layers and thus it works as a regularization. Finally, the authors proposed a method called SenseMode that determines the mode under a small computational budget limitation.   

The analysis given in this paper is interesting. It shed light to understand what SD is essentially doing. The proposed SenseMode is also useful.  
On the other hand, as the reviewers pointed out, there are some issues in its writing, i.e., typos, confusing notations and unclear descriptions. In particular, the definition of the linear mapping $\varphi$ in the definition of SenseMode should be explicitly given in the final version. I guess everybody would be confused at this point.    

In summary, this paper has sufficient novelty and gives a good insight for SD, which is beneficial to the community. I think this paper can be accepted by NeurIPS.
Decision: Accept (Poster)