Regularization in ResNet with Stochastic Depth
Year: 2022

Abstract:
Regularization plays a major role in modern deep learning. From classic techniques such as L1, L2 penalties to other noise-based methods such as Dropout, regularization often yields better generalization properties by avoiding overfitting. Recently, Stochastic Depth (SD) has emerged as an alternative regularization technique for residual neural networks (ResNets) and has proven to boost the performance of ResNet on many tasks [Huang et al., 2016]. Despite the recent success of SD, little is known about this technique from a theoretical perspective. This paper provides a hybrid analysis combining perturbation analysis and signal propagation to shed light on different regularization effects of SD. Our analysis allows us to derive principled guidelines for choosing the survival rates used for training with SD.

heading: 1 Introduction
text: Stochastic Depth (SD) is a well-established regularization method that was first introduced by Huang et al. [2016]. It is similar in principle to Dropout [Hinton et al., 2012, Srivastava et al., 2014] and DropConnect [Wan et al., 2013]. It belongs to the family of noise-based regularization techniques, which includes other methods such as noise injection in data [Webb, 1994, Bishop, 1995] and noise injection throughout the network [Camuto et al., 2020]. While Dropout, resp. DropConnect consists of removing some neurons, resp. weights, at each iteration, SD randomly drops full layers, and only updates the weights of the resulting subnetwork at each training iteration. As a result of this mechanism, SD can be exclusively used with residual neural networks (ResNets). There exists a stream of papers in the literature on the regularization effect of Dropout for linear models [Wager et al., 2013, Mianjy and Arora, 2019, Helmbold and Long, 2015, Cavazza et al., 2017]. Recent work by Wei et al. [2020] extended this analysis to deep neural networks using second-order perturbation analysis. It disentangled the explicit regularization of Dropout on the loss function and the implicit regularization on the gradient. Similarly, Camuto et al. [2020] studied the explicit regularization effect induced by adding Gaussian Noise to the activations and empirically illustrated the benefits of this regularization scheme. However, to the best of our knowledge, no analytical study of SD exists in the literature. This paper aims to fill this gap by studying the regularization effect of SD from an analytical point of view; this allows us to derive principled guidelines on the choice of the survival probabilities for network layers. Concretely, our contributions are four-fold:
• We show that SD acts as an explicit regularizer on the loss function by penalizing a notion of information discrepancy between keeping and removing certain layers.
• We prove that the uniform mode, defined as the choice of constant survival probabilities, is related to maximum regularization using SD.
• We study the large depth behaviour of SD and show that in this limit, SD mimics Gaussian Noise Injection by implicitly adding data-adaptive Gaussian noise to the pre-activations.
∗Equal contribution. Correspondence to: <soufiane.hayou@yahoo.fr; fadhel.ayed@huawei.com>
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
• By defining the training budget L̄ as the desired average depth, we show the existence of two different regimes: small budget and large budget regimes. We introduce a new algorithm called SenseMode to compute the survival rates under a fixed training budget and provide a series of experiments that validates our Budget hypothesis introduced in Section 5.

heading: 2 Stochastic Depth Neural Networks
text: Stochastic depth neural networks were first introduced by Huang et al. [2016]. They are standard residual neural networks with random depth. In practice, each block in the residual network is multiplied by a random Bernoulli variable δl (l is the block’s index) that is equal to 1 with some survival probability pl and 0 otherwise. The mask is re-sampled after each training iteration, making the gradient act solely on the subnetwork composed of blocks with δl = 1.
We consider a slightly different version where we apply the binary mask to the pre-activations instead of the activations. We define a depth L stochastic depth ResNet by
y0(x; δ) = Ψ0(x,W0),
yl(x; δ) = yl−1(x; δ) + δlΨl(yl−1(x; δ),Wl), 1 ≤ l ≤ L, yout(x; δ) = Ψout(yL(x; δ),Wout),
(1)
where Wl are the weights in the lth layer, Ψ is a mapping that defines the nature of the layer, yl are the pre-activations, and δ = (δl)1≤l≤L is a vector of Bernoulli variables with survival parameters p = (pl)1≤l≤L. δ is re-sampled at each iteration. For the sake of simplification, we consider constant width ResNet and we further denote by N the width, i.e. for all l ∈ [L− 1], yl ∈ RN . The output function of the network is given by s(yout) where s is some convenient mapping for the learning task, e.g. the Softmax mapping for classification tasks. We denote by o the dimension of the network output, i.e. s(yout) ∈ Ro which is also the dimension of yout. For our theoretical analysis, we consider a Vanilla model with residual blocks composed of a Fully Connected linear layer
Ψl(x,W ) = Wφ(x),
where φ(x) is the activation function. The weights are initialized with He init [He et al., 2015], e.g. for ReLU, W lij ∼ N (0, 2/N). There are no principled guidelines on choosing the survival probabilities. However, the original paper by Huang et al. [2016] proposes two alternatives that appear to make empirical consensus: the uniform and linear modes, described by
Uniform: pl = pL, Linear: pl = 1− lL (1− pL),
where we conveniently parameterize both alternatives using pL.
Training budget L̄. We define the training budget L̄ to be the desired average depth of the subnetworks with SD. The user typically fixes this budget, e.g., a small budget can be necessary when training is conducted on small capacity devices.
Depth of the subnetwork. Given the mode p = (pl)1≤l≤L, after each iteration, the subnetwork has a depth Lδ = ∑L l=1 δl with an average Lp := Eδ[Lδ] = ∑L l=1 pl. Given a budget L̄, there is a infinite number of modes p such that Lp = L̄. In the next lemma, we provide probabilistic bounds on Lδ using standard concentration inequalities. We also show that with a fixed budget L̄, the uniform mode is linked to maximal variability. Lemma 1 (Concentration of Lδ). For any β ∈ (0, 1), we have that with probability at least 1− β,
|Lδ − Lp| ≤ vp u−1 ( log(2/β)
vp
) , (2)
where Lp = E[Lδ] = ∑L l=1 pl, vp = Var[Lδ] = ∑L l=1 pl(1− pl), and u(t) = (1 + t) log(1 + t)− t.
Moreover, for a given average depth Lp = L̄, the upperbound in Eq. (2) is maximal for the uniform choice of survival probabilities p = ( L̄ L , ..., L̄ L ) .
(a) Distributions of Lδ for a Resnet100 with average survival rate L̄/L = 0.5 for the uniform and linear modes.
(b) Training time of Dropout and SD on CIFAR10 with ResNet56 for 100 epochs.
Lemma 1 shows that with high probability, the depth of the subnetwork that we obtain with SD is within an `1 error of vp u −1 (
log(2/β) vp
) from the average depth Lp. Given a fixed
budget L̄, this segment is maximized for the uniform mode p = (L̄/L, . . . , L̄/L). Fig. 1a highlights this result. This was expected since the variance of the depth Lδ is also maximized by the uniform mode. The uniform mode corresponds to maximum entropy of the random depth, which would intuitively results in maximum regularization. We depict this behaviour in more details in Section 4.
SD vs Dropout. From a computational point of view, SD has the advantage of reducing the effective depth during training. Depending on the chosen budget, the subnetworks might be significantly shallower than the entire network (Lemma 1). This depth compression can be effectively leveraged for computational training time gain (Fig. 1b). It is not the case with Dropout. Indeed, assuming that the choice of dropout probabilities is such that we keep the same number of parameters on average compared to SD, we still have to multiply matricesL times during the forward/backward propagation. It is not straightforward to leverage the sparsity obtained by Dropout for computational gain. In practice, Dropout requires an additional step of sampling the mask for every neuron, resulting in longer training times than without Dropout (Fig. 1b). However, there is a trade-off between how small the budget is and the performance of the trained model with SD (Section 6).

heading: 3 Effect of Stochastic Depth at initialization
text: Empirical evidence strongly suggests that Stochastic Depth allows training deeper models [Huang et al., 2016]. Intuitively, at each iteration, SD updates only the parameters of a subnetwork with average depth Lp = ∑ l pl < L, which could potentially alleviate any exploding/vanishing gradient issue. This phenomenon is often faced when training ultra deep neural networks. To formalize this intuition, we consider the model’s asymptotic properties at initialization in the infinite-width limit N → +∞. This regime has been the focus of several theoretical studies [Neal, 1995, Poole et al., 2016, Schoenholz et al., 2017, Yang, 2020, Xiao et al., 2018, Hayou et al., 2019, 2020, 2021a] since it allows to derive analytically the distributions of different quantities of untrained neural networks. Specifically, randomly initialized ResNets, as well as other commonly-used architectures such as Fully connected Feedforward networks, convolutional networks and LSTMs, are equivalent to Gaussian Processes in the infinite-width limit. An important ingredient in this theory is the Gradient Independence assumption. Let us formally state this assumption first.
Assumption 1 (Gradient Independence). In the infinite width limit, we assume that the weightsW used for back-propagation are an iid version of the weightsW used for forward propagation.
Assumption 1 is ubiquitous in the literature on the signal propagation in deep neural networks. It has been used to derive theoretical results on signal propagation in randomly initialized deep neural network [Schoenholz et al., 2017, Poole et al., 2016, Yang and Schoenholz, 2017, Hayou et al., 2021b,a] and is also a key tool in the derivation of the Neural Tangent Kernel [Jacot et al., 2018, Arora et al., 2019, Hayou et al., 2020]. Recently, it has been shown by Yang [2020] that Assumption 1 yields the exact computation for the gradient covariance in the infinite width limit. See Appendix A0.3 for a detailed discussion about this assumption. Throughout the paper, we provide numerical results that substantiate the theoretical results that we derive using this assumption. We show that Assumption 1 yields an excellent match between theoretical results and numerical experiments.
Leveraging this assumption, Yang and Schoenholz [2017], Hayou et al. [2021a] proved that ResNet suffers from exploding gradient at initialization. We show in the next proposition that SD helps mitigate the exploding gradient behaviour at initialization in infinite width ResNets.
Proposition 1. Let φ = ReLU and L(x, z) = `(yout(x; δ), z) for (x, z) ∈ Rd × Ro, where `(z, z′) is some differentiable loss function. Let q̃l(x, z) = EW,δ ‖∇ylL‖ 2
‖∇yLL‖2 , where the numerator and denomi-
nator are respectively the norms of the gradients with respect to the inputs of the lth and Lth layers . Then, in the infinite width limit, under Assumption 1, for all l ∈ [L] and (x, z) ∈ Rd × Ro, we have
• With Stochastic Depth, q̃l(x, z) = ∏L k=l+1(1 + pk),
• Without Stochastic Depth (i.e. δ = 1), q̃l(x, z) = 2L−l.
Proposition 1 indicates that with or without SD, the gradient explodes exponentially at initialization as it backpropagates through the network. However, with SD, the exponential growth is characterized by the mode p. Intuitively, if we choose pl 1 for some layer l, then the contribution of this layer in the exponential growth is negligible since 1 + pl ≈ 1. From a practical point of view, the choice of pl 1 means that the lth layer is hardly present in any subnetwork during training, thus making its contribution to the gradient negligible on average (w.r.t δ). For a ResNet with L = 50 and uniform mode p = (1/2, . . . , 1/2), SD reduces the gradient exploding by six orders of magnitude. Fig. 2a and Fig. 2b
illustrates the exponential growth of the gradient for the uniform and linear modes, as compared to the growth of the gradient without SD. We compare the empirical/theoretical growth rates of the magnitude of the gradient in Table 1; the results show a good match between our theoretical result (under Assumption 1) and the empirical ones. Further analysis can be found in Section A4. Stable ResNet. Hayou et al. [2021a] have shown that introducing the scaling factor 1/ √ L in front of the residual blocks is sufficient to avoid the exploding gradient at initialization, as illustrated in Figure 2c. The hidden layers in Stable ResNet (with SD) are given by,
yl(x; δ) = yl−1(x; δ) + δl√ L Ψl(yl−1(x; δ),Wl), 1 ≤ l ≤ L. (3)
The intuition behind the choice of the scaling factor 1/ √ L comes for the variance of yl. At initialization, with standard ResNet (Eq. (1)), we have Var[yl] = Var[yl−1] + Θ(1), which implies that Var[yl] = Θ(l). With Stable ResNet (Eq. (3)), this becomes Var[yl] = Var[yl−1] + Θ(1/L), resulting in Var[yl] = Θ(1) (See Hayou et al. [2021a] for more details). In the rest of the paper, we restrict our analysis to Stable ResNet; this will help isolate the regularization effect of SD in the limit of large depth without any variance/gradient exploding issue.
Nevertheless, the natural connection between SD and Dropout, coupled with the line of work on the regularization effect induced by the latter [Wager et al., 2013, Mianjy and Arora, 2019, Helmbold and Long, 2015, Cavazza et al., 2017, Wei et al., 2020], would indicate that the benefits of SD are not limited to controlling the magnitude of the gradient. Using a second order Taylor expansion, Wei et al. [2020] have shown that Dropout induces an explicit regularization on the loss function. Intuitively, one should expect a similar effect with SD. In the next section, we elucidate the explicit regularization effect of SD on the loss function, and we shed light on another regularization effect of SD that occurs in the large depth limit.

heading: 4 Regularization effect of Stochastic Depth

heading: 4.1 Explicit regularization on the loss function
text: Consider a dataset D = X × T consisting of n (input, target) pairs {(xi, ti)}1≤i≤n with (xi, ti) ∈ Rd × Ro. Let ` : Rd × Ro → R be a smooth loss function, e.g. quadratic loss, cross-entropy loss etc. Define the model loss for a single sample (x, t) ∈ D by
L(W , x; δ) = `(yout(x; δ), t), L(W , x) = Eδ [`(yout(x; δ), t)] , whereW = (Wl)0≤l≤L. The empirical loss given by L(W ) = 1n ∑n i=1 Eδ [`(yout(xi; δ), ti)] .
To isolate the regularization effect of SD on the loss function, we use a second order approximation of the loss function around δ = 1, this allows us to marginalize out the mask δ. The full derivation is provided in Appendix A2. Let zl(x; δ) = Ψl(Wl, yl−1(x; δ)) be the activations. For some pair (x, t) ∈ D, we obtain
L(W , x) ≈ L̄(W , x) + 1 2L L∑ l=1 pl(1− pl)gl(W , x), (4)
where L̄(W , x) ≈ `(yout(x;p), t) (more precisely, L̄(W , x) is the second order Taylor approximation of `(yout(x;p), t) around p = 12), and gl(W , x) = zl(x;1)T∇2yl [` ◦Gl](yl(x;1))zl(x;1) with Gl is the function defined by yout(x;1) = Gl(yl−1(x;1) + 1√Lzl(x;1)).
The first term L̄(W , x) in Eq. (4) is the loss function of the average network (i.e. replacing δ with its mean p). Thus, Eq. (4) shows that training with SD entails training the average network with an explicit regularization term that implicitly depends on the weightsW .
SD enforces flatness. The presence of the hessian in the penalization term provides a geometric interpretation of the regularization induced by SD: it enforces a notion of flatness determined by the hessian of the loss function with respect to the hidden activations zl. This flatness is naturally inherited by the weights, thus leading to flatter minima. Recent works by [Keskar et al., 2016, Jastrzebski et al., 2018, Yao et al., 2018] showed empirically that flat minima yield better generalization compared to minima with large second derivatives of the loss. Wei et al. [2020] have shown that a similar behaviour occurs in networks with Dropout.
Let Jl(x) = ∇ylGl(yl(x;1)) be the Jacobian of the output layer with respect to the hidden layer yl with δ = 1, and H(x) = ∇2z`(z)|z=yout(x;1) the hessian of the loss function `. The hessian matrix inside the penalization terms gl(W , x) can be decomposed as in [LeCun et al., 2012, Sagun et al., 2017]
∇2yl [` ◦Gl](yl(x;1)) = Jl(x) TH(x)Jl(x) + Γl(x),
where Γ depends on the hessian of the network output. Γ is generally non-PSD, and therefore cannot be seen as a regularizer. Moreover, it has been shown empirically that the first term generally dominates and drives the regularization effect [Sagun et al., 2017, Wei et al., 2020, Camuto et al., 2020]. Thus, we restrict our analysis to the regularization effect induced by the first term, and we consider the new version of gl defined by
gl(W , x) = ζl(x,W ) T H(x) ζl(x,W ) = Tr ( H(x) ζl(x,W )ζl(x,W ) T ) , (5)
where ζl(x,W ) = Jl(x)zl(x;1). The quality of this approximation is discussed in Appendix A4.
Information discrepancy. The vector ζl represents a measure of the information discrepancy between keeping and removing the lth layer. Indeed, ζl measures the sensitivity of the model output to the lth layer,
yout(x;1)− yout(x;1l) ≈ ∇δlyout(x; δ)|δ=1 = ζl(x,W ),
where 1l is the vector of 1′s everywhere with 0 in the lth coordinate. With this in mind, the regularization term gl in Eq. (5) is most significant when the information discrepancy is well-aligned with the hessian of the loss function, i.e. SD penalizes mostly the layers with information discrepancy that violates the flatness, confirming our intuition above.
2Note that we could obtain Eq. (4) using the Taylor expansion around δ = p. However, in this case, the Hessian will depend on p, which complicates the analysis of the role of p in the regularization term.
Quadratic loss. With the quadratic loss `(z, z′) = ‖z − z′‖22, the hessian H`(x) = 2I is isotropic, i.e. it does not favorite any direction over the others. Intuitively, we expect the penalization to be similar for all the layers. In this case, we have gl(W , x) = 2 ‖ζl(x)‖22, and the loss is given by
L(W ) ≈ L̄(W ) + 1 2L L∑ l=1 pl(1− pl)gl(W ), (6)
where gl(W ) = 2n ∑n i=1 ‖ζl(xi,W )‖22 is the regularization term marginalized over inputs X .
Eq. (6) shows that the mode p has a direct impact on the regularization term induced by SD. The latter tends to penalize mostly the layers with survival probability pl close to 50%. The mode p = (1/2, . . . , 1/2) is therefore a universal maximizer of the regularization term, given fixed weightsW . However, given a training budget L̄, the mode p that maximizes the regularization term 1 2L ∑L l=1 pl(1− pl)gl(W ) depends on the values of gl(W ). We show this in the next lemma. Lemma 2 (Max regularization). Consider the empirical loss L given by Eq. (6) for some fixed weights W (e.g. W could be the weights at any training step of SGD). Then, given a training budget L̄, the regularization is maximal for p∗l = min ( 1,max(0, 12 − Cgl(W ) −1) ) , where C is a normalizing constant, that has the same sign as L− 2L̄. The global maximum is obtained for pl = 1/2.
Lemma 2 shows that under fixed budget, the mode p∗ that maximizes the regularization induced by SD is generally layerdependent (6= uniform). However, we show that at initialization, on average (w.r.tW ), p∗ is uniform. Theorem 1 (p∗ is uniform at initialization). Assume φ = ReLU andW are initialized withN (0, 2N ). Then, in the infinite width limit, under Assumption 1, for all l ∈ [1 : L], we have
EW [gl(W )] = EW [g1(W )].
As a result, given a budget L̄, the average regularization term 1 2L ∑L l=1 pl(1 − pl)EW [gl(W )] is maximal for the uniform mode p∗ = (L̄/L, . . . , L̄/L).
The proof of Theorem 1 is based on some results from the signal propagation theory in deep neural network. We provide an overview of this theory in Appendix A0. Theorem 1 shows that, given a training budget L̄ and a randomly initialized ResNet with N (0, 2/N) and N large, the average (w.r.t W ) maximal regularization at initialization is almost achieved by the uniform mode. This is because the coefficients EW [gl(W )] are equal under Assumption 1, which we highlight in Fig. 3. As a result, we would intuitively expect that the uniform mode performs best when the budget L̄ is large, e.g. L is large and L̄ ≈ L, since in this case, at each iteration, we update the weights of an overparameterized subnetwork, which would require more regularization compared to the small budget regime. We formalize this intuition in Section 5.
In the next section, we show that SD is linked to another regularization effect that only occurs in the large depth limit; in this limit, we show that SD mimics Gaussian Noise Injection methods by adding Gaussian noise to the pre-activations.

heading: 4.2 Stochastic Depth mimics Gaussian noise injection
text: Recent work by Camuto et al. [2020] studied the regularization effect of Gaussian Noise Injection (GNI) on the loss function and showed that adding isotropic Gaussian noise to the activations zl improves generalization by acting as a regularizer on the loss. The authors suggested adding a zero mean Gaussian noise parameterized by its variance. At training time t, this translates to replacing ztl by z t l +N (0, σ2l I), where ztl is the value of the activations in the lth layer at training time t, and σ2l is a parameter that controls the noise level. Empirically, adding this noise tends to boost the performance by making the model robust to over-fitting. Using similar perturbation analysis as in the previous section, we show that when the depth is large, SD mimics GNI by implicitly adding a non-isotropic data-adaptive Gaussian noise to the pre-activations yl at each training iteration. We bring to the reader’s attention that the following analysis holds throughout the training (it is not limited to the initialization), and does not require the infinite-width regime.
Consider an arbitrary neuron yiαL in the (αL) th layer for some fixed α ∈ (0, 1). yiαL(x, δ) can be approximated using a first order Taylor expansion around δ = 1. We obtain similarly,
yiαL(x, δ) ≈ ȳiαL(x) + 1√ L αL∑ l=1 ηl 〈zl,∇ylGil(yl(x;1))〉 (7)
where Gil is defined by y i αL(x;1) = G i l(yl(x;1)), ηl = δl − pl, and ȳiαL(x) = yiαL(x,1) + 1√ L ∑αL l=1(pl − 1) 〈zl,∇ylGil(yl(x;1))〉 ≈ yiαL(x,p).
Let γα,L(x) = 1√L ∑αL l=1 ηl 〈zl,∇ylGil(yl(x;1))〉. With SD, yiαL(x; δ) can therefore be seen as a perturbed version of yiαL(x;p) (the pre-activation of the average network) with noise γα,L(x). The scaling factor 1/ √ L ensures that γα,L remains bounded (in `2 norm) as L grows. Without this scaling, the variance of γα,L will generally explode. The term γα,L captures the randomness of the binary mask δ, which up to a factor α, resembles to the scaled mean in Central Limit Theorem(CLT) and can be written as γα,L(x) = √ α × 1√
αL
∑αL l=2Xl,L(x) where Xl,L(x) = ηl 〈zl,∇ylGil(yl(x;1))〉.
Ideally, we would like to apply CLT to conclude on the Gaussianity of γα,L(x) in the large depth limit. However, the random variables Xl are generally not i.i.d (they have different variances) and they also depend on L. Thus, standard CLT argument fails. Fortunately, there is a more general form of CLT known as Lindeberg’s CLT which we use in the proof of the next theorem. Theorem 2. Let x ∈ Rd, Xl,L(x) = ηl µl,L(x) where µl,L(x) = 〈zl,∇ylGil(yl(x;1)〉, and σ2l,L(x) = Varδ[Xl,L(x)] = pl(1− pl)µl,L(x)2 for l ∈ [L]. Assume that
1. There exists a ∈ (0, 1/2) such that for all L, and l ∈ [L], pl ∈ (a, 1− a).
2. limL→∞ maxk∈[L] µ 2 k,L(x)∑L
l=1 µ 2 l,L(x)
= 0.
3. vα,∞(x) := limL→∞
∑L l=1 σ 2 l,L(x)
L exists and is finite.
Then, γα,L(x) D−→
L→∞ N (0, α vα,∞(x)).
Figure 4: (Theorem 2) Assumption 2 as a function the depth L and epoch.
Fig. 4 provides an empirical verification of the second condition of Theorem 2 across all training epochs. There is a clear downtrend as the depth increases; this trend is consistent throughout training, which supports the validity of the second condition in Theorem 2 at all training times. Theorem 2 shows that training a ResNet with SD involves
implicitly adding the noise γα,L(x) to yiαL. This noise becomes asymptotically normally distributed
3, confirming that SD implicitly injects input-dependent Gaussian noise in this limit. Camuto et al. [2020] studied GNI in the context of input-independent noise and concluded on the benefit of such methods on the overall performance of the trained network. We empirically confirm the results of Theorem 2 in Section 6 using different statistical normality tests.
Similarly, we study the implicit regularization effect of SD induced on the gradient in Appendix A3, and show that under some assumptions, SD acts implicitly on the gradient by adding Gaussian noise in the large depth limit.

heading: 5 The Budget Hypothesis
text: We have seen in Section 4 that given a budget L̄, the uniform mode is linked to maximal regularization with SD at initialization (Theorem 1). Intuitively, for fixed weightsW , the magnitude of standard regularization methods such as ‖.‖1 or ‖.‖2 correlates with the number of parameters; the larger the model, the bigger the penalization term. Hence, in our case, we would require the regularization term to correlate (in magnitude) with the number of parameters, or equivalently, the number of trainable layers. Assuming L 1, and given a fixed budget L̄, the number of trainable layers at each training
3The limiting variance vα,∞(x) depends on the input x, suggesting that γα,L(.) might converge in distribution to a Gaussian process in the limit of large depth, under stronger assumptions. We leave this for future work.
iteration is close to L̄ (Lemma 1). Hence, the magnitude of the regularization term should depend on how large/small the budget L̄ is, as compared to L. Small budget regime (L̄/L 1). In this regime, the effective depth Lδ of the subnetwork is small compared to L. As the ratio L̄/L gets smaller, the sampled subnetworks become shallower, suggesting that the regularization need not be maximal in this case, and therefore p should not be uniform in accordance with Theorem 1. Another way to look at this is through the bias-variance trade-off principle. Indeed, as L̄/L→ 0, the variance of the model decreases (and the bias increases), suggesting less regularization is needed. The increase in bias inevitably causes a deterioration of the performance; we call this the Budget-performance trade-off. To derive a more sensible choice of p for small budget regimes, we introduce a new Information Discrepancy based algorithm (Section 4). We call this algorithm Sensitivity Mode or briefly SenseMode. This algorithm works in two steps:
1. Compute the sensitivity (S) of the loss w.r.t the layer at initialization using the approximation,
Sl = L(W ;1)− L(W ;1l) ≈ ∇δlL(W ; δ)|δ=1.
Sl is a measure of the sensitivity of the loss to keeping/removing the lth layer. 2. Use a mapping ϕ to map S to the mode, p = ϕ(S), where ϕ is a linear mapping from
the range of S to [pmin, 1] and pmin is the minimum survival rate (fixed by the user). φ is the linear mapping from the range of S to the segment [pmin, 1]. In other words, pl = pmin + α × Sl, where the constant alpha is chosen in order to satisfy the budget constraint: ∑ l pl = L̃.
Large budget regime (L̄/L ∼ 1). In this regime, the effective depth Lδ of the subnetworks is close to L, and thus, we are in the overparameterized regime where maximal regularization could boost the performance of the model by avoiding over-fitting. Thus, we anticipate the uniform mode to perform better than other alternatives in this case. We are now ready to formally state our hypothesis, Budget hypothesis. Assuming L 1, the uniform mode outperforms SenseMode in the large budget regime, while SenseMode outperforms the uniform mode in the small budget regime.
We empirically validate the Budget hypothesis and the Budget-performance trade-off in Section 6.

heading: 6 Experiments
text: The objective of this section is two-fold: we empirically verify the theoretical analysis developed in sections 3 and 4 with a Vanilla ResNet model on a toy regression task; we also empirically validate the Budget Hypothesis on the benchmark datasets CIFAR-10 and CIFAR-100 [Krizhevsky et al., 2009]. Notebooks and code to reproduce all experiments, plots and tables presented are available in the supplementary material. We perform comparisons at constant training budgets.
Implementation details: Vanilla Stable ResNet is composed of identical residual blocks each formed of a Linear layer followed by ReLU. ResNet110 follows [He et al., 2016, Huang et al., 2016]; it comprises three groups of residual blocks; each block consists of a sequence ConvolutionBatchNorm-ReLU-Convolution-BatchNorm. We use the adjective "Stable" (Stable Vanilla ResNet, Stable ResNet110) to indicate that we scale the blocks using a factor 1/ √ L as described in Section 3. We build on an open-source implementation of standard ResNet4. The toy regression task consists of estimating the function fβ : x 7→ sin(βTx), where the inputs x and parameter β are in R256, sampled from a standard Gaussian. CIFAR-10, CIFAR-100 contain 32-by-32 color images, representing respectively 10 and 100 classes of natural scene objects. We present here our main conclusions. Further implementation details and other insightful results are in the Appendix A4.
Gaussian Noise Injection: We proceed by empirically verifying the Gaussian behavior of the neurons as described in Theorem 4. For each input x, we sample 200 masks and the corresponding y(x; δ). We then compute the p-value pvx of the Shapiro-Wilk test of normality [Shapiro and Wilk, 1965]. In Fig. 5 we represent the distribution of the p-values {pvx | x ∈ X}. We can see that the Gaussian behavior holds throughout training (left). On the right part of the Figure, we can see that the Normal behavior becomes accurate after approximately 20 layers. In the Appendix we report further experiments with different modes, survival rates, and a different test of normality to verify both Theorem 2 and the critical assumption 2.
4https://github.com/felixgwu/img_classification_pk_pytorch
Empirical verification of the Budget Hypothesis: We compare the three modes: Uniform, Linear, and SenseMode on two benchmark datasets using a grid survival proportions. The values and standard deviations reported are obtained using four runs. For SenseMode, we use the simple rule pl ∝ |Sl|, where Sl is the sensitivity (see section 5). We report in Table 2 the results for Stable ResNet110. Results with Stable ResNet56 are reported in Appendix A4.
L̄/L Uniform SenseMode Linear
0.1 55.2 ± 0.4 51.3 ± 0.6 − 0.2 38.3 ± 0.3 36.4 ± 0.4 − 0.3 31.4 ± 0.2 30.1 ± 0.5 − 0.4 30.9 ± 0.2 28.5 ± 0.4 − 0.5 28.4 ± 0.3 29.5 ± 0.5 36.5 ± 0.4 0.6 26.5 ± 0.4 29.9 ± 0.6 30.9 ± 0.4 0.7 25.8 ± 0.1 29.5 ± 0.3 27.3 ± 0.3 0.8 25.5 ± 0.1 30.0 ± 0.3 25.7 ± 0.2 0.9 25.5 ± 0.3 28.3 ± 0.2 25.5 ± 0.2 1 26.5 ± 0.2
(b) CIFAR100
The empirical results are coherent with the Budget Hypothesis. When the training budget is large, i.e. L̄/L ≥ 0.5, the Uniform mode outperforms the others. We note nevertheless that when L̄/L ≥ 0.9, the Linear and Uniform models have similar performance. This seems reasonable as the Uniform and Linear probabilities become very close for such survival proportions. When the budget is low, i.e. L̄/L < 0.5, the SenseMode outperforms the uniform one (the linear mode cannot be used with budgets L̄ < L/2 when L 1, since ∑ pl/L > 1/2−1/(2L) ∼ 1/2), thus confirming the Budget hypothesis. Table 2 also shows a clear Budget-performance trade-off.

heading: 7 Related work
text: The regularization effect of Dropout in the context of linear models has been the topic of a stream of papers [Wager et al., 2013, Mianjy and Arora, 2019, Helmbold and Long, 2015, Cavazza et al., 2017]. This analysis has been recently extended to neural networks by Wei et al. [2020] where authors used a similar approach to ours to depict the explicit and implicit regularization effects of Dropout. To the best of our knowledge, our paper is the first to provide analytical results for the regularization effect of SD, and study the large depth behaviour of SD, showing that the latter mimics Gaussian Noise Injection [Camuto et al., 2020]. A further analysis of the implicit regularization effect of SD is provided in Appendix A3.

heading: 8 Limitations and extensions
text: In this work, we provided an analytical study of the regularization effect of SD using a second order Taylor approximation of the loss function. Although the remaining higher order terms are usually dominated by the second order approximation (See the quality of the second order approximation in Appendix A4), they might also be responsible for other regularization effects. This is a common limitation in the literature on noise-based regularization in deep neural networks [Camuto et al., 2020, Wei et al., 2020]. Further research is needed to isolate the effect of higher order terms. We also believe that SenseMode opens an exciting research direction knowing that the low budget regime has not been much explored yet in the literature. We believe that one can probably get even better results with more elaborate maps ϕ such that p = ϕ(S). Another interesting extension of an algorithmic nature is the dynamic use of SenseMode throughout training. We are currently investigating this topic which we leave for future work.

Review summaries:
The paper focuses on understanding the regularization effect of stochastic depth in residual neural networks. The authors give 3 main results:   (a) Stochastic depth helps to mitigate (to some extent) the gradient explosion phenomena observed at initialization in the Vanilla ResNet model,   (b) Stochastic depth mimics Gaussian Noise Injection (GNI) in deeper models by implicitly adding some non-isotropic Gaussian noise at each layer during training. Thereby, they show that Stochastic depth has the same regularization effects as GNI.  (c) They propose SenseMode, a principled method to choose the survival rates of each layer for training. SenseMode performs better than the existing methods in certain cases.  The authors corroborate all their propositions in an experimental study.

The paper provides theoretical analysis for stochastic depth from a regularization perspective. The paper further illustrates the different regularization effects of stochastic depth based on perturbation analysis and signal propagation. The analysis leads to principled guidelines for choosing the survival rates. 

This paper studies the regularization effects involved in the stochastic depth training of ResNets. Specifically, by applying the second-order Taylor approximation to the loss function and the intermediate features, the authors show that the stochastic depth training involves a regularization term that promotes flatness and acts like Gaussian noise to the features. Based on this theoretical insight, the authors suggest a strategy for setting the survival probabilities of sub-networks depending on the computational budget, which corresponds to controlling the regularization effect. This strategy developed upon the new insight in the paper is validated by empirical experiments.