Title: AVOIDING DEGRADATION IN DEEP FEED-FORWARD NETWORKS BY PHASING OUT SKIP-CONNECTIONS Anonymous

Abstract: A widely observed phenomenon in deep learning is the degradation problem: increasing the depth of a network leads to a decrease in performance on both test and training data. Novel architectures such as ResNets and Highway networks have addressed this issue by introducing various flavors of skip-connections or gating mechanisms. However, the degradation problem persists in the context of plain feed-forward networks. In this work we propose a simple method to address this issue. The proposed method poses the learning of weights in deep networks as a constrained optimization problem where the presence of skip-connections is penalized by Lagrange multipliers. This allows for skip-connections to be introduced during the early stages of training and subsequently phased out in a principled manner. We demonstrate the benefits of such an approach with experiments on MNIST, fashion-MNIST, CIFAR-10 and CIFAR-100 where the proposed method is shown to greatly decrease the degradation effect and is often competitive with ResNets.

1. INTRODUCTION:

The representation view of deep learning suggests that neural networks learn an increasingly abstract representation of input data in a hierarchical fashion (Zeiler & Fergus, 2014; Goodfellow et al., 2016; Greff et al., 2016) . Such representations may then be exploited to perform various tasks such as image classification, machine translation and speech recognition.

A natural conclusion of the representation view is that deeper networks will learn more detailed and abstract representations as a result of their increased capacity. However, in the case of feed-forward networks it has been observed that performance deteriorates beyond a certain depth, even when the network is applied to training data. Recently, Residual Networks (ResNets; He et al. 2016a) and Highway Networks (Srivastava et al., 2015) have demonstrated that introducing various flavors of skip-connections or gating mechanisms makes it possible to train increasingly deep networks. However, the aforementioned degradation problem persists in the case of plain deep networks (i.e., networks without skip-connections of some form).

A widely held hypothesis explaining the success of ResNets is that the introduction of skipconnections serves to improve the conditioning of the optimization manifold as well as the statistical properties of gradients employed during training. Raiko et al. (2012) and Schraudolph (2012) show that the introduction of specially designed skip-connections serves to diagonalize the Fisher information matrix, thereby bringing standard gradient steps closer to the natural gradient. More recently, Balduzzi et al. (2017) demonstrated that the introduction of skip-connections helps retain the correlation structure across gradients. This is contrary to the gradients of deep feed-forward networks, which resemble white noise. More generally, the skip-connections are thought to reduce the effects of vanishing gradients by introducing a linear term (He et al., 2016b) .

The goal of this work is to address the degradation issue in plain feed-forward networks by leveraging some of the desirable optimization properties of ResNets. We approach the task of learning parameters for a deep network under the framework of constrained optimization. This strategy allows us to introduce skip-connections penalized by Lagrange multipliers into the architecture of our network. In our setting, skip-connections play an important role during the initial training of the network and are subsequently removed in a principled manner. Throughout a series of experiments we demonstrate that such an approach leads to improvements in generalization error when compared to architectures without skip-connections and is competitive with ResNets in some cases.

The contributions of this work are as follows:

• We propose alternative training strategy for plain feed-forward networks which reduces the degradation in performance as the depth of the network increases. The proposed method introduces skip-connections which are penalized by Lagrange multipliers. This allows for the presence of skip-connections to be iteratively phased out during training in a principled manner. The proposed method is thereby able to enjoy the optimization benefits associated with skip-connections during the early stages of training. • A number of benchmark datasets are used to demonstrate the empirical capabilities of the proposed method. In particular, the proposed method greatly reduces the degradation effect compared to plain networks and is on several occasions competitive with ResNets.

2. RELATED WORK:

The hierarchical nature of many feed-forward networks is loosely inspired by the structure of the visual cortex where neurons in early layers capture simple features (e.g., edges) which are subsequently aggregated in deeper layers (Hubel & Wiesel, 1962) . This interpretation of neural networks suggests that the depth of a network should be maximized, thereby allowing the network to learn more abstract (and hopefully useful) representations (Bengio et al., 2013) . However, a widely reported phenomenon is that deeper networks are more difficult to train. This is often termed the degradation effect in deep networks (Srivastava et al., 2015; He et al., 2016a) . This effect has been partially attributed to optimization challenges such as vanishing and shattered gradients (Hochreiter et al., 2001; Balduzzi et al., 2017) .

In the past these challenges have been partially addressed via the use of supervised and unsupervised pre-training (Bengio et al., 2009) and more recently through careful parameter initialization (Glorot & Bengio, 2010; He et al., 2015) and batch normalization (Ioffe & Szegedy, 2015) . In the past couple of years further improvements have been obtained via the introduction of skip-connections.

ResNets (He et al., 2016a; b) introduce residual blocks consisting of a residual function F together with a skip-connection. Formally, the residual block is defined as:

x l+1 = F l (x l , W l ) + W l x l (1)

where F l : R n → R n represents some combination of affine transformation, non-linearity and batch normalization parameterized by W l . The matrix W l parameterizes a linear projection to ensure the dimensions are aligned1 . More generally, ResNets are closely related to Highway Networks (Srivastava et al., 2015) where the output of each layer is defined as:

EQUATION

where • denotes element-wise multiplication. In Highway Networks the output of each layer is determined by a gating function

T (x l , H l ) = sigmoid (H l x l )

inspired from LSTMs. We note that both ResNets and Highway Networks were introduced with the explicit goal of training deeper networks. Inspired by the success of the these methods, many variations have been proposed. Huang et al. (2016a) propose DenseNet, where skip-connections are passed from all previous activations. Huang et al. (2016b) propose to shorten networks during training by randomly dropping entire layers, leading to better gradient flow and information propagation, while using the full network at test time.

Recently, the goal of learning deep networks without skip-connections has begun to receive more attention. Zagoruyko & Komodakis (2017) propose a novel re-parameterization of weights in feedforward networks which they call the Dirac parameterization. Instead of explicitly adding a skipconnection, they model the weights as a residual of the Dirac function, effectively moving the skipconnection inside the non-linearity. In related work, Balduzzi et al. (2017) propose to initialize weights in a CReLU activation function in order to preserve linearity during the initial phases of training. This is achieved by initializing the weights in a mirrored block structure. During training the weights are allowed to diverge, resulting in non-linear activations.

Finally, we note that while the aforementioned approaches have sought to train deeper networks via modifications to the network architecture (i.e., by adding skip-connections) success has also been obtained by modifying the non-linearities (Clevert et al., 2015; Klambauer et al., 2017) .

3. VARIABLE ACTIVATION NETWORKS:

The goal of this work is to train deep feed-forward networks without suffering from the degradation problem described in previous sections. To set notation, we denote x 0 as the input and x L as the output of a feed-forward network with L layers. Given training data {y, x 0 } it is possible to learn parameters {W l } L l=1 by locally minimizing some objective function

EQUATION

First-order methods are typically employed due to the complexity of the objective function in equation (3). However, directly minimizing the objective is not practical in the context of deep networks: beyond a certain depth performance quickly deteriorates on both test and training data. Such a phenomenon does not occur in the presence of skip-connections. Accordingly, we take inspiration from ResNets and propose to modify equation ( 1) in the following manner2 :

EQUATION

where α l ∈ [0, 1] n determines the weighting given to the skip-connection. More specifically, α l is a vector were the entry i dictates the presence and magnitude of a skip-connection for neuron i in layer l. Due to the variable nature of parameters α l in equation ( 4), we refer to networks employing such residual blocks as Variable Activation Networks (VAN).

The objective of the proposed method is to train a feed-forward network under the constraint that α l = 1 for all layers, l. When the constraint is satisfied all skip-connections are removed. The advantage of such a strategy is that we only require α l = 1 at the end of training. This allows us to initialize α l to some other value, thereby relaxing the optimization problem and obtaining the advantages associated with ResNets during the early stages of training. In particular, whenever α l = 1 information is allowed to flow through the skip-connections, alleviating issues associated with shattered and vanishing gradients.

As a result of the equality constraint on α l , the proposed activation function effectively does not introduce any additional parameters. All remaining weights can be trained by solving the following constrained optimization problem:

{ Ŵl } L l=1 = argmin C y, x L ; {W l , α l } L l=1

such that α l = 1 for l = 1, . . . , L.

(5)

The associated Lagrangian takes the following simple form (Boyd & Vandenberghe, 2004) :

EQUATION

where each λ l ∈ R n are the Lagrange multipliers associated with the constraints on α l . In practice, we iteratively update α l via stochastic gradients descent (SGD) steps of the form:

α l ← α l -η ∂C ∂α l + λ l (7)

where η is the step-size parameter for SGD. Throughout the experiments we will often take the non-linearity in F l to be ReLU. Although not strictly required, we clip the values α l to ensure they remain in the interval [0, 1] n .

From equation ( 6), we have that the gradients with respect to Lagrange multipliers are of the form:

EQUATION

We note that since we require α l ∈ [0, 1] n , the values of λ l are monotonically decreasing. As the value of Lagrange multiplier decreases, this in turn pushes α l towards 1 in equation ( 7). We set the step-size for the Lagrange multipliers, η , to be a fraction of η. The motivation behind such a choice is to allow the network to adjust as we enforce the constraint on α l .

4. EXPERIMENTS:

The purpose of the experiments presented in this section is to demonstrate that the proposed method serves to effectively alleviate the degradation problem in deep networks. We first demonstrate the capabilities of the proposed method using a simple, non-convolutional architecture on the MNIST and Fashion-MNIST datasets (Xiao et al., 2017) in Section 4.1. More extensive comparisons are then considered on the CIFAR datasets (Krizhevsky & Hinton, 2009) in Section 4.2.

4.1. MNIST AND FASHION-MNIST:

Networks of varying depths were trained on both MNIST and Fashion-MNIST datasets. Following Srivastava et al. (2015) the networks employed in this section were thin, with each layer containing 50 hidden units. In all networks the first layer was a fully connected plain layer followed by l layers or residual blocks (depending on the architecture) and a final softmax layer. The proposed method is benchmarked against several popular architectures such as ResNets and Highway Networks as well as the recently proposed DiracNets (Zagoruyko & Komodakis, 2017) . Plain networks without skipconnections are also considered. Finally, we also considered VAN network where the constraint α l = 1 was not enforced. This corresponds to the case where λ l = 0 for all l. This comparison is included in order to study the capacity and flexibility of VAN networks without the need to satisfy the constraint to remove skip-connections. For clarity, refer to such networks as VAN (λ = 0) networks.

For all architectures the ReLU activation function was employed together with batch-normalization.

In the case of ResNets and VAN, the residual function consisted of batch-normalization followed by ReLU and a linear projection.

The depth of the network varied from l = 1 to l = 30 hidden layers. All networks were trained using SGD with momentum. The learning rate is fixed at η = 0.001 and the momentum parameter at 0.9.

Training consisted of 50 epochs with a batch-size of 128. In the case of VAN networks the α l values were initialized to 0 for all layers. As such, during the initial stages of training VAN networks where equivalent to ResNets. The step-size parameter for Lagrange multipliers, η , was set to be one half of the SGD step-size, η. Finally, all Lagrange multipliers, λ l , are initialized to -1.

RESULTS:

The results are shown in Figure 1 where the test accuracy is shown as a function of the network depth for both the MNIST and Fashion-MNIST datasets. In both cases we see clear evidence of the degradation effect: the performance of plain networks deteriorates significantly once the network depth exceeds some critical value (approximately 10 layers). As would be expected, this is not the case for ResNets, Highway Networks and DiracNets as such architectures have been explicitly designed to avoid this behavior. We note that VAN networks do not suffer such a pronounced degradation as the depth increases. This provides evidence that the gradual removal of skip-connections via Lagrange multipliers leads to improved generalization performance compared to plain networks. Finally, we note that VAN networks obtain competitive results across all depths. Crucially, we note that VAN networks outperform plain networks across all depths, suggesting that the introduction of variable skip-connections may lead to convergence at local optima with better generalization performance. Finally, we note that VAN (λ = 0) networks, where no constraint is placed on skip-connections, obtain competitive results across all depths. We note that with the exception of plain networks, the performance of all remaining architectures is stable as the number of layers increases.

4.2. CIFAR:

As a more challenging benchmark we consider the CIFAR-10 and CIFAR-100 datasets. These consist of 60000 32×32 pixel color images with 10 and 100 classes respectively. The datasets are divided into 50000 training images and 10000 test images.

We follow He et al. (2016a) and train deep convolutional networks consisting of four blocks each consisting of n residual layers. The residual function is of the form conv-BN-ReLU-conv-BN-ReLU. This corresponds to the pre-activation function (He et al., 2016b) . The convolutional layers consist of 3 × 3 filters with downsampling at the beginning of blocks 2, 3 and 4. The network with a fully connected softmax layer, resulting in a depth of 8n + 2. The architecture is described in Table 1 .

Networks were trained using SGD with momentum over 165 epochs. The learning rate was set to η = 0.1 and divided by 10 at the 82nd and 125th epoch. The momentum parameter was set to 0.9. Networks were trained using mini-batches of size 128. Data augmentation followed Lee et al. (2015) : this involved random cropping and horizontal flips. Weights were initialized following He et al. (2015) . As in Section 4.1, we initialize α l = 0 for all layers. Furthermore, we set the step-size parameter for the Lagrange multipliers, η , to be one tenth of η and all Lagrange multipliers, λ l , are initialized to -1. On CIFAR-10 we ran experiments with n ∈ {1, 2, 3, 4, 5, 6, 8, 10} yielding networks with depths ranging from 10 to 82. For CIFAR-100 experiments were run with n ∈ {1, 2, 3, 4}. 

Results for experiments on CIFAR-10 are shown in Figure 2 . The left panel shows the mean test accuracy over five independent training sessions for ResNets, VAN, VAN (λ = 0) and plain networks. While plain networks provide competitive results for networks with fewer than 30 layers, their performance quickly deteriorates thereafter. We note that a similar phenomenon is observed in VAN networks but the effect is not as dramatic. In particular, the performance of VANs is similar to ResNets for networks with up to 40 layers. Beyond this depth, ResNets outperform VAN by an increasing margin. This holds true for both VAN and VAN (λ = 0) networks, however, the difference is reduced in magnitude in the case of VAN (λ = 0) networks. These results are in line with He et al. (2016b) , who argue that scalar modulated skip-connections (as is the case in VANs where the scalar is 1 -α l ) will either vanish or explode in very deep networks whenever the scalar is not the identity.

The right panel of Figure 2 shows the training and test error for a 26 layer network. We note that throughout all iterations, both the test and train accuracy of the VAN network dominates that of the plain network. The thick gold line indicates the mean residuals of the α l parameters across all layers. This is defined as

1 L L l=1 (1 -α l ) 2

and is a measure of the extent to which skip-connections are present in the network. Recall that if all α l values are set to one then all skip-connections are removed (see equation ( 4)). From Figure 2 , it follows that skip-connections are fully removed from the VAN network at approximately the 120 th iteration. More detailed traces of Lagrange multipliers and α l are provided in Appendix B.

A comparison of the performance of VAN networks in provided in Table 2 . We note that while VAN networks do not outperform ResNets, they do outperform other alternatives such as Highway networks and FitNets (Romero et al., 2014) when networks of similar depths considered. However, it is important to note that both Highway networks and FitNets did not employ batch-normalization, which is a strong regularizer. In the case of both VAN and VAN (λ = 0) networks, the best performance is obtained with networks of 26 layers while ResNets continue to improve their performance as depth increases. Finally, current state-of-the-art performance, obtained by Wide ResNets (Zagoruyko & Komodakis, 2016) and DenseNet Huang et al. (2016a) , are also provided in Table 2 Figure 3 provides results on the CIFAR-100 dataset. This dataset is considerably more challenging as it consists of a larger number of classes as well as fewer examples per class. As in the case of CIFAR-10, we observe a fall in the performance of both VAN and plain networks beyond a certain depth; in this case approximately 20 layers for plain networks and 30 layers for VANs. Despite this drop in performance, Table 2 indicates that the performance of VAN networks with both 18 and 26 layers are competitive with many alternatives proposed in the literature. Furthermore, we note that the performance of VAN (λ = 0) networks is competitive with ResNets in the context of the CIFAR-100 dataset. 

A VAN RE-FORMULATION:

The original formulation for the VAN residual block was as follows:

EQUATION

We thank an anonymous reviewer for suggesting that such a formulation may be detrimental to the performance of very deep VAN networks. The reason for this is that scaling constant within each block is always less than one, implying that the contributions of lower layers vanish exponentially as the depth increases. This argument is also provided in He et al. (2016b) who perform similar experiments with ResNets.

In order to validate this hypothesis, we compare the performance of VAN networks employing the residual block described in equation ( 4) and the residual block described in equation ( 9). The results, shown in Figure 4 , provide evidence in favor of the proposed hypothesis. While both formulations for VAN networks obtain similar performances for shallow networks, as the depth of the network increases there is a more pronounced drop in the performance of VAN networks which employ residual blocks described in equation ( 9).

In a further experiment, we also studied the performance of ResNets with the following residual block:

EQUATION

)

The results in Figure 4 demonstrate that ResNets which employ the residual blocks defined in equation (10) show a clear deterioration in performance as the depth of the network increases. Such a degradation in performance is not present when standard ResNets are employed. 

B LAGRANGE MULTIPLIER TRACES:

In this section we provide addition figures demonstrating the evolution of Lagrange multipliers, λ l throughout training. We note that the updates to Lagrange multipliers are directly modulated by the current value of each α l (see equation ( 8)). As such, we also visualize the mean residuals of the α l parameters across all layers. This is defined as 1 L L l=1 (1 -α l ) 2 and is a measure of the extent to which skip-connections are present in the network. Once all skip-connections have been removed, this residual will be zero and the values of Lagrange multipliers will no longer change.

This is precisely what we find in Figure 5 . The left panel plots the mean value of Lagrange multipliers across all layers, while the right panel shows the mean residual of α l . We observe that for networks of different depths, once the constraint to remove skip-connections is satisfied, the value of Lagrange multipliers remains constant. This occurs at different times; sooner for more shallow networks whilst later on for deeper networks. 

:

Unless stated otherwise we will assume F retains the dimension of x l and set W l to the identity.

Our original formulation consisted of a convex sum over F and x l . However, we found this approach to be the most successful empirically. Appendix A provides a comparison to original formulations as well as some theoretical justifications for this improvement in performance.

Figures and Tables:

Figure1: Results on MNIST (left) and fashion-MNIST (right) for various different architectures as the depth of the network varies from 1 to 30. Mean average test accuracy over 10 independent training sessions is shown. We note that with the exception of plain networks, the performance of all remaining architectures is stable as the number of layers increases.

Figure 2: Left: Results on CIFAR-10 dataset are shown as the depth of networks increase. We note that the performance of both VAN and plain networks deteriorates as the depth increases, but the effect is far less pronounced for VAN networks. Right: Training and test error curves are shown for networks with 26 layers. We also plot the mean α residuals: 1 L

Figure 3: Left: Results on CIFAR-100 dataset are shown as the depth increases from 10 to 34 layers. We note that the performance of both VAN and plain networks deteriorates as the depth increases, but the effect is far less pronounced for plain networks. Right: Training and test error curves are shown for VAN and plain networks with 18 layers. The mean α residuals, 1 L

Figure 4: Results are shownVAN and  ResNet networks with various different residual blocks. We note that the use of residual blocks with non-identity scaling coefficients leads to a larger drop in performance as the network depth increases. This drop is attributed to vanishing contributions from lower blocks (as all scalings are less than one).

Figure 5: Lagrange multipliers, λ l are shown on the left panel for networks of varying depth. After a certain number of iterations, the values of the Lagrange multiplier plateau as the constraint to remove skip-connections is satisfied. This results in no updates to the values of the Lagrange multipliers (see equation (8)). The right panel shows the mean α l residual. This residual directly modulates the magnitude of changes in Lagrange multipliers.

Architecture of varying depth employed on CIFAR-10 and CIFAR-100 datasets. Downsampling occurs at the beginning of block 2, 3 and 4 with a stride of two.

Table content: <table><tr><td colspan="2">Layer name Output size</td><td>Convolution</td></tr><tr><td>conv1</td><td>32 × 32</td><td>3 × 3, 64</td></tr><tr><td>Block 1</td><td>32 × 32</td><td>3×3,64 3×3,64 × n</td></tr><tr><td>Block 2</td><td>16 × 16</td><td>3×3,128 3×3,128 × n</td></tr><tr><td>Block 3</td><td>8 × 8</td><td>3×3,256 3×3,256 × n</td></tr><tr><td>Block 4</td><td>4 × 4</td><td>3×3,512 3×3,512 × n</td></tr></table>

Comparison of VAN networks to results of other convolutional networks on CIFAR-10 and CIFAR-100. For VAN networks we report the best value as well as the mean and standard deviation over five independent training runs (in brackets). Results are ordered by performance on CIFAR-10. We add a * to denote results which did not employ batch-normalization. In the past, the degradation issue has been successfully addressed via the introduction of skip-connections. As such, the goal of this work is to propose a new training regime which retains the optimization benefits associated with ResNets while ultimately phasing out skip-connections. This is achieved by posing network training as a constrained optimization problem where skip-connections are introduced during the early stages of training and subsequently phased out in a principled manner using Lagrange multipliers.

Table content: <table><tr><td>Architecture</td><td colspan="3"># Layers CIFAR-10 (test error %) CIFAR-100 (test error %)</td></tr><tr><td>Highway Network  *</td><td>32</td><td>8.80</td><td>-</td></tr><tr><td>FitNet  *</td><td>19</td><td>8.39</td><td>35.04</td></tr><tr><td>Highway Network  *</td><td>19</td><td>7.54</td><td>32.39</td></tr><tr><td>DiracNet (width-1)</td><td>34</td><td>7.10</td><td>-</td></tr><tr><td>ELU  *</td><td>18</td><td>6.55</td><td>24.28</td></tr><tr><td>VAN (λ = 0)</td><td>26</td><td>6.29 (6.40 ± 0.16)</td><td>27.04 (27.42 ± 0.26)</td></tr><tr><td>VAN (λ = 0)</td><td>34</td><td>6.28 (6.45 ± 0.14)</td><td>26.46 (26.81 ± 0.31)</td></tr><tr><td>VAN</td><td>18</td><td>6.23 (6.49 ± 0.16)</td><td>28.20 (28.42 ± 0.36)</td></tr><tr><td>VAN</td><td>26</td><td>6.08 (6.35 ± 0.21)</td><td>27.70 (28.01 ± 0.39)</td></tr><tr><td>DiracNet (width-2)</td><td>34</td><td>5.60</td><td>26.72</td></tr><tr><td>ResNet</td><td>164</td><td>5.46</td><td>24.33</td></tr><tr><td>Wide ResNet (width-10)</td><td>28</td><td>4.00</td><td>19.25</td></tr><tr><td>DenseNet</td><td>160</td><td>3.46</td><td>17.18</td></tr><tr><td>5 DISCUSSION</td><td/><td/><td/></tr><tr><td colspan="4">This manuscript presents a simple method for training deep feed-forward networks which greatly</td></tr><tr><td colspan="2">reduces the degradation problem.</td><td/><td/></tr></table>

Peer review (iclr2018):
 Update (original review below): The authors have addressed several of the reviewers* comments and improved the paper. The motivation has certainly been clarified, but in my opinion it is still hazy. The paper does use skip connections, but the difference is that they are phased out over training. So I think that the motivation behind introducing this specific difference should be clear. Is it to save the additional (small) overhead of using skip connections? Nevertheless, the additional experiments and clarifications are very welcome. For the newly added case of VAN(lambda=0), please note the strong similarity to https://arxiv.org/abs/1611.01260 (ICLR2017 reviews at https://openreview.net/forum?id=Sywh5KYex). In that report .alpha_l is a scalar instead of a vector. Although it is interesting, the above case case also calls into question the additional value brought by the use of constrained optimization, a main contribution of the paper. In light of the above, I have increased my score since I find this to be an interesting approach, but in my opinion the significance of the results as they stand is low. The paper demonstrates that it is possible to obtain very deep plain networks (without skip connections) with improved performance through the use of constrained optimization that gradually removes skip connections, but the value of this demonstration is unclear because a) consistent improvements over past work or the .lambda=0 case were not found, and b) The technique still relies on skip connections in a sense so it*s not clear that it suggests a truly different method of addressing the degradation problem. Original Review ============= Summary: The contribution of this paper is a method for training deep networks such that skip connections are present at initialization, but gradually removed during training, resulting in a final network without any skip connections. The paper first proposes an approach based on a formulation of deep networks with (non-parameterized, non-gated) skip connections with an equality constraint that effectively removes the skip connections when satisfied. It is proposed to optimize the formulation using the method of Lagrange multipliers. A toy model with a single unit is used to illustrate the basic ideas behind the method. Finally, experimental results for the task of image classification are reported using the MNIST, Fashion-MNIST, and CIFAR datasets. Quality and significance: The proposed methodology is simple and straightforward. The analysis with the toy network is interesting and helps illustrate the method. However, my main concerns with this paper are related to motivation and experiments. The motivation of the work is not clear at all. The stated goal is to address some of the issues related to the role of depth in deep networks, but I think it should be clarified which specific issues in particular are relevant to this method and how they are addressed. One could additionally consider that removing the skip connections at the end of training reduces the computational expense (slightly), but beyond that the expected utility of this investigation is very hazy from the description in the paper. For MNIST and MNIST-Fashion experiments, the motivation is mentioned to be similar to Srivastava et al. (2015), but in that study the corresponding experiment was designed to test if deeper networks could be optimized. Here, the generalization error is measured instead, which is heavily influenced by regularization. Moreover, only some architectures appear to employ batch normalization, which is a potent regularizer. The general difference between plain and non-plain networks is very likely due to optimization difficulties alone, and due to the above issues further comparisons can not be made from the results. For the CIFAR experiments, the experiment design is reasonable for a general comparison. Similar experimental setups have been used in previous papers to report that a proposed method can achieve good results, but there is no doubt that this does not make a rigorous comparison without employing expensive hyper-parameter searches. This is not the fault of the present paper but an unfortunate tradition in the field. Nevertheless, it is important to note that direct comparison should not be made among approaches with key differences. For the reported results, Fitnets and Highway Networks did not use Batch Normalization (which is a powerful regularizer) while VANs and Resnets do. Moreover, it is important to report the training performance of deeper VANs (which have a worse generalization error) to clarify if the VANs suffered difficulties in optimization or generalization. Clarity: The paper is generally well-written and easy to read. There are some clarity issues related to the use of the term "activation function" and a typo in an equation but the authors are already aware of these.