Title: AVOIDING DEGRADATION IN DEEP FEED-FORWARD NETWORKS BY PHASING OUT SKIP-CONNECTIONS Anonymous

Abstract: A widely observed phenomenon in deep learning is the degradation problem: increasing the depth of a network leads to a decrease in performance on both test and training data. Novel architectures such as ResNets and Highway networks have addressed this issue by introducing various flavors of skip-connections or gating mechanisms. However, the degradation problem persists in the context of plain feed-forward networks. In this work we propose a simple method to address this issue. The proposed method poses the learning of weights in deep networks as a constrained optimization problem where the presence of skip-connections is penalized by Lagrange multipliers. This allows for skip-connections to be introduced during the early stages of training and subsequently phased out in a principled manner. We demonstrate the benefits of such an approach with experiments on MNIST, fashion-MNIST, CIFAR-10 and CIFAR-100 where the proposed method is shown to greatly decrease the degradation effect and is often competitive with ResNets.

1. INTRODUCTION:

The representation view of deep learning suggests that neural networks learn an increasingly abstract representation of input data in a hierarchical fashion (Zeiler & Fergus, 2014; Goodfellow et al., 2016; Greff et al., 2016) . Such representations may then be exploited to perform various tasks such as image classification, machine translation and speech recognition.

A natural conclusion of the representation view is that deeper networks will learn more detailed and abstract representations as a result of their increased capacity. However, in the case of feed-forward networks it has been observed that performance deteriorates beyond a certain depth, even when the network is applied to training data. Recently, Residual Networks (ResNets; He et al. 2016a) and Highway Networks (Srivastava et al., 2015) have demonstrated that introducing various flavors of skip-connections or gating mechanisms makes it possible to train increasingly deep networks. However, the aforementioned degradation problem persists in the case of plain deep networks (i.e., networks without skip-connections of some form).

A widely held hypothesis explaining the success of ResNets is that the introduction of skipconnections serves to improve the conditioning of the optimization manifold as well as the statistical properties of gradients employed during training. Raiko et al. (2012) and Schraudolph (2012) show that the introduction of specially designed skip-connections serves to diagonalize the Fisher information matrix, thereby bringing standard gradient steps closer to the natural gradient. More recently, Balduzzi et al. (2017) demonstrated that the introduction of skip-connections helps retain the correlation structure across gradients. This is contrary to the gradients of deep feed-forward networks, which resemble white noise. More generally, the skip-connections are thought to reduce the effects of vanishing gradients by introducing a linear term (He et al., 2016b) .

The goal of this work is to address the degradation issue in plain feed-forward networks by leveraging some of the desirable optimization properties of ResNets. We approach the task of learning parameters for a deep network under the framework of constrained optimization. This strategy allows us to introduce skip-connections penalized by Lagrange multipliers into the architecture of our network. In our setting, skip-connections play an important role during the initial training of the network and are subsequently removed in a principled manner. Throughout a series of experiments we demonstrate that such an approach leads to improvements in generalization error when compared to architectures without skip-connections and is competitive with ResNets in some cases.

The contributions of this work are as follows:

• We propose alternative training strategy for plain feed-forward networks which reduces the degradation in performance as the depth of the network increases. The proposed method introduces skip-connections which are penalized by Lagrange multipliers. This allows for the presence of skip-connections to be iteratively phased out during training in a principled manner. The proposed method is thereby able to enjoy the optimization benefits associated with skip-connections during the early stages of training. • A number of benchmark datasets are used to demonstrate the empirical capabilities of the proposed method. In particular, the proposed method greatly reduces the degradation effect compared to plain networks and is on several occasions competitive with ResNets.

2. RELATED WORK:

The hierarchical nature of many feed-forward networks is loosely inspired by the structure of the visual cortex where neurons in early layers capture simple features (e.g., edges) which are subsequently aggregated in deeper layers (Hubel & Wiesel, 1962) . This interpretation of neural networks suggests that the depth of a network should be maximized, thereby allowing the network to learn more abstract (and hopefully useful) representations (Bengio et al., 2013) . However, a widely reported phenomenon is that deeper networks are more difficult to train. This is often termed the degradation effect in deep networks (Srivastava et al., 2015; He et al., 2016a) . This effect has been partially attributed to optimization challenges such as vanishing and shattered gradients (Hochreiter et al., 2001; Balduzzi et al., 2017) .

In the past these challenges have been partially addressed via the use of supervised and unsupervised pre-training (Bengio et al., 2009) and more recently through careful parameter initialization (Glorot & Bengio, 2010; He et al., 2015) and batch normalization (Ioffe & Szegedy, 2015) . In the past couple of years further improvements have been obtained via the introduction of skip-connections.

ResNets (He et al., 2016a; b) introduce residual blocks consisting of a residual function F together with a skip-connection. Formally, the residual block is defined as:

x l+1 = F l (x l , W l ) + W l x l (1)

where F l : R n → R n represents some combination of affine transformation, non-linearity and batch normalization parameterized by W l . The matrix W l parameterizes a linear projection to ensure the dimensions are aligned1 . More generally, ResNets are closely related to Highway Networks (Srivastava et al., 2015) where the output of each layer is defined as:

EQUATION

where • denotes element-wise multiplication. In Highway Networks the output of each layer is determined by a gating function

T (x l , H l ) = sigmoid (H l x l )

inspired from LSTMs. We note that both ResNets and Highway Networks were introduced with the explicit goal of training deeper networks. Inspired by the success of the these methods, many variations have been proposed. Huang et al. (2016a) propose DenseNet, where skip-connections are passed from all previous activations. Huang et al. (2016b) propose to shorten networks during training by randomly dropping entire layers, leading to better gradient flow and information propagation, while using the full network at test time.

Recently, the goal of learning deep networks without skip-connections has begun to receive more attention. Zagoruyko & Komodakis (2017) propose a novel re-parameterization of weights in feedforward networks which they call the Dirac parameterization. Instead of explicitly adding a skipconnection, they model the weights as a residual of the Dirac function, effectively moving the skipconnection inside the non-linearity. In related work, Balduzzi et al. (2017) propose to initialize weights in a CReLU activation function in order to preserve linearity during the initial phases of training. This is achieved by initializing the weights in a mirrored block structure. During training the weights are allowed to diverge, resulting in non-linear activations.

Finally, we note that while the aforementioned approaches have sought to train deeper networks via modifications to the network architecture (i.e., by adding skip-connections) success has also been obtained by modifying the non-linearities (Clevert et al., 2015; Klambauer et al., 2017) .

3. VARIABLE ACTIVATION NETWORKS:

The goal of this work is to train deep feed-forward networks without suffering from the degradation problem described in previous sections. To set notation, we denote x 0 as the input and x L as the output of a feed-forward network with L layers. Given training data {y, x 0 } it is possible to learn parameters {W l } L l=1 by locally minimizing some objective function

EQUATION

First-order methods are typically employed due to the complexity of the objective function in equation (3). However, directly minimizing the objective is not practical in the context of deep networks: beyond a certain depth performance quickly deteriorates on both test and training data. Such a phenomenon does not occur in the presence of skip-connections. Accordingly, we take inspiration from ResNets and propose to modify equation ( 1) in the following manner2 :

EQUATION

where α l ∈ [0, 1] n determines the weighting given to the skip-connection. More specifically, α l is a vector were the entry i dictates the presence and magnitude of a skip-connection for neuron i in layer l. Due to the variable nature of parameters α l in equation ( 4), we refer to networks employing such residual blocks as Variable Activation Networks (VAN).

The objective of the proposed method is to train a feed-forward network under the constraint that α l = 1 for all layers, l. When the constraint is satisfied all skip-connections are removed. The advantage of such a strategy is that we only require α l = 1 at the end of training. This allows us to initialize α l to some other value, thereby relaxing the optimization problem and obtaining the advantages associated with ResNets during the early stages of training. In particular, whenever α l = 1 information is allowed to flow through the skip-connections, alleviating issues associated with shattered and vanishing gradients.

As a result of the equality constraint on α l , the proposed activation function effectively does not introduce any additional parameters. All remaining weights can be trained by solving the following constrained optimization problem:

{ Ŵl } L l=1 = argmin C y, x L ; {W l , α l } L l=1

such that α l = 1 for l = 1, . . . , L.

(5)

The associated Lagrangian takes the following simple form (Boyd & Vandenberghe, 2004) :

EQUATION

where each λ l ∈ R n are the Lagrange multipliers associated with the constraints on α l . In practice, we iteratively update α l via stochastic gradients descent (SGD) steps of the form:

α l ← α l -η ∂C ∂α l + λ l (7)

where η is the step-size parameter for SGD. Throughout the experiments we will often take the non-linearity in F l to be ReLU. Although not strictly required, we clip the values α l to ensure they remain in the interval [0, 1] n .

From equation ( 6), we have that the gradients with respect to Lagrange multipliers are of the form:

EQUATION

We note that since we require α l ∈ [0, 1] n , the values of λ l are monotonically decreasing. As the value of Lagrange multiplier decreases, this in turn pushes α l towards 1 in equation ( 7). We set the step-size for the Lagrange multipliers, η , to be a fraction of η. The motivation behind such a choice is to allow the network to adjust as we enforce the constraint on α l .

4. EXPERIMENTS:

The purpose of the experiments presented in this section is to demonstrate that the proposed method serves to effectively alleviate the degradation problem in deep networks. We first demonstrate the capabilities of the proposed method using a simple, non-convolutional architecture on the MNIST and Fashion-MNIST datasets (Xiao et al., 2017) in Section 4.1. More extensive comparisons are then considered on the CIFAR datasets (Krizhevsky & Hinton, 2009) in Section 4.2.

4.1. MNIST AND FASHION-MNIST:

Networks of varying depths were trained on both MNIST and Fashion-MNIST datasets. Following Srivastava et al. (2015) the networks employed in this section were thin, with each layer containing 50 hidden units. In all networks the first layer was a fully connected plain layer followed by l layers or residual blocks (depending on the architecture) and a final softmax layer. The proposed method is benchmarked against several popular architectures such as ResNets and Highway Networks as well as the recently proposed DiracNets (Zagoruyko & Komodakis, 2017) . Plain networks without skipconnections are also considered. Finally, we also considered VAN network where the constraint α l = 1 was not enforced. This corresponds to the case where λ l = 0 for all l. This comparison is included in order to study the capacity and flexibility of VAN networks without the need to satisfy the constraint to remove skip-connections. For clarity, refer to such networks as VAN (λ = 0) networks.

For all architectures the ReLU activation function was employed together with batch-normalization.

In the case of ResNets and VAN, the residual function consisted of batch-normalization followed by ReLU and a linear projection.

The depth of the network varied from l = 1 to l = 30 hidden layers. All networks were trained using SGD with momentum. The learning rate is fixed at η = 0.001 and the momentum parameter at 0.9.

Training consisted of 50 epochs with a batch-size of 128. In the case of VAN networks the α l values were initialized to 0 for all layers. As such, during the initial stages of training VAN networks where equivalent to ResNets. The step-size parameter for Lagrange multipliers, η , was set to be one half of the SGD step-size, η. Finally, all Lagrange multipliers, λ l , are initialized to -1.

RESULTS:

The results are shown in Figure 1 where the test accuracy is shown as a function of the network depth for both the MNIST and Fashion-MNIST datasets. In both cases we see clear evidence of the degradation effect: the performance of plain networks deteriorates significantly once the network depth exceeds some critical value (approximately 10 layers). As would be expected, this is not the case for ResNets, Highway Networks and DiracNets as such architectures have been explicitly designed to avoid this behavior. We note that VAN networks do not suffer such a pronounced degradation as the depth increases. This provides evidence that the gradual removal of skip-connections via Lagrange multipliers leads to improved generalization performance compared to plain networks. Finally, we note that VAN networks obtain competitive results across all depths. Crucially, we note that VAN networks outperform plain networks across all depths, suggesting that the introduction of variable skip-connections may lead to convergence at local optima with better generalization performance. Finally, we note that VAN (λ = 0) networks, where no constraint is placed on skip-connections, obtain competitive results across all depths. We note that with the exception of plain networks, the performance of all remaining architectures is stable as the number of layers increases.

4.2. CIFAR:

As a more challenging benchmark we consider the CIFAR-10 and CIFAR-100 datasets. These consist of 60000 32×32 pixel color images with 10 and 100 classes respectively. The datasets are divided into 50000 training images and 10000 test images.

We follow He et al. (2016a) and train deep convolutional networks consisting of four blocks each consisting of n residual layers. The residual function is of the form conv-BN-ReLU-conv-BN-ReLU. This corresponds to the pre-activation function (He et al., 2016b) . The convolutional layers consist of 3 × 3 filters with downsampling at the beginning of blocks 2, 3 and 4. The network with a fully connected softmax layer, resulting in a depth of 8n + 2. The architecture is described in Table 1 .

Networks were trained using SGD with momentum over 165 epochs. The learning rate was set to η = 0.1 and divided by 10 at the 82nd and 125th epoch. The momentum parameter was set to 0.9. Networks were trained using mini-batches of size 128. Data augmentation followed Lee et al. (2015) : this involved random cropping and horizontal flips. Weights were initialized following He et al. (2015) . As in Section 4.1, we initialize α l = 0 for all layers. Furthermore, we set the step-size parameter for the Lagrange multipliers, η , to be one tenth of η and all Lagrange multipliers, λ l , are initialized to -1. On CIFAR-10 we ran experiments with n ∈ {1, 2, 3, 4, 5, 6, 8, 10} yielding networks with depths ranging from 10 to 82. For CIFAR-100 experiments were run with n ∈ {1, 2, 3, 4}. 

Results for experiments on CIFAR-10 are shown in Figure 2 . The left panel shows the mean test accuracy over five independent training sessions for ResNets, VAN, VAN (λ = 0) and plain networks. While plain networks provide competitive results for networks with fewer than 30 layers, their performance quickly deteriorates thereafter. We note that a similar phenomenon is observed in VAN networks but the effect is not as dramatic. In particular, the performance of VANs is similar to ResNets for networks with up to 40 layers. Beyond this depth, ResNets outperform VAN by an increasing margin. This holds true for both VAN and VAN (λ = 0) networks, however, the difference is reduced in magnitude in the case of VAN (λ = 0) networks. These results are in line with He et al. (2016b) , who argue that scalar modulated skip-connections (as is the case in VANs where the scalar is 1 -α l ) will either vanish or explode in very deep networks whenever the scalar is not the identity.

The right panel of Figure 2 shows the training and test error for a 26 layer network. We note that throughout all iterations, both the test and train accuracy of the VAN network dominates that of the plain network. The thick gold line indicates the mean residuals of the α l parameters across all layers. This is defined as

1 L L l=1 (1 -α l ) 2

and is a measure of the extent to which skip-connections are present in the network. Recall that if all α l values are set to one then all skip-connections are removed (see equation ( 4)). From Figure 2 , it follows that skip-connections are fully removed from the VAN network at approximately the 120 th iteration. More detailed traces of Lagrange multipliers and α l are provided in Appendix B.

A comparison of the performance of VAN networks in provided in Table 2 . We note that while VAN networks do not outperform ResNets, they do outperform other alternatives such as Highway networks and FitNets (Romero et al., 2014) when networks of similar depths considered. However, it is important to note that both Highway networks and FitNets did not employ batch-normalization, which is a strong regularizer. In the case of both VAN and VAN (λ = 0) networks, the best performance is obtained with networks of 26 layers while ResNets continue to improve their performance as depth increases. Finally, current state-of-the-art performance, obtained by Wide ResNets (Zagoruyko & Komodakis, 2016) and DenseNet Huang et al. (2016a) , are also provided in Table 2 Figure 3 provides results on the CIFAR-100 dataset. This dataset is considerably more challenging as it consists of a larger number of classes as well as fewer examples per class. As in the case of CIFAR-10, we observe a fall in the performance of both VAN and plain networks beyond a certain depth; in this case approximately 20 layers for plain networks and 30 layers for VANs. Despite this drop in performance, Table 2 indicates that the performance of VAN networks with both 18 and 26 layers are competitive with many alternatives proposed in the literature. Furthermore, we note that the performance of VAN (λ = 0) networks is competitive with ResNets in the context of the CIFAR-100 dataset. 

A VAN RE-FORMULATION:

The original formulation for the VAN residual block was as follows:

EQUATION

We thank an anonymous reviewer for suggesting that such a formulation may be detrimental to the performance of very deep VAN networks. The reason for this is that scaling constant within each block is always less than one, implying that the contributions of lower layers vanish exponentially as the depth increases. This argument is also provided in He et al. (2016b) who perform similar experiments with ResNets.

In order to validate this hypothesis, we compare the performance of VAN networks employing the residual block described in equation ( 4) and the residual block described in equation ( 9). The results, shown in Figure 4 , provide evidence in favor of the proposed hypothesis. While both formulations for VAN networks obtain similar performances for shallow networks, as the depth of the network increases there is a more pronounced drop in the performance of VAN networks which employ residual blocks described in equation ( 9).

In a further experiment, we also studied the performance of ResNets with the following residual block:

EQUATION

)

The results in Figure 4 demonstrate that ResNets which employ the residual blocks defined in equation (10) show a clear deterioration in performance as the depth of the network increases. Such a degradation in performance is not present when standard ResNets are employed. 

B LAGRANGE MULTIPLIER TRACES:

In this section we provide addition figures demonstrating the evolution of Lagrange multipliers, λ l throughout training. We note that the updates to Lagrange multipliers are directly modulated by the current value of each α l (see equation ( 8)). As such, we also visualize the mean residuals of the α l parameters across all layers. This is defined as 1 L L l=1 (1 -α l ) 2 and is a measure of the extent to which skip-connections are present in the network. Once all skip-connections have been removed, this residual will be zero and the values of Lagrange multipliers will no longer change.

This is precisely what we find in Figure 5 . The left panel plots the mean value of Lagrange multipliers across all layers, while the right panel shows the mean residual of α l . We observe that for networks of different depths, once the constraint to remove skip-connections is satisfied, the value of Lagrange multipliers remains constant. This occurs at different times; sooner for more shallow networks whilst later on for deeper networks. 

:

Unless stated otherwise we will assume F retains the dimension of x l and set W l to the identity.

Our original formulation consisted of a convex sum over F and x l . However, we found this approach to be the most successful empirically. Appendix A provides a comparison to original formulations as well as some theoretical justifications for this improvement in performance.

Figures and Tables:

Figure1: Results on MNIST (left) and fashion-MNIST (right) for various different architectures as the depth of the network varies from 1 to 30. Mean average test accuracy over 10 independent training sessions is shown. We note that with the exception of plain networks, the performance of all remaining architectures is stable as the number of layers increases.

Figure 2: Left: Results on CIFAR-10 dataset are shown as the depth of networks increase. We note that the performance of both VAN and plain networks deteriorates as the depth increases, but the effect is far less pronounced for VAN networks. Right: Training and test error curves are shown for networks with 26 layers. We also plot the mean α residuals: 1 L

Figure 3: Left: Results on CIFAR-100 dataset are shown as the depth increases from 10 to 34 layers. We note that the performance of both VAN and plain networks deteriorates as the depth increases, but the effect is far less pronounced for plain networks. Right: Training and test error curves are shown for VAN and plain networks with 18 layers. The mean α residuals, 1 L

Figure 4: Results are shownVAN and  ResNet networks with various different residual blocks. We note that the use of residual blocks with non-identity scaling coefficients leads to a larger drop in performance as the network depth increases. This drop is attributed to vanishing contributions from lower blocks (as all scalings are less than one).

Figure 5: Lagrange multipliers, λ l are shown on the left panel for networks of varying depth. After a certain number of iterations, the values of the Lagrange multiplier plateau as the constraint to remove skip-connections is satisfied. This results in no updates to the values of the Lagrange multipliers (see equation (8)). The right panel shows the mean α l residual. This residual directly modulates the magnitude of changes in Lagrange multipliers.

Architecture of varying depth employed on CIFAR-10 and CIFAR-100 datasets. Downsampling occurs at the beginning of block 2, 3 and 4 with a stride of two.

Table content: <table><tr><td colspan="2">Layer name Output size</td><td>Convolution</td></tr><tr><td>conv1</td><td>32 × 32</td><td>3 × 3, 64</td></tr><tr><td>Block 1</td><td>32 × 32</td><td>3×3,64 3×3,64 × n</td></tr><tr><td>Block 2</td><td>16 × 16</td><td>3×3,128 3×3,128 × n</td></tr><tr><td>Block 3</td><td>8 × 8</td><td>3×3,256 3×3,256 × n</td></tr><tr><td>Block 4</td><td>4 × 4</td><td>3×3,512 3×3,512 × n</td></tr></table>

Comparison of VAN networks to results of other convolutional networks on CIFAR-10 and CIFAR-100. For VAN networks we report the best value as well as the mean and standard deviation over five independent training runs (in brackets). Results are ordered by performance on CIFAR-10. We add a * to denote results which did not employ batch-normalization. In the past, the degradation issue has been successfully addressed via the introduction of skip-connections. As such, the goal of this work is to propose a new training regime which retains the optimization benefits associated with ResNets while ultimately phasing out skip-connections. This is achieved by posing network training as a constrained optimization problem where skip-connections are introduced during the early stages of training and subsequently phased out in a principled manner using Lagrange multipliers.

Table content: <table><tr><td>Architecture</td><td colspan="3"># Layers CIFAR-10 (test error %) CIFAR-100 (test error %)</td></tr><tr><td>Highway Network  *</td><td>32</td><td>8.80</td><td>-</td></tr><tr><td>FitNet  *</td><td>19</td><td>8.39</td><td>35.04</td></tr><tr><td>Highway Network  *</td><td>19</td><td>7.54</td><td>32.39</td></tr><tr><td>DiracNet (width-1)</td><td>34</td><td>7.10</td><td>-</td></tr><tr><td>ELU  *</td><td>18</td><td>6.55</td><td>24.28</td></tr><tr><td>VAN (λ = 0)</td><td>26</td><td>6.29 (6.40 ± 0.16)</td><td>27.04 (27.42 ± 0.26)</td></tr><tr><td>VAN (λ = 0)</td><td>34</td><td>6.28 (6.45 ± 0.14)</td><td>26.46 (26.81 ± 0.31)</td></tr><tr><td>VAN</td><td>18</td><td>6.23 (6.49 ± 0.16)</td><td>28.20 (28.42 ± 0.36)</td></tr><tr><td>VAN</td><td>26</td><td>6.08 (6.35 ± 0.21)</td><td>27.70 (28.01 ± 0.39)</td></tr><tr><td>DiracNet (width-2)</td><td>34</td><td>5.60</td><td>26.72</td></tr><tr><td>ResNet</td><td>164</td><td>5.46</td><td>24.33</td></tr><tr><td>Wide ResNet (width-10)</td><td>28</td><td>4.00</td><td>19.25</td></tr><tr><td>DenseNet</td><td>160</td><td>3.46</td><td>17.18</td></tr><tr><td>5 DISCUSSION</td><td/><td/><td/></tr><tr><td colspan="4">This manuscript presents a simple method for training deep feed-forward networks which greatly</td></tr><tr><td colspan="2">reduces the degradation problem.</td><td/><td/></tr></table>

Peer review (iclr2018):
 EDIT: The rating has been changed. See thread below for explanation / further comments. ORIGINAL REVIEW: In this paper, the authors present a new training strategy, VAN, for training very deep feed-forward networks without skip connections (henceforth called VDFFNWSC) by introducing skip connections early in training and then gradually removing them. I think the fact that the authors demonstrate the viability of training VDFFNWSCs that could have, in principle, arbitrary nonlinearities and normalization layers, is somewhat valuable and as such I would generally be inclined towards acceptance, even though the potential impact of this paper is limited because the training strategy proposed is (by deep learning standards) relatively complicated, requires tuning two additional hyperparameters in the initial value of .lambda as well as the step size for updating .lambda, and seems to have no significant advantage over just using skip connections throughout training. So my rating based on the message of the paper would be 6/10. However, there appear to be a range of issues. As long as those issues remain unresolved, my rating is at is but if those issues were resolved it could go up to a 6. +++ Section 3.1 problems +++ - I think the toy example presented in section 3.1 is more confusing than it is helpful because the skip connection you introduce in the toy example is different from the skip connection you introduce in VANs. In the toy example, you add (1 - .alpha)wx whereas in the VANs you add (1 - .alpha)x. Therefore, the type of vanishing gradient that is observed when tanh saturates, which you combat in the toy model, is not actually combated at all in the VAN model. While it is true that skip connections combat vanishing gradients in certain situations, your example does not capture how this is achieved in VANs. - The toy example seems to be an example where Lagrangian relaxation fails, not where it succeeds. Looking at figure 1, it appears that you start out with some alpha < 1 but then immediately alpha converges to 1, i.e. the skip connection is eliminated early in training, because wx is further away from y than tanh(wx). Most of the training takes place without the skip connection. In fact, after 10^4 iterations, training with and without skip connection seem to achieve the same error. It appears that introducing the skip connection was next to useless and the model failed to recognize the usefulness of the skip connection early in training. - Regarding the optimization algorithm involving .alpha^* at the end of section 3: It looks to me like a hacky, unprincipled method with no guarantees that just happened to work in the particular example you studied. You motivate the choice of .alpha^* by wanting to maximize the reduction in the local linear approximation to .mathcal{C} induced by the update on w. However, this reduction grows to infinity the larger the update is. Does that mean that larger updates are always better? Clearly not. If we wanted to reduce the size of the objective according to the local linear approximation, why wouldn*t we choose infinitely large step sizes? Hence, the motivation for the algorithm you present is invalid. Here is an example where this algorithm fails: consider the point (x,y,w,.alpha,.lambda) = (100, .sigma(100), 1.0001, 1, 1). Here, w has almost converged to its optimum w* = 1. Correspondingly, the derivative of C is a small negative value. However, .alpha* is actually 0, and this choice would catapult w far away from w*. If I haven*t made a mistake in my criticisms above, I strongly suggest removing section 3.1 entirely or replacing it with a completely new example that does not suffer from the above issues. +++ ResNet scaling +++ There is a crucial difference between VANs and ResNets. In the VAN initial state (alpha = 0.5), both the residual path and the skip path are multiplied by 0.5 whereas for ResNet, neither is multiplied by 0.5. Because of this, the experimental results between the two architectures are incomparable. In a question I posed earlier, you claimed that this scaling makes no difference when batch normalization is used. I disagree. Let*s look at an example. Consider ResNet first. It can be written as x + r_1 + r_2 + .. + r_B, where r_b is the value computed by residual block b. Now let*s assume we insert a scaling constant after each residual block, say c = 0.5. Then the result is c^{B}x + c^{B-1}r_1 + c^{B-2}r_2 + .. + r_B. Therefore, contributions of lower blocks vanish exponentially. This effect is not combated by batch normalization. So the learning dynamics for VAN and ResNet are very different because of this scaling. Therefore, there is an open question: are the differences in results between VAN and ResNet in your experiments caused by the removal of skip connections during training or by this scaling? Without this information, the experiments have limited value. In fact, I suspect that the vanishing of the contribution of lower blocks bears more responsibility for the declining performance of VAN at higher depths than the removal of skip connections. If my assessment of the situation is correct, I would like to ask you to repeat your experiments with the following two settings: - ResNet where after each block you multiply the result of the addition by 0.5, i.e. x_{l+1} = 0.5.mathcal{F}(x_l) + 0.5x_l - VAN with the following altered equation: x_{l+1} = .mathcal{F}(x_l) + (1-.alpha)x_l, i.e. please remove the alpha in front of .mathcal{F}. Also, initialize .alpha to zero. This ensures that VAN starts out as a regular ResNet. +++ writing issues +++ Title: - "VARIABLE ACTIVATION NETWORKS: A SIMPLE METHOD TO TRAIN DEEP FEED-FORWARD NETWORKS WITHOUT SKIP-CONNECTIONS" This title can be read in two different ways. (A) [Train] [deep feed-forward networks] [without skip-connections] and (B) [Train] [deep feed-forward networks without skip connections]. In (A), the `without skip-connections* modifies the `train* and suggests that training took place without skip connections. In (B), the `without skip-connections* modifies `deep feed-forward networks* and suggests that the network trained has no skip connections. You must mean (B), because (A) is false. Since it is not clear from reading the title whether (A) or (B) is true, please reword it. Abstract: - "Part of the success of ResNets has been attributed to improvements in the conditioning of the optimization problem (e.g., avoiding vanishing and shattered gradients). In this work we propose a simple method to extend these benefits to the context of deep networks without skip-connections." Again, this is ambiguous. To me, this sentence implies that you extend the benefit of avoiding vanishing and exploding gradients to fully-connected networks without skip connections. However, nowhere in your paper do you show that trained VANs have less exploding / vanishing gradients than fully-connected networks trained the old-fashioned way. Again, please reword or include evidence. - "where the proposed method is shown to outperform many architectures without skip-connections" Again, this sentence makes no sense to me. It seems to imply that VAN has skip connections. But in the abstract you defined VAN as an architecture without skip connections. Please make this more clear. Introduction: - "Indeed, Zagoruyko & Komodakis (2016) demonstrate that it is better to increase the width of ResNets than the depth, suggesting that perhaps only a few layers are learning useful representations." Just because increasing width may be better than increasing depth does not mean that deep layers don*t learn useful representations. In fact, the claim that deep layers don*t learn useful representations is directly contradicted by the paper. section 3.1: - replace "to to" by "to" in the second line section 4: - "This may be a result of the ensemble nature of ResNets (Veit et al., 2016), which does not play a significant role until the depth of the network increases." The ensemble nature of ResNet is a drawback, not an advantage, because it causes a lack of high-order co-adaptataion of layers. Therefore, it cannot contribute positively to the performance or ResNet. As mentioned in earlier comments, please reword / clarify your use of "activation function". It is generally used a synonym for "nonlinearity", so please use it in this way. Change your claim that VAN is equivalent to PReLU. Please include your description of how your method can be extended to networks which do allow for skip connections. +++ Hyperparameters +++ Since the initial values of .lambda and .eta* are new hyperparameters, include the values you chose for them, explain how you arrived at those values and plot the curve of how .lambda evolves for at least some of the experiments.