Title: Learning Identity Mappings with Residual Gates
Authors: Pedro H. P. Savarese
Leonardo O. Mazza
Daniel R. Figueiredo
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Optimization

Abstract: We propose a layer augmentation technique that adds shortcut connections with a linear gating mechanism, and can be applied to almost any network model. By using a scalar parameter to control each gate, we provide a way to learn identity mappings by optimizing only one parameter. We build upon the motivation behind Highway Neural Networks and Residual Networks, where a layer is reformulated in order to make learning identity mappings less problematic to the optimizer. The augmentation introduces only one extra parameter per layer, and provides easier optimization by making degeneration into identity mappings simpler. Experimental results show that augmenting layers provides better optimization, increased performance, and more layer independence. We evaluate our method on MNIST using fully-connected networks, showing empirical indications that our augmentation facilitates the optimization of deep models, and that it provides high tolerance to full layer removal: the model retains over 90% of its performance even after half of its layers have been randomly removed. In our experiments, augmented plain networks -- which can be interpreted as simplified Highway Neural Networks -- outperform ResNets, raising new questions on how shortcut connections should be designed. We also evaluate our model on CIFAR-10 and CIFAR-100 using augmented Wide ResNets, achieving 3.65% and 18.27% test error, respectively.

TL;DR: This paper proposes adding simple gates to layers to make learning identity mappings trivial. It also introduces Gated Plain Networks and Gated Residual Networks.

Review discussion:
Can you please elaborate on how SDI is assumed to be a credible measure?
Assuming that random pruning in case of Augmentation drops layers irrespective of corresponding k value. It is surprising to see that performance does not degrade even when layers with high k might have been dropped. Can you please explain?
How can the authors claim that the approach can beat the SOTA with more resources when they did not run any such experiments?
Thanks for the feedback.
The SDI is a lower bound of the squared path length between two points in the parameter surface. It does not capture how the convergence truly occurs -- that is, it does not consider how the parameters are updated in each iteration until the target point is achieved. That being said, it only captures a notion of distance between two parameter states. I have plans to add a more formal explanation (possibly in the appendix due to the paper already being longer than suggested).
Regarding the random pruning, the k's have extremely low variance after the network is trained.
Therefore there is little difference between a "high" and a "low" k.
However, the setting is different with Residual Networks: the k values of blocks where there is no dimension increase (that is, no convolution in the shortcut connection) are extremely uniform among themselves, as are the k values of blocks with dimension increase.
However, the two groups have a very different mean (around 0.8 for the first group vs 1.5 in the second). I will add this finding in the next version of the paper.
I agree that there is little experimental evidence to support such claim, considering the vast amount of techniques in the literature that work well for small experiments but fail on larger ones. The intuition for such claim was that, since the k values are initialized as 1.0 (a detail which I forgot to add to the paper, as I was informed), then doing no optimization on such parameters is equivalent to a vanilla residual block, which is what achieves the SOTA. Given the non-convex nature of the optimization, however, there are no strong guarantees that the optimizer might end up optimizing the parameters in a "bad way", performance-wise. We have managed to run more complex networks (~37M parameters) and achieve around 3.71% error in preliminary tests -- such results will be added to the next version of the paper.
Since the model is directly related to both highway and residual networks, why is it not compared to both?
Can you clarify what is the central claim, and how do the experiments support it? Is it better optimization, better regularization, or both?
Is it reliable to compare different models using the same learning rate?
After all, the optimal learning rate depends on the model.
Thanks a lot for the feedback!
I'll add Highway Networks to the CIFAR-10 results table. It hadn't crossed my mind since ResNets became the standard due to their increased performance, but I agree it is only fair to compare to both models.
I'm also making the claims clearer in the second version, which will give a stronger focus on the advantages regarding optimization. I still plan to run experiments to check how the gates are connected to regularization, especially giving a proper try on L2 penalization on the gates' weights.
Regarding the different learning rates, I have re-run all Wide ResNet experiments with the original implementation details -- 0.1/0.02/0.004/0.0008 learning rates, 0.0005 weight decay, and so on. The results are way superior (3.71% on CIFAR-10 on preliminary tests without the recent additions proposed by Sergey on the second version of the Wide ResNet paper).
- "assume that the squared parameter-wise distance between .. surrogate to the paths length". Can you elaborate more on this? Why should this be the case?
- Why do you substitute mean and variance with initialization mean variance in derivation in Eq 15 and 16?
-  Regarding "A comparison of the Total Squared Distance to Identity…": where is the comparison?
- Why there is no Highway Neural Networks result in Table 5?
- In Table 5, why does  He et al. (2015b)  give 6.61 which does not match with 6.43 in that paper?
- Regarding "Our results don’t surpass the state-of-the-art, which was expected considering the hardware limitations.
However, taking into account the improvement observed when augmenting a smaller Wide ResNet, we believe that the technique proposed can be used to surpass the state-of-the-art", you may use that result for sanity check during the empirical work but how can you claim that you will get any improvement over baselines in final comparison?
- Baselines have Imagenet results. In order to obtain satisfactory comparison, you would need that too. Are there any results on Imagenet comparison?
Thanks a lot for your feedback, I greatly appreciate it.
- "assume that the squared parameter-wise distance between .. surrogate to the paths length". Can you elaborate more on this? Why should this be the case?
Actually, the SDI is just a lower bound to the path's length, since it ignores how the parameters are updated. It is equivalent to assuming that the optimizer always makes "perfect" updates (that is, in the correct direction), thus following the shortest path between the current parameter state and the "optimal" one. I will work on making this explanation better for the next version.
- Why do you substitute mean and variance with initialization mean variance in derivation in Eq 15 and 16?
Those are the mean and variances of each parameter in the considered tensor/matrix. My intent is to analyze how easily a network can learn identity mappings, therefore I consider the initial state of the network, which is when parameters are initialized.
For example, if I were to initialize all layers' parameters with values which correspond to identity mappings -- for example, the identity matrix or the dirac tensor for plain fully-connected or conv networks -- then the network would require zero updates to learn the identity, which would immediately be observed with a SDI of zero. This can be observed with recurrent nets' behaviors when initializing the transition matrix as the identity.
- Why there is no Highway Neural Networks result in Table 5?
Thanks for pointing that out, I will add them to the next version.
- In Table 5, why does  He et al. (2015b)  give 6.61 which does not match with 6.43 in that paper?
In that paper, 6.43 is reported as the best result out of 5 runs, while 6.61 is the mean. I do not believe it is fair to use best-out-of-5 results for comparison, especially since, as long as I'm aware, it would be the only result in that table to be best-out-of-x instead of mean/median/single-run. Please correct me if I'm wrong.
- Regarding "Our results don’t surpass the state-of-the-art, which was expected considering the hardware limitations.
However, taking into account the improvement observed when augmenting a smaller Wide ResNet, we believe that the technique proposed can be used to surpass the state-of-the-art", you may use that result for sanity check during the empirical work but how can you claim that you will get any improvement over baselines in final comparison?
I agree that there is little experimental evidence to support such claim, considering the vast amount of techniques in the literature that work well for small experiments but fail on larger ones. The intuition for such claim was that, since the k values are initialized as 1.0 (a detail which I forgot to add to the paper, as I was informed), then doing no optimization on such parameters is equivalent to a vanilla residual block, which is what achieves the SOTA. Given the non-convex nature of the optimization, however, there are no strong guarantees that the optimizer might end up optimizing the parameters in a "bad way", performance-wise. We have managed to run more complex networks (~37M parameters) and achieve around 3.71% error in preliminary tests -- such results will be added to the next version of the paper.
- Baselines have Imagenet results. In order to obtain satisfactory comparison, you would need that too. Are there any results on Imagenet comparison?
Unfortunately, not so far. I plan to add results on CIFAR-100 to the next version, but considering hardware limitations I believe it is rather unlikely that I will have time to run the model on ImageNet.
The paper presents a layer architecture where a single parameter is used to  gate the output response of layer to amplify or suppress it. It is shown that such an architecture can ease optimization of a deep network as it is easy to learn identity mappings in layers helping in better gradient propagation to lower layers (better supervision).
Using an introduced SDI metric it shown that gated residual networks can most easily learn identity mappings compared to other architectures.
Although good theoretical reasoning is presented the observed experimental evidence of learned k values does not seem to strongly support the theory given that learned  k values are mostly very small and not varying much across layers.
Also, experimental validation of the approach is not quite strong in terms of reported performances and number of large scale experiments.
Thanks for your comments!
We have added a proper evaluation on the learned k values, and achieved interesting conclusions on the pattern of the final k parameters.
For example, in Residual Networks the lowest k values are of blocks that increase the dimension of the feature maps, suggesting that these units might be less relevant to a network, and conversely just maintaining the shortcut connection (in this case, with a 1x1 convolution) might be a good alternative.
We focused our experimental validation on understanding the effects of the k parameter. Achieving such satisfying results (3.65%/18.27%) on CIFAR was not expected at all, and we believe that achieving these results without requiring to tune hyperparameters or to compare several potential architectures was a positive factor of the proposed technique.
In the recent revisions of our work, we have performed the following main changes:
- Run complex models on CIFAR-10 and on CIFAR-100, achieving 3.65% and 18.27% test error, respectively. We have then removed our belief that the technique could achieve SOTA results, since now we have empirical indications of it.
- Added evaluation of the final k parameters for fully-connected and convolutional networks, showing interesting patterns for both architectures.
- Added layer pruning experiment to Wide GResNet.
- Made the intuition behind the distance to identity clearer.
We thank all the received feedback.
This paper proposes to learn a single scalar gating parameter instead of a full gating tensor in highway networks. The claim is that such gating is easier to learn and allows a network to flexibly utilize computation.
The basic idea of the paper is simple and is clearly presented. It is a natural simplification of highway networks to allow easily "shutting off" layers while keeping number of additional parameters low.
However, in this regard the paper leaves out a few key points.
Firstly, it does not mention that the gates in highway networks are data-dependent which is potentially more powerful than learning a fixed gate for all units and independent of data.
Secondly, it does not do a fair comparison with highway networks to show that this simpler formulation is indeed easier to learn.
Did the authors try their original design of u = g(k)f(x) + (1 - g(k))x where f(x) is a plain layer instead of a residual layer? Based on the arguments made in the paper, this should work fine. Why wasn't it tested? If it doesn't work, are the arguments incorrect or incomplete?
For the MNIST experiments, since the hyperparameters are fixed, the plots are misleading if any dependence on hyperparameters exists for the different models. This experiment appears to be based on Srivastava et al (2015). If it is indeed designed to test optimization at aggressive depths, then apart from doing a hyperparameter search, the authors should not use regularization such as dropout or batch norm, which do not appear in the theoretical arguments for the architecture.
For CIFAR experiments, the obtained improvements compared to the baseline (wide resnets) are very small and therefore it is important to report the standard deviations (or all results) in both cases. It's not clear that the differences are significant.
Some questions regarding g(): Was g() always ReLU? Doesn't this have potential problems with g(k) becoming 0 and never recovering? Does this also mean that for the wide resnet in Fig 7, most residual blocks are zeroed out since k < 0?
Thanks for the feedback.
- it does not mention that the gates in highway networks are data-dependent which is potentially more powerful than learning a fixed gate for all units and independent of data
We do mention this difference, however it goes against our main goals when designing the gates, more specifically:
+ Having gates which are data-dependent makes the number of parameters grow.
In case we used a data-dependent scalar to control the gates, we would need a Nx1 matrix (considering a fully-connected network) to achieve this. This would imply in N times more parameters than our approach.
+ This would make the analysis extremely more complex. It would not be possible to understand as easily the exact effects of the extra dimension that each k parameter adds to the cost surface. I believe it would be actually impossible with the current theoretical tools we have, since the problem would be as hard as understanding how a new layer affects the cost function. This would result in understanding the whole cost function of a Deep Network, since it it is decomposable in a limited number of layers. Having a single data-independent scalar and a linear gate offers the huge advantage of understanding very easily its impact on the model.
+ One of our proposals is to use the method as a way for layers to fully deactivate. This is only feasible if the parameters that deactivate the layer are data-independent, because if they were data-dependent we would have the constraint: \forall x, u(x) = x.
That is, the gates would have to deactivate for every possible input. Since the number of parameters of a data-dependent layer is lower bounded by N, we would require at least N parameters to be set to zero. This would offer little advantage compared to how Residual layers behave as identity mappings for specific parameter sets. In our approach, layers truly become identity mappings, instead of just having the ability to behave as such mapping for specific data points.
- Did the authors try their original design of u = g(k)f(x) + (1 - g(k))x where f(x) is a plain layer instead of a residual layer? Based on the arguments made in the paper, this should work fine. Why wasn't it tested? If it doesn't work, are the arguments incorrect or incomplete?
Such layers have already been partially explored in "Identity Mappings in Deep Residual Networks", and perform similarly to Residual Networks. Unfortunately following the ~8 page limit was not possible with all discussions and experiments we had in mind.
EDIT: We have realized that adding such results without having over 10 pages was doable.
Therefore we will add them to the conference version in case of acceptance.
- For the MNIST experiments, since the hyperparameters are fixed, the plots are misleading if any dependence on hyperparameters exists for the different models. This experiment appears to be based on Srivastava et al (2015). If it is indeed designed to test optimization at aggressive depths, then apart from doing a hyperparameter search, the authors should not use regularization such as dropout or batch norm, which do not appear in the theoretical arguments for the architecture.
We have not used Dropout for such experiments. We believe that the internal covariate shift is an orthogonal problem to the one of learning identity mappings -- even though they both negatively impact the optimization of deep networks --, therefore we decided on using BN for all our tests.
- Some questions regarding g(): Was g() always ReLU? Doesn't this have potential problems with g(k) becoming 0 and never recovering? Does this also mean that for the wide resnet in Fig 7, most residual blocks are zeroed out since k < 0?
During prototyping we have tested sigmoid, linear and ReLU. Sigmoid had three major disadvantages: first, if just performed worse, second, we lost the linear interpretation of the extra dimension (where the slices of the new dimension are linear combinations of the original model and the identity mapping, roughly speaking), lastly, g(k) could not be >1, which means the gate could not properly suppress the shortcut connection (as the network does in the last layer for both experiments we ran).
Linear and ReLU perform the same, except for when layers deactivate. We chose to use ReLU exactly for its deactivation: we propose our method as one whose layers can be fully pruned during training, hence making future epochs faster. Using the ReLU as g(k) is equivalent to using a linear gate where layers are removed from the model once k < 0.
Of course, nothing stops a linear function to be used as g.
As for Fig 7, thanks a lot for pointing the mistake, the values in the graph are actually the real values minus 1. We are not currently sure what caused this, but we will be updating the paper with the correct graph soon.
Just a quick update to your feedback before we upload a new version:
- Did the authors try their original design of u = g(k)f(x) + (1 - g(k))x where f(x) is a plain layer instead of a residual layer? Based on the arguments made in the paper, this should work fine. Why wasn't it tested? If it doesn't work, are the arguments incorrect or incomplete?
We have gathered results for the original design in order to compare it with the other 3 in the first experiment (Plain, Residual and Gated Residual). Interestingly, it seems that this design is even better when dealing with increased depth (the 100-layered setting, for example). We would like to thank you for the suggestion, since such valuable results could have been gone unnoticed otherwise.
- For the MNIST experiments, since the hyperparameters are fixed, the plots are misleading if any dependence on hyperparameters exists for the different models. This experiment appears to be based on Srivastava et al (2015). If it is indeed designed to test optimization at aggressive depths, then apart from doing a hyperparameter search, the authors should not use regularization such as dropout or batch norm, which do not appear in the theoretical arguments for the architecture.
We are currently running a new batch of experiments with 200+ layered networks. We believe that this new experiment might be more informative on how each model behaves with aggressive depths, even with batch normalization.
This paper proposes a network called Gated Residual Networks layer design that adds gating to shortcut connections with a scalar to regulate the gate. The authors claim that this approach will improve the training Residual Networks.
It seems the authors could get competitive performance on CIFAR-10 to state of art models with only Wide Res Nets. Wide Gated ResNet requires much more parameters than DenseNet (and other Res Net variants) for obtaining a little improvement over Dense Net.  More importantly, the authors state that they obtained the best results on CIFAR-10 and CIFAR-100 but the updated version of DenseNet (Huang et al. (2016b)) has new results for a version called DenseNet-BC which outperforms all of the results that authors reported (3.46 for CIFAR-10 and 17.18 for CIFAR-100 with 25.6M parameters, DenseNet-BC still outperforms with 15.3M parameters which is much less that 36.5M). The Res Net variants papers with state of art results report result for Image Net.
Therefore the empirical results need also the Image Net to demonstrate that improvement claimed is achieved.
The proposed trick adopts Highway Neural Networks and Residual Networks with an intuitive motivation. It is not sufficiently novel and the empirical results do not prove sufficient effectiveness of this incremental approach.
Hi and thanks for the feedback!
- Wide Gated ResNet requires much more parameters than DenseNet (and other Res Net variants) for obtaining a little improvement over Dense Net
Gated WideResNets are by no means the focus of our paper, but an example of how our layer augmentation works. We are not proposing a new type of ResNet, but a general tool to aid optimization of general networks. It suffices to see that our technique is also applicable to DenseNets, or nearly any other kind of network (including ResNeXts). We augmented Wide ResNets because they were the current SOTA during our experiments, and we wanted to provide experimental indications that our technique could be used to improve state-of-the-art models. We feel that our technique is more closely related to general tools such as Dropout and Batch Norm: it is a general technique that can be added to nearly any model and provides advantages such as easier optimization.
Our claim is not that it improves ResNets, but that it can be used to improve any layer design. Due to the linear nature of the gates, an optimization on the new dimension is purely linear, and adds no further non-linear complexity to the model. It is also easily reversible -- we actually initialize the gates in such a way that they do not impact the model --  just like BN once we consider its post-norm affine transformation.
Moreover, the comparison between the Gated WideResNet and DenseNets had the goal of showing that a simple change with negligible impact on training time and number of parameters could make a model surpass the previous state-of-the-art. The main comparison, however, is the one between Gated and Non-Gated models, where the improvement due to the linear gates could be put to trial.
My evaluation did not change after the rebuttal period because the authors replies and revisions do not address satisfactorily the drawbacks stated in review.
In the last revision of our work, we have performed the following main changes:
- Added results with u = g(k)f(x) + (1 - g(k))x (Gated Plain Networks), along with some intuitions of why it outperformed non-augmented Residual Networks. This result suggests that an extensive study on different gating mechanisms for Highway Neural Networks can be extremely fruitful, once the original design is equivalent to a Highway Net with scalar gates. This also goes against the suggestions in the literature not to add gates to shortcut connections in order to keep an uncorrupted gradient flow through the network.
- Added Gated Plain Networks to the table with mean k values, along with an explanation for the significant difference when compared to mean k's of Gated ResNets.
- Added 'Understanding deep learning requires rethinking generalization' to bibliography.
- Rephrased a few parts when augmented models are compared to Highway Nets, showing more clearly the differences between the two designs.
Also added a brief discussion regarding the impact for Highway Nets of the new results (Gated Plain Nets).
We are currently gathering results to introduce the following changes to the next revision:
- Add results for more aggressive depths (200+ layers), in order to better compare different models.
- Add results for Gated Plain Net on CIFAR.
We thank all the received feedback.
In the last revision of our work, we have performed the following main changes:
- Added 6 extra experiments comparing Wide ResNet and their augmented counterparts (Table 4). With this we hope to provide stronger indications of the generality and advantages of our technique.
We are currently running the same 6 models on CIFAR-100 to add them to the table. Are have also run Gated PlainNets (u = g(k)f(x) + (1 - g(k))x) on CIFAR-10, as suggested by reviewer 3, and will add results after running on CIFAR-100 as well.
We thank all the received feedback.
In the last revision of our work, we have performed the following main changes:
- Added 4 extra experiments comparing Wide ResNet and their augmented counterparts (Table 5) on CIFAR-100. With this we hope to provide stronger indications of the generality and advantages of our technique.
We are currently running Gated PlainNets on CIFAR-100.
We thank all the received feedback.
We have also added a section (3.3) that connects our technique to the Unrolled Iterative Estimation interpretation of Highway and ResNets. We further elaborate on how the learned k values are indicators of the abstraction jumps in representations defined in Greff et al, and analyze the k values for both experiments (MNIST and CIFAR) under this perspective.

Meta-review:
role: PC
data: comment: Although this was a borderline paper, the reviewers ultimately concluded that, given how easy it would be for a practitioner to independently devise the methodological trick of the paper, the paper did not demonstrate that the idea was sufficiently useful to merit acceptance.

Decision: Reject