Title: The loss surface of residual networks: Ensembles and the role of batch normalization
Authors: Etai Littwin
Lior Wolf
Venue: ICLR 2017
Keywords: Deep learning
Theory

Abstract: Deep Residual Networks present a premium in performance in comparison to conventional
networks of the same depth and are trainable at extreme depths. It has
recently been shown that Residual Networks behave like ensembles of relatively
shallow networks. We show that these ensemble are dynamic: while initially
the virtual ensemble is mostly at depths lower than half the network’s depth, as
training progresses, it becomes deeper and deeper. The main mechanism that controls
the dynamic ensemble behavior is the scaling introduced, e.g., by the Batch
Normalization technique. We explain this behavior and demonstrate the driving
force behind it. As a main tool in our analysis, we employ generalized spin glass
models, which we also use in order to study the number of critical points in the
optimization of Residual Networks.

TL;DR: Residual nets are dynamic ensembles

Review discussion:
-  The issue regarding unrealistic assumptions of spin glass analysis for landscape of neural networks was posed as an open problem in COLT 2015.[1] Can you discuss the effect of this problem for your analysis?
- Regarding assumption that minimal of (12) should hold, is this assumption is realistic or not?
-Choromanska(AISTATS 2015) supports the claims based on theoretical results with many empirical results however you do not provide such analysis. Can you support your claims with reasonable empirical results?
- What is the empirical setup for Figure 1?
-Fractal net paper on arxiv claim that residuals are incidental. Can you elaborate on that based on your framework? What about densely connected conv networks(huang,2016 )?
[1] A. Choromanska, Y. LeCun, G. Ben Arous, Open Problem: The landscape of the loss surfaces of multilayer networks, in the Conference on Learning Theory (COLT), Open Problems, 2015
Thank you for your comments!
1.In this work we use spin glass analysis in order to understand the dynamic behavior ResNets display during training and to study their loss surface.
In particular, we use at one point or another the assumptions of redundancy in network parameters, near uniform distribution of network weights, independence between the inputs and the paths and independence between the different copies of the input as described in Choromanska(AISTATS 2015). The last two assumptions, i.e., the two independence assumptions, are deemed in COLT 2015 [1] as unrealistic, while the remaining are considered plausible.
Our analysis of critical points in ensembles (Sec. 5) requires all of the above assumptions.
However, theorems 1 and 2, as well as the analysis of the dynamic behaviour of residual nets (Sec. 4), which constitute our main contributions, do not assume the last assumption, i.e., the independence between the different copies of the input.
2. The assumption that the expectation in Lemma 2 is the specified lower bound is realistic when applying the assumptions of redundancy and uniformity in network parameters.
Indeed, given that Lambda=n it holds exactly, since all weight configurations of a particular length in Eq.9 will appear the same number of times.
When Lambda is not a n, uniformity dictates that each configuration of weights would appear approximately equally regardless of the inputs, and the expectation values would be very close to the lower bound.
3. Our work is theoretical and is centered around our analysis. We did present experimental evidence that the novel phenomenon we point to, i.e., the dynamic behavior of ResNets during training, is indeed real.
4. Fig. 1(d) and 1(e)  report the experimental results of a straightforward setting. The task is to classify a mixture of 10 multivariate Gaussians in 50D. The input is therefore of size 50. The loss employed is the cross entropy loss of ten classes. The network has 10 blocks, each containing 20 hidden neurons, a batch normalization layer, and a skip connection. Training was performed on 10,000 samples, using SGD with minibatches of 50 samples. Note that Fig. 1(a-c), 1(f) are calculated and not estimated empirically.
5. Our results are aligned with some of the results shown in the fractal net paper. In Sec. 4.3 there, the authors note empirically that the deepest column trains last. This is reminiscent of our claim that the deeper networks of the ensemble become more prominent as training progresses. The authors of FractalNets hypothesize  that this is a result of the shallower columns being stabilized at a certain point of the training process. In our work, we explicitly define the exact driving force that comes into play.
In addition, our work offers an insight into the mechanics of the recently proposed densely connected networks. Following the analysis we provide at Sec. 3, the additional shortcut paths decrease the initial capacity of the network by offering many more short paths from input to output, thereby contributing to the ease of optimization when training starts. The driving force mechanism described in Sec. 4.2 will then cause the effective capacity of the network to increase.
Note that the analysis presented in Sec. 3 can be generalized to architectures with arbitrary skip connections, including dense nets. This is done directly by including all of the induced sub networks in Eq. 8. The reformulation of Eq. 9 would still holds, given that \Psi_r is modified accordingly.
Hi, sorry for the delay!
A question on equation 8 and gamma_r = \binom{p-1}{r-1}n^r
It seems like the assumption is that there are p-1 skip connections skipping a single layer each.
So if we are counting paths from input to output, surely there are no paths with length < p/2? (So the length sum should be from r=p/2 in equation 8?)
In our setup we have skip connections skipping every layer but the first, and so the shortest path is when the input goes through the first layer and skips all the rest (path of length 1).
We looked back at the comment of AnonReviewer3. We are the first to analyze the dynamic behavior of ResNets during training and point to a novel phenomenon, which we analyze theoretically. We reveal both the underlying reason for this phenomenon and its profound effect on the training process.
However, we might not have demonstrated convincingly enough the existence of this phenomenon on conventional datasets. We therefore ran a few last minute experiments to show exactly this.
We took the ResNet code of https://github.com/facebook/fb.resnet.torch and trained on either CIFAR-10 or CIFAR-100 for networks of depth 32. As noted in our paper, the dynamic behavior can be present in the Batch Normalization gamma coefficient or in the weight matrices themselves
“the mechanism for [the] dynamic property of residual networks can also be observed without the use of batch normalization, as a steady increase in the L2 norm of the weights, as shown in Fig. 1(e).”
In the experiments we just ran, it seems that until the learning rate is reduced, the dynamic behavior is manifested in the Batch Normalization coefficient gamma and then it moves to the convolution layers themselves. For the following plots, we therefore absorb the gamma into the convolutional layer using https://github.com/e-lab/torch-toolbox/tree/master/BN-absorber.
There are two types of plots: one depicts the magnitude of the various convolutional layers for multiple epochs (similar in type to Fig. 1(d) in the paper); the second type plots the sum of these norms over all convolutional layers as a function of epoch (similar to Fig. 1(e)).
Cifar10 norm per layer http://imgur.com/X306qCt each graph is a different epoch, waving is due to the interleaving archtecture of the convolutional layers.
Cifar 10 mean norm per epoch http://imgur.com/PFFGlnv
Cifar 100 norm per layer http://imgur.com/lz462NJ each graph is a different epoch
Cifar 100 mean norm per epoch http://imgur.com/DBY0z68
As can be seen, the dynamic phenomenon we describe is very prominent in the public ResNet implementation when applied to conventional datasets: the dominance of paths with fewer skip connections increases over time. We find it very interesting that once the learning rate is reduced in epoch 81 the phenomenon we describe speeds up.
Below are the gamma values when not absorbed. The graphs report the norms of the gamma coefficient vectors.
Cifar 10 gamma norm per layer http://imgur.com/p1bb9BM each graph is a different epoch (since there is no monotonic increase between the epochs in this graph, it is harder to interpret)
Cifar 10 mean gamma norm per epoch http://imgur.com/LVB35UU
Cifar 100 gamma norm per layer http://imgur.com/tUWbRya each graph is a different epoch
Cifar 100 mean gamma norm per epoch http://imgur.com/qO9DFlj
As future work, we would like to better understand why the gammas start to decrease once the learning rate is reduced. As shown above, taking the magnitude of the convolutions into account the dynamic phenomenon we found actually becomes more prominent. The change of location from the gamma coefficient of the Batch Normalization layers to the convolutions themselves is a fascinating phenomenon by itself, which might indicate that Batch Normalization is no longer required at this point.
Indeed, Batch Normalization enables larger training rates and this shift happens exactly when the training rate is reduced. A complete analysis is out of the scope of the current paper.
[Please note that due to notation overloading, the Batch Normalization coefficient gamma is denoted in our paper as lambda, and following Choromanska et al (2015), gamma denotes the number of paths.]
Summary:
In this paper, the authors study ResNets through a theoretical formulation of a spin glass model. The conclusions are that ResNets behave as an ensemble of shallow networks at the start of training (by examining the magnitude of the weights for paths of a specific length) but this changes through training, through which the scaling parameter C (from assumption A4) increases, causing it to behave as an ensemble of deeper and deeper networks.
Clarity:
This paper was somewhat difficult to follow, being heavy in notation, with perhaps some notation overloading. A summary of some of the proofs in the main text might have been helpful.
Specific Comments:
- In the proof of Lemma 2, I'm not sure where the sequence beta comes from (I don't see how it follows from 11?)
- The ResNet structure used in the paper is somewhat different from normal with multiple layers being skipped? (Can the same analysis be used if only one layer is skipped? It seems like the skipping mostly affects the number of paths there are of a certain length?)
- The new experiments supporting the scale increase in practice are interesting! I'm not sure about Theorems 3, 4 necessarily proving this link theoretically however, particularly given the simplifying assumption at the start of Section 4.2?
Specifically for the proof of Lemma 2, \beta is the multiplicity of each input value from the p-dimensional input vector x in the expression \xi. We will change it to another symbol and clarify this point. The next version to be released very soon, will include these additional clarifications and we hope that the issue will thus be resolved.
In our framework, as you pointed out, the skipping connectiones affect the number of paths of a specific length. This number is manifested in the parameter gamma_r.  The architecture used in our analysis includes a skip connection per layer and gives rise to paths of every length between 1 the r. This simplifies the notation and was therefore beneficial in our analysis. Exactly the same analysis as in Sec. 3 can be performed with 2 layers or more per skipped block. The difference would change gamma_r while keeping everything else the same.
The results of Sec. 5 employ the general spherical spin glass model as presented and analyzed in  [Auffinger 2013]. This model includes every order of spin interaction between 1 and infinity, and thus fits the one skip per layer model, for large p. The extrapolation of Theorem 5 to the case of skipping a multi-layer block is technically possible, however it requires additional approximations.
Sec. 4, Theorems 3 and 4, as pointed out, studies each skip connection by itself. It can be applied to a block of arbitrary size. Since it holds for each block individually, it holds also for a network of multiple blocks.
We just revised our manuscript based on the very meaningful discussions we had with the reviewing team. For convenience, the changes are marked in red.
In addition to incorporating everything we have promised so far and adding discussion based on the reviewers’ suggestions, we were able to relax the assumptions used for the results of Sec.  4. This change addresses a concern raised by AnonReviewer3 and was done by using the assumption-free expression for the output of the network in Eq. 9 in order to compute the loss in Lemma 3.
Overall, believe that what can be considered as the main results of our manuscript, i.e., the dynamic behavior during the training of ResNets, is now essentially assumption free and is also well supported experimentally. We thank the reviewers for their crucial role in improving the manuscript.
This paper shows how spin glass techniques that were introduced in Choromanska et al. to analyze surface loss of deep neural networks can be applied to deep residual networks. This is an interesting contribution but it seems to me that the results are too similar to the ones in Choromanska et al. and thus the novelty is seriously limited. Main theoretical techniques described in the paper were already introduced and main theoretical results mentioned there were in fact already proved. The authors also did not get rid of lots of assumptions from Choromanska et al. (path-independence, assumptions about weights distributions, etc.).
We completely disagree with the novelty assertions made by the AnonReviewer1. Most of our main results are very distant from the results of Choromanska et al. and are entirely novel both technically and conceptually.
For example, the detailed study of the driving force in ResNets.
We also disagree with the assertion that “The authors also did not get rid of lots of assumptions from Choromanska et al.”, since, as the previous discussion on openreview.net reveals, we make selective use of these assumptions and Theorems 3 and 4 are virtually assumption free. Please refer to the discussion section of the revised manuscript, uploaded a few days ago.
We are at a great disadvantage in this discussion since the assertions of AnonReviewer1 are made in general terms, which makes it difficult for us to understand (1) which results are deemed as incremental and (2) in what way these are judged as incremental, (3) which theoretical techniques are claimed to have already been done and (4) where these were done.
We would be grateful if the reviewer could point to specific theorems that might be problematic and would agree to weigh their importance in the context of the entire body of presented results. In order to promote such a discussion, we address below each theorem and some lemmas along the dimensions of closest analogy in the literature and technical novelty.  We believe that the facts clearly support the novelty of our work both conceptually and technically. If AnonReviewer1 disagrees, we respectfully ask to consider addressing specific results.
Theorem 1
What it shows: An expression for the effective depth of a deep ResNet, in the limit of infinite depth.
Closest analogy in the literature: It has been argued that ResNets exhibit the properties of ensembles, however we are not aware of a similar mathematical analysis.
Technical novelty: This theorem presents a novel mathematical treatment of the effective depth of ResNets.
Theorem 2
What it shows: Deep ResNets behave as ensembles concentrated around a narrow band near the maximum.
Closest analogy in the literature: We are not aware of a similar claim.
Technical novelty: The proof of this claim is quite involved and is not trivial.
Theorem 3
What it shows: The driving force behind the capacity increase of residual nets during training, when batch normalization is applied.
Closest analogy in the literature: As far as we are aware, there is no other work which points out to this type dynamic behaviour in ResNets or elsewhere.
Technical novelty: We are not aware of any similar analysis. Nothing in the proof follows existing results.
Theorem 4
What it shows: The driving force behind the capacity increase of residual nets during training, without batch normalization.
Closest analogy in the literature: As far as we are aware, no similar analysis exists.
Technical novelty: We are not aware of any similar analysis.
Theorem 5
What it shows: The loss surface of ensembles, in the context of general spin glass models. Here we use results of spin glass theory to demonstrate the landscape of the loss in ensembles, compared with single models, in terms of the number of critical points.
Closest analogy in the literature: This comparison presents a novel viewpoint on ensembles, as far as we are aware.
Technical novelty: In this theorem we use the results of spin glass theory. Most of the technical heavy lifting for this specific theorem was done in Auffinger 2013
Lemma 1+2
What is shown: The similarity between the loss of ResNets, and the hamiltonian of a general spin glass model.
Closest analogy in the literature: Although a similar analysis was performed of traditional networks, this analogy presented additional technical difficulties, and is novel in its claim.
Technical novelty: Some technical aspects of this analogy are novel and not presented in the original work of Choromanska et al. especially Lemma 2.
Lemma 4
What it shows: The effective depth of ResNets can be controlled through weight scaling.
Closest analogy in the literature: We are not aware of a similar claim.
Technical novelty: We are not aware of any similar analysis.
The path-independence assumption is not realistic. Do the authors have any ideas how these assumptions can be relaxed ?
We acknowledge that the assumption that paths are independent of the input is an unrealistic one.
However, the assumption already appears as useful in promoting deep learning research in the literature and was used by Choromanska et al throughout their analysis.
Note that we use this assumption selectively. Theorems 3 and 4, which can be considered as our main contributions, do not employ any unrealistic assumptions.
This paper extend the Spin Glass analysis of Choromanska et al. (2015a) to Res Nets which yield the novel dynamic ensemble results for Res Nets and the connection to Batch Normalization and the analysis of their loss surface of Res Nets.
The paper is well-written with many insightful explanation of results.
Although the technical contributions extend the Spin Glass model analysis of the ones by Choromanska et al. (2015a), the updated version could eliminate one of the unrealistic assumptions and the analysis further provides novel dynamic ensemble results and the connection to Batch Normalization that gives more insightful results about the structure of Res Nets.
It is essential to show this dynamic behaviour in a regime without batch normalization to untangle the normalization effect on ensemble feature.
Hence authors claim that steady increase in the L_2 norm of the weights will maintain the this feature but setting for Figure 1 is restrictive to empirically support the claim.
At least results on CIFAR 10 without batch normalization for showing effect of L_2 norm increase and results that support claims about Theorem 4 would strengthen the paper.
This work provides an initial rigorous framework to analyze better the inherent structure of the current state of art Res Net architectures and its variants which can stimulate potentially more significant results towards careful understanding of current state of art models (Rather than always to attempting to improve the performance of Res Nets by applying intuitive incremental heuristics, it is important to progress on some solid understanding too).
We thank AnonReviewer3 for the supportive review. The reviewer asked that we provide CIFAR-10 results for ResNets without Batch Normalization. These results are provided below. The experiments are almost identical to the previous experiments with a few exceptions: (1) All Batch Normalization layers were removed; (2) Since it is harder to train without Batch Normalization, we focus on ResNets of depth 20; and (3) We employ an initial learning rate that is ten times smaller, otherwise NaN occurs after a few epochs.
CIFAR-10 Norm of the convolutional layers’ weights per layer for multiple epochs http://imgur.com/JdRAM7j
CIFAR-10 Mean norm of weights per epoch http://imgur.com/46ngxb6
CIFAR-100 Norm of weights per layer for multiple epochs http://imgur.com/ANKeRz0
CIFAR-100 Mean norm of weights per epoch http://imgur.com/MzHOFgE
Note that there is a steady increase in the weight norm until epoch 81 when the learning rate is reduced, at which point the norms stabilize. This could stem from the networks being slower to train without Batch Normalization and epoch 81 being too early for reducing the learning rate.
Authors' responses did not make me change my mind. I still think it is a clear rejection paper.
We respectfully expected the reviewer to explain and support the broad assertions that were made.
Instead, the comment above only reflects opinions ("my mind"; "I still think").
1. The reviewer did not provide the details that are required in order to hold the discussion on a factual level.
For example, we asked to know which theorems "were already introduced" and where the "main theoretical results mentioned there were in fact already proved".
2. It is hard to tell from the one line comment which parts of our very detailed and factual response were unconvincing to the reviewer.
3. We find the title of the reviewer's comment to be overly aggressive and in sheer contrast to the amount of factual support provided in the review.
If the title sounds to be aggressive, I'd like to sincerely apologize for it. That was not my intent.  I fixed it. That however does not change my score. I gave detailed comments regarding the paper in previous passes and the responses of the authors did not convince me.

Meta-review:
role: PC
data: comment: The paper presents an analysis of residual networks and argues that the residual networks behave as ensembles of shallow networks, whose depths are dynamic. The authors argue that their model provides a concrete explanation to the effectiveness of resnets. 
 
 However, I have to agree with reviewer 1 that the assumption of path independence is deeply flawed. In my opinion, it was also flawed in the original paper. Using that as a justification to continue this line of research is not the right approach. We cannot construct a single practical scenario where path independence may be expected to hold. So we should not be encouraging papers to continue this line of flawed reasoning.
 
 I thus cannot recommend acceptance of this paper.

Decision: Reject