Title: Topology and Geometry of Half-Rectified Network Optimization
Authors: C. Daniel Freeman
Joan Bruna
Venue: ICLR 2017
Keywords: Theory
Deep learning

Abstract: The loss surface of deep neural networks has recently attracted interest 
in the optimization and machine learning communities as a prime example of 
high-dimensional non-convex problem. Some insights were recently gained using spin glass 
models and mean-field approximations, but at the expense of strongly simplifying the nonlinear nature of the model.

In this work, we do not make any such approximation and study conditions 
on the data distribution and model architecture that prevent the existence 
of bad local minima. Our theoretical work quantifies and formalizes two 
important folklore facts: (i) the landscape of deep linear networks has a radically different topology 
from that of deep half-rectified ones, and (ii) that the energy landscape 
in the non-linear case is fundamentally controlled by the interplay between the smoothness of the data distribution and model over-parametrization. Our main theoretical contribution is to prove that half-rectified single layer networks are asymptotically connected, and we provide explicit bounds that reveal the aforementioned interplay.

The conditioning of gradient descent is the next challenge we address. 
We study this question through the geometry of the level sets, and we introduce
an algorithm to efficiently estimate the regularity of such sets on large-scale networks. 
Our empirical results show that these level sets remain connected throughout 
all the learning phase, suggesting a near convex behavior, but they become 
exponentially more curvy as the energy level decays, in accordance to what is observed in practice with 
very low curvature attractors.

TL;DR: We provide theoretical, algorithmical and experimental results concerning the optimization landscape of deep neural networks

Review discussion:
At some point, would one expect overparameterization to lead to overfitting?  Can anything be said about whether the amount of overparameterization needed to reduce problems from local minima would still allow one to avoid overfitting?
Absolutely, at some point, overparameterization would lead to overfitting.
While saying anything precisely about when this would occur in general is difficult, empirically, we find that very little overparameterization is needed to connect equivalent models with high probability.
For example, the specially constructed counterexample in Appendix E has local minima with 12 neurons, but the addition of a single extra hidden neuron (i.e., a 2-4-2 instead of 2-3-2 architecture) makes it "easy" to connect two equivalent models.  That we were also able to "easily" connect many different models in wildly different architectures with high test accuracy is also suggestive of connectivity appearing long before overparameterization.
However, we can't rule out the possibility of a pathological construction whereby overfitting plagues training immediately once the model architecture is made large enough to remove local minima.
I am a bit confused with the presentation in section 2.3.2. The beginning of the section uses matrices W1 and W2 (e.g. Eq (6)) as the weights. This continues to be the case in the following paragraph W=W1 and beta=W2.
However, in the third paragraph, the authors switch to vectors w1 and w2. How these vectors relate to the original weight marices W1 and W2?
We apologize for the confusion in notation.  The lowercase $w$ are rows of the weight matrices---i.e., rows of W^1 in equation (6).
So w_1 and w_2 are two generic rows of not necessarily the same weight matrix.  More explicitly, this is used in the proof of Theorem 2.4 to bound the error along the interpolated path between two sets of parameters row by row in that weight matrix (see for example the short discussion right below equation (20).
This work contributes to understanding the landscape of deep networks in terms of its topology and geometry. The paper analyzes the former theoretically, and studies the latter empirically.
Although the provided contributions are very specific (ReLU nets with single hidden layer, and a heuristic to calculate the normalized geodesic), the results are original and of interest.
Thus, they could potentially be used as stepping stones for deeper developments in this area.
Pros:
1. Providing new theory about existence of "poor" local minima for ReLU networks with a hidden unit that relies on input distribution properties as well as the size of the hidden layer.
2. Coming up with a heuristic algorithm to compute the normalized geodesic between two solution points. The latter reflects how curved the path between the two is.
Cons:
The results are very specific in both topology and geometry analysis.
1. The analysis is performed only over a "single" hidden layer ReLU network. Given the importance of depth in deep architectures, this result cannot really explain the kinds of architectures we are interested in practically.
2. The normalized geodesic criterion is somewhat limited in representing how easy it is to connect two equally good points.
For example, there might exist a straight line between the two (which is considered as easy by the geodesic criterion), but this line might be going through a very narrow valley, challenging gradient based optimization algorithms (and thus extremely difficult to navigate in practice).
In addition, the proposed algorithm for computing the normalized geodesic is a greedy heuristic, which as far as I can tell, makes it difficult to know how we can trust in the estimated geodesics obtained by this algorithm.
With all cons said, I stress that I understand both problems tackled in the paper are challenging, and thus I find the contributions valuable and interesting.
We appreciate the extensive feedback from the reviewer!  We have a couple comments on the reviewer's feedback.
"The results are very specific in both topology and geometry analysis.
1. The analysis is performed only over a "single" hidden layer ReLU network. Given the importance of depth in deep architectures, this result cannot really explain the kinds of architectures we are interested in practically."
Indeed, one long term goal would be an extension of our results to deeper architectures.  We find the numerical experiments encouraging in that connectivity appears to be easy to achieve in deep convnets, which seems to suggest that a more general strategy exists for connecting models to one another (the paths used in our proof construction are quite specific to the architecture, as the reviewer points out, and were chosen so as to be analytically tractable).
The reviewer might also be interested in our our response to Reviewer 1 discussing the generalization error results of Zhang et al.
"2. The normalized geodesic criterion is somewhat limited in representing how easy it is to connect two equally good points.
For example, there might exist a straight line between the two (which is considered as easy by the geodesic criterion), but this line might be going through a very narrow valley, challenging gradient based optimization algorithms (and thus extremely difficult to navigate in practice)."
We definitely agree that there could be easy paths missed by our algorithm because they might be in narrow, hard-to-find valleys.  And indeed, in practice, the paths that are generated by our algorithm are sampled from those regions of weight space that are tractable to explore via gradient descent.
While it would be great to be able to find the truly minimal geodesic path connecting two randomly trained networks, we suspect that this may become intractable once networks are trained sufficiently close to their global minimum.  Philosophically, it might even be undesirable to identify two networks as "connected" if it takes an intractable-to-find path in weight space to connect them.
That is, there are two separate questions that are independently interesting: 1. Can networks be connected to one another in principle? and 2. Can networks be connected to one another in practice?   We hope that our work can help illuminate both of these questions.
"In addition, the proposed algorithm for computing the normalized geodesic is a greedy heuristic, which as far as I can tell, makes it difficult to know how we can trust in the estimated geodesics obtained by this algorithm."
We agree that the greedy heuristic makes it difficult to fully trust the measured normalized geodesics.  We would like to emphasize, though, that the greedy algorithm, in those cases where it succeeds in finding a connecting path, does provide an upper bound to the true normalized geodesic length.
We've considered several alternative algorithms, and hope to pursue the alternative algorithm discussed in Appendix A in the future.
Thank you for the comments. I stress that the cons I listed is not to criticize the work, but rather to point out how far we are from an ideal result (which goes beyond a single conference paper). I do understand the challenges and I find the methods and findings of this paper interesting.
This paper studies the energy landscape of the loss function in neural networks.  It is generally clearly written and nicely provides intuitions for the results.  One main contribution is to show that the level sets of the loss becomes connected as the network is increasingly overparameterized.  It also quantifies, in a way, the degree of disconnectedness possible in terms of the increase in loss that one must allow to find a connected path.  It would seem that this might have some implications for the likelihood of escaping local minima with stochastic gradient descent.  The paper also presents a simple algorithm for finding geodesic paths between two networks such that the loss is decreasing along the path.  Using this they show that the loss seems to become more nonconvex when the loss is smaller.  This is also quite interesting.
The work does have some significant limitations, which is not surprising given the difficulty of fully analyzing the network loss function.
However, the authors are quite clear about these limitations, which especially include not yet analyzing deep networks and analyzing only the oracle loss, and not the empirical loss.  I would have also appreciated a little more practical discussion of the bound in Theorem 2.4.  It is hard to tell whether this bound is tight enough to be practically relevant.
We appreciate the feedback from the reviewer!  We have a couple of comments below:
"The work does have some significant limitations, which is not surprising given the difficulty of fully analyzing the network loss function.
However, the authors are quite clear about these limitations, which especially include not yet analyzing deep networks and analyzing only the oracle loss, and not the empirical loss.  I would have also appreciated a little more practical discussion of the bound in Theorem 2.4.  It is hard to tell whether this bound is tight enough to be practically relevant."
Indeed, we are also quite interested in further studying the tradeoff between model overparameterization and ease-of-training.  Saying too much more than this is difficult because the bound used in Theorem 2.4 is strongly problem dependent.  Using our normalized geodesic measure, it should be straightforward to probe this tradeoff on individual problems by simply varying network size.  And, encouragingly, it seems that the amount of overparameterization necessary to provide connections, in practice, is small.
If we had to speculate, the ease with which we can connect networks might be because state-of-the-art networks that are used on modern datasets are actually already quite over-parameterized.  See for example Zhang et al.'s "Understanding deep learning requires rethinking generalization", where the authors demonstrate that neural networks can easily completely memorize entire training sets.
Thus, modern networks might be considerably larger than is necessary to be connected.
This is an incremental result (several related results that the authors of the paper mentioned here were already published). The authors claim that they can get rid of the technical assumptions from the previous papers but the results they propose are significantly weaker and also quite technical. The main theoretical result - Theorem 2.4 is not convincing at all.
Furthermore, the paper is badly written. No theoretical intuition is given, the experimental section is weak and in some places the formatting is wrong.
We appreciate the review, and regret that the reviewer found the work unconvincing.  We have several clarifying questions for the reviewer so as to improve the quality of our manuscript:
"This is an incremental result (several related results that the authors of the paper mentioned here were already published)."
Which results in particular is the reviewer referring to?  The connectivity of linear networks is, of course, not new, as we make clear in the manuscript, but we are not aware of any proof of linear network connectivity that leverages the topology of level sets of the loss function as the primary proof technique.  If we have overlooked some of the literature in our discussion in the Introduction, we would of course be more than happy to add the relevant citations to our manuscript.
Our proof of the connectivity of linear networks is meant--in part--to help motivate the primary innovation of the paper: that thinking about how easy or difficult it is to traverse the level sets of the loss function can provide intuition about the topology of the loss function as well as the prevalence of local minima.
Thus, while the result of the linear case proof is not novel, we find that our particular proof is still useful for intuition-building.
"The authors claim that they can get rid of the technical assumptions from the previous papers but the results they propose are significantly weaker and also quite technical."
Could the reviewer be more specific about how our results are "weaker"?  We admit that the degree to which our results are "technical" may be a matter of philosophy, but did the reviewer have a particular alternative proof in mind which uses fewer technical assumptions?  We would be happy to soften our language, as we may not be aware of the "state of the art" so to speak.
"The main theoretical result - Theorem 2.4 is not convincing at all."
Again, we regret that the reviewer found the result unconvincing.  Could the reviewer clarify what they mean by this?  We will expand the discussion around Theorem 2.4 to make the practical relevance of the bound slightly more clear in the next iteration of the manuscript.
"Furthermore, the paper is badly written. No theoretical intuition is given, the experimental section is weak and in some places the formatting is wrong."
The weakness of the experimental section notwithstanding, the primary purpose of the section was to be exploratory.  We aimed to provide evidence that a wide variety of models that are significantly more complicated than single ReLU networks *also* appear to have the same sort of connectedness leveraged by our proof technique.  It is also meant to provide intuition for the proof technique--that these paths through the space of weights which keep test loss low are actually tractable to calculate, and not abstract mathematical objects.
That said, could the reviewer be more specific about what they find "weak"?  We would be more than happy to expand parts of the experimental section, especially if the intent was not clear.
We are aware of the slight page-break in eq. (10), and the broken links in Appendix E.  If the reviewer spotted any more formatting errors, we will gladly fix them!
First of all, this paper is very badly written. Let me comment more on that here.  The experimental section in the main body of the paper contains only Figure 1 but the authors refer to other experimental results from the Appendix. The Appendix should serve as a place for the proofs and additional experiments, but the most important experimental findings should be in the main body of the paper. Results from Fig.1 definitely are not of this characteristic.
Furthermore, it is not clear which datasets the experimental findings refer to.
Besides, the results on MNIST should not be really taken into account. This is a toy set where the authors should only test some initial conjectures. Deriving any substantial conclusions based on the MNIST experiment is very risky.
Bad formatting in many places, for instance Theorem 2.4, but there are many other places, where there are issues.
No intuition regarding core theoretical results is given and the results themselves are very technical. The authors make very rough comparison with previous work, several of the techniques they use in the paper were already used before in that context (see for instance the proof of Theorem 2.4)
In terms of the content, this is a very incremental work. Several conditions from the other papers on the topic can be relaxed with the use of such tools as Janson's inequality (the path-independence issue). Theoretical results up to section 2.3.3 are already well-known (see papers on the surface loss from NIPS'16 workshops). I have also problems with what the authors claim is the main result (Theorem 2.3.3). I dont think that the formula for epsilon in the statement of the theorem is correct, at least I dont see at all how it follows from the given proof (which I guess is in fact just a sketch of the proof even though the authors put it in the Appendix, where they could put the entire proof).
Furthermore, I think that the presented sketch of the proof is not correct, but it is hard to say since all the details are hidden so the reviewer needs to guess.
Many results in the paper are given in a "hand-waving" form which is not appropriate for such conferences as ICLR (educated guesses that form certain parameters to be small enough the Taylor expansion gives some desired results and many more place where theoretical claims are simply not clear at all).
I consider myself to have pretty strong theoretical background in that field and the way the theoretical results are presented do not convince me that these theoretical results are correct. And this is, if I understand right, what the authors claim is the main contribution of the paper.
I will hesitate to improve my score, since I have strong feeling that the proofs are not complete and furthermore the presentation of these results does not help me to figure out myself how to fix the parts that are not clear.
To sum it up, I think that it is an interesting contribution but way below ICLR standards. I would suggest the authors to work on improving the presentation and please work on better notation too. That might be a very interesting journal publication, conditioned that the authors spend more time of presenting their theoretical claims.
We appreciate the rapid feedback!  We have several followup comments and clarifying questions for the reviewer:
"First of all, this paper is very badly written. Let me comment more on that here.  The experimental section in the main body of the paper contains only Figure 1 but the authors refer to other experimental results from the Appendix. The Appendix should serve as a place for the proofs and additional experiments, but the most important experimental findings should be in the main body of the paper. Results from Fig.1 definitely are not of this characteristic.
Furthermore, it is not clear which datasets the experimental findings refer to."
We absolutely agree that the important experiments should be within the experimental section of the paper, and it was our intent that the experiments currently within our experimental section *are* the most important experiments.  The primary experimental result of the paper is that, for many different problem architectures, it is easy to connect equally powerful models to one another, which is what the 5 different experiments in Fig. 1 establish.
The only experiment in the Appendix is the counterexample task which demonstrates the algorithm failing to find a connecting path for a pathological construction.  This experiment is not central to our result, but we felt it was an important diagnostic case to include to show what happens when the algorithm fails.
We apologize for a lack of clarity about the datasets used in the paper.  In lieu of ballooning the paper's length with experiment details, we provided Github links to Tensorflow implementations of all of the experiments that are run within the caption of Fig. 1.  We admit, however, that this could make it difficult to tease out exactly how the experiments are constructed on a first read--is there a particular dataset that the reviewer would like more detail on?
"Besides, the results on MNIST should not be really taken into account. This is a toy set where the authors should only test some initial conjectures. Deriving any substantial conclusions based on the MNIST experiment is very risky."
We agree that the MNIST dataset is a toy set, and we also agree that it should only be used for testing initial conjectures.
We are puzzled by the reviewer's comments, because this was our aim for the MNIST dataset.
In part, the normalized geodesic lengths calculated by our algorithm are a proxy for problem difficulty, thus we feel that verifying that our algorithm recapitulates the well-known fact that MNIST is easy was actually one of the nice sub-results of our manuscript.
For example, if you compare the MNIST and CIFAR10 data in Fig. 1 (i.e., rows 3 and 4 of the figure), you can see that the normalized length grows considerably earlier (in terms of test accuracy) for the CIFAR10 dataset than it does for the MNIST dataset.
In fact, even to extremely high test accuracy, the MNIST dataset still has a relatively straight connecting path between equivalent models (indicated by the normalized geodesic length being close to 1), suggesting that the loss surface is still quite convex even at high test accuracy (a known result).
"Bad formatting in many places, for instance Theorem 2.4, but there are many other places, where there are issues."
On closer inspection, we notice the figure labels in Fig. 1 (i.e., column and row labels) seem to have been eaten by the uploading process, as well as page breaks in a couple of equations.  These have been remedied in the most recent iteration.  We would greatly appreciate if the reviewer indicated other formatting issues that they may have spotted.  Edit: The (??) equation label in the text right below equation (19) will be fixed shortly--it was lost when I was performing equation alignment fixes.  (the (??) should be a (17)).
"No intuition regarding core theoretical results is given and the results themselves are very technical. The authors make very rough comparison with previous work, several of the techniques they use in the paper were already used before in that context (see for instance the proof of Theorem 2.4)"
We apologize, but we are legitimately unaware of work that establishes something akin to our Theorem 2.4, except perhaps other than work that has been concurrently submitted to ICLR 2017.  If the reviewer could point us towards a specific reference, we would, again, gladly incorporate that reference into our manuscript.
"In terms of the content, this is a very incremental work. Several conditions from the other papers on the topic can be relaxed with the use of such tools as Janson's inequality (the path-independence issue). Theoretical results up to section 2.3.3 are already well-known (see papers on the surface loss from NIPS'16 workshops). I have also problems with what the authors claim is the main result (Theorem 2.3.3). I dont think that the formula for epsilon in the statement of the theorem is correct, at least I dont see at all how it follows from the given proof (which I guess is in fact just a sketch of the proof even though the authors put it in the Appendix, where they could put the entire proof).
Furthermore, I think that the presented sketch of the proof is not correct, but it is hard to say since all the details are hidden so the reviewer needs to guess."
We apologize for any lack of clarity in the proof.  We have significantly expanded the text of the proof to fill in more details.  We hope that the reviewer finds the new discussion in the Appendix more clear.
If the reviewer finds a specific step of the proof to be inaccurate/wrong/puzzling, we would be happy to further supplement the proof with detail!
As discussed in my previous comment, even though the result that "linear networks have no local minima" is not new, our proof utilizes a new approach, and we leverage this new proof strategy to make progress on the question of the connectedness of neural networks with nonlinearities present.
Thus, we feel that including such "known" results in our paper is necessary for properly motivating Theorem 2.4.  We would like to reemphasize that we have tried to be clear in the text about which results are new, and which results are new proofs of older results (see for example our discussion of Kawaguchi's result shortly after Proposition 2.2).  If this distinction is *not* clear, we will happily amend the text.
"Many results in the paper are given in a "hand-waving" form which is not appropriate for such conferences as ICLR (educated guesses that form certain parameters to be small enough the Taylor expansion gives some desired results and many more place where theoretical claims are simply not clear at all).
I consider myself to have pretty strong theoretical background in that field and the way the theoretical results are presented do not convince me that these theoretical results are correct. And this is, if I understand right, what the authors claim is the main contribution of the paper.
I will hesitate to improve my score, since I have strong feeling that the proofs are not complete and furthermore the presentation of these results does not help me to figure out myself how to fix the parts that are not clear."
Again, we apologize for the lack of clarity.
Regarding the use of Taylor expansions in Proposition 2.3, it is simply to provide a convenient upper bound that controls a trigonometric expression of \alpha. We are not making any assumptions about the value of alpha, although the bound is typically going to become sharper for small values of alpha.
In fact, in Corollary 2.5 we explicitly make alpha converge to 0.
If the reviewer could be more specific about where they found it to be too "hand-wavy", we would appreciate it.  Hopefully the expanded proofs in the Appendix resolved many of these issues, but if any confusion remains, please let us know!
Can you elaborate more on the comparison with previous results regarding loss surface of deep neural nets ?
Dear AnonReviewer3,
Our results are part of an ongoing effort to understand the optimization landscape of neural networks. There are works that attempt to characterize and classify the critical points (local optima, saddles, strict saddles, etc) under different architectures and making different assumptions, and others tackle a more global question, namely studying conditions that make SGD succeed (or not).
Specifically, here are some similarities/differences between our results and some of the previously published ones. If the reviewer is aware of other related work we would gladly update our related work section.
-- [1], [2], [3] study the loss surface of deep networks from the perspective of statistical physics. Under appropriate mean-field assumptions, one can recast the loss of a deep neural network as a spherical spin glass. Many properties of this loss are well-understood, in particular the structure of its critical points. Our results differ from this line of work in the sense that we do not make the set of assumptions leading to the spin-glass model. This does not mean our results are better or worse;
so far we cover the two-layer ReLU model under sparse regularization, so our results can also be improved and generalized.
-- [7] provides valuable empirical data relating the first and second-order critical points in deep learning models. Their conclusions are very much in-line with our results: as SGD progresses, it tends towards regions of the parameter space that are locally flat along multiple directions, more than the intrinsic number of symmetries of the model. These empirical findings are consistent with our own numerical experiments, where we show that the normalized geodesic length increases as the energy decreases.
-- [4] and [5] These works study the interplay between ease of optimization and model overparametrization. This is one of the main points that our theory quantifies. The mentioned works study this question from different angles. [4] considers this question in the regime where the number of parameters exceeds the number of training points, and the arguments employed are substantially different from ours. [5] considers paths in the parameter space, but they are not constructed to connect two arbitrarily chosen points in the same level set.
-- [6] Studies "negative" results concerning the question of model architecture (or shaping the regression function) vs input data distribution. Again, our mathematical results particularize this interplay in the context of ReLU one-layer networks. This means we can say more precise things in our setting, as shown by the fact that we also produce a positive result (albeit asymptotically).
Note: [8] and [9] are also related to our analysis, but these are concurrent ICLR submissions which I did not have time to study yet.
References:
[1] A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun. The loss surfaces of multilayer networks. In Proc. AISTATS, 2015.
[2] Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, and Y. Bengio. Iden- tifying and attacking the saddle point problem in high-dimensional non-convex opti- mization. In Advances in Neural Information Processing Systems, pages 2933–2941, 2014.
[3] K. Kawaguchi. Deep learning without poor local minima. arXiv preprint arXiv:1605.07110, 2016.
[4] D. Soudry and Y. Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. arXiv preprint arXiv:1605.08361, 2016.
[5] I. Safran and O. Shamir. On the quality of the initial basin in overspecified neural networks. arXiv preprint arXiv:1511.04210, 2015.
[6] O. Shamir. Distribution-specific hardness of learning neural networks. arXiv:1609.01037, 2016.
[7] L. Sagun, L. Bottou, Y. LeCun, SINGULARITY OF THE HESSIAN IN DEEP LEARNING, arxiv:1611.07476
[8] https://openreview.net/forum?id=Hk85q85ee
[9] https://openreview.net/forum?id=B1YfAfcgl
I'd like to sincerely thank all the authors for clarifications. Unfortunately I still think that it is a strong reject paper. Authors' responses did not solve main problems that I pointed out. This paper is badly written, the results are incremental and presented in the unclear way. The proofs are presented in such a way that I cannot really verify whether proposed theoretical claims (which to the best of my understanding constitute the core of the paper) are correct even though I consider myself to be very familiar with the mathematical tools used here.
The authors should consider submitting this paper to a journal, but after a major revision.

Meta-review:
role: PC
data: comment: The paper presents an analysis of deep ReLU networks, and contrasts it with linear networks. It makes good progress towards providing a theoretical explanation of the difficult problem of characterizing the critical points of this highly nonconvex function. 

I agree with the authors that their approach is superior to earlier works based on spin-glass models or other approximations that make highly unrealistic assumptions. The ReLU structure allows them to provide concrete analysis, without making such approximations, as opposed to more general activation units.

The relationship between smoothness of data distribution and the level of model overparamterization is used to characterize the ability to reach the global optimum. The intuition that having no approximation error (due to overparameterization) leads to more tractable optimization problem is intuitive. Such findings have been seen before (e.g. in tensor decomposition, the global optima can be reached when the amount of noise is small). I recommend the authors to connect it to such findings for other nonconvex problems.  The paper does not address overfitting, as they themselves point out in the conclusion. However, I do expect it to be a very challenging problem.

Decision: Accept (Poster)