Title: Local minima in training of deep networks
Authors: Grzegorz Swirszcz
Wojciech Marian Czarnecki
Razvan Pascanu
Venue: ICLR 2017
Keywords: Theory
Deep learning
Supervised Learning
Optimization

Abstract: There has been a lot of recent interest in trying to characterize the error surface of deep models. This stems from a long standing question. Given that deep networks are highly nonlinear systems optimized by local gradient methods, why do they not seem to be affected by bad local minima? It is widely believed that training of deep models using gradient methods works so well because the error surface either has no local minima, or if they exist they need to be close in value to the global minimum. It is known that such results hold under strong assumptions which are not satisfied by real models. In this paper we present examples showing that for such theorem to be true additional assumptions on the data, initialization schemes and/or the model classes have to be made. We look at the particular case of finite size datasets. We demonstrate that in this scenario one can construct counter-examples (datasets or initialization schemes) when the network does become susceptible to bad local minima over the weight space.

TL;DR: As a contribution to the discussion about error surface and the question why "deep and cheap" learning works so well we present concrete examples of local minima and obstacles arising in the training of deep models.

Review discussion:
1. Most of your analysis appears to involve saturated RELUs not receiving any gradient for a large number of examples, following a problematic initialization. How much of this analysis do you expect to carry through to non-saturating units (e.g. leaky RELU)?
2. In proposition 4, you state that there exists a dataset such that the gradient based techniques achieve zero training error with probability at least 1-epsilon for an arbitrary epsilon and for any feedforward  network.
While in the proof you show that epsilon is not randomly chosen but instead it depends on the initialization scheme of weights (precisely the mean of normal distribution), Could you clarify your proposition's formulation?
First of all, we would like to thank you for your helpful questions, we made small changes to the paper addressing them.
Additionally:
Regarding question 1:
We decided to restrict ourselves to ReLU (and a sigmoid in one case of the "jellyfish" example) activation to keep our examples related to practical situations. ReLU are after all the most commonly used activation functions. Our counterexamples fall into two categories. One is illustrating the importance of the initialization, the other one on the data structures. The regression examples do not rely only on not receiving any gradients from a lot of examples, one of them (Figure 4 (b)) has actually all ReLUs activated at at least one datapoint in the bad local minimum. The mechanism of getting stuck in a local minimum is (unfortunately) not solely caused by ReLUs simply saturating.
The "bad initialization" examples we presented rely mostly on not receiving any gradient. The point we were making was the importance of the initialization, and trying to have more active units leads to technical complications, which we found were making the paper less reader-friendly. One can also construct examples with all ReLUs activated that still perform badly, at a price of making the theoretical arguments more technical.
Finally, those results do generalize to other activation functions, the constructions do not carry over in a straightforward manner, but the results hold. We plan to focus more on those other cases as a part of our future research agenda.
Regarding question 2:
We have changed the formulation of Proposition 4, thank you for pointing out the lack of clarity in our statement of the result.
Is it possible to sustain the distinction between structure and data properties? All nonlinear results I know of assume something about the dataset (typically Gaussian). The error surface structure is inextricably linked to the dataset as well as the model parameters. It is not surprising that finite sized datasets and networks can be degenerate (the spin glass and random Gaussian field results compute the _average_ behavior).
How would these results translate to sigmoid or soft ReLU nonlinearities? Is this pathology specific to the hard ReLU?
The results showing that Adam underperforms GD on the Jellyfish problem (particularly on the sigmoid) are striking, maybe the learning algorithm must figure in the picture along side initialization and dataset properties. Do you have any further intuitions for this?
What initialization was used for the Figure 2 results? Simply scaling the input, if the weight matrices are initialized with zero biases, will not change the regions over which each ReLU activates.
Another explanation is that, with such small inputs, the input-output association is that much weaker and learning takes longer to converge (i.e., the \tau=.00001 results would hit an accuracy of 1 if allowed to run far longer; these results may be an artifact of a constant budget of updates, rather than using a convergence criterion). Even a linear network may give this behavior.
"Previous results (Dauphin et al., 2013; Saxe et al., 2014; Choromanska et al., 2015) provide insightful description of the error surface of deep models divorced from the dataset or initialization." Doesn't a complete charaterization of the error surface cover all parameter points, and hence all possible initializations?
For instance, Saxe et al. considers the impact of initialization. And as argued earlier, isn't the role of the dataset very much a part of prior results?
Are you attempting to separate saddle points from local minima? Most previous results say that saddle points exist, and so there necessarily exist initializations from which gradient learning will not converge (if you initialize at any saddle point, or on the attracting separatrix of a saddle point). It seems like proposition 4, for instance, is almost certainly a saddle point.
Thank you for that insightful set of remarks and questions. They helped us to improve our presentation - we have submitted an improved .pdf.
In addition we address the particular questions in the discussion below:
[Is it possible to sustain the distinction between structure and data properties?]
We believe that this distinction is not possible to be sustained in generality. We address this throughout the discussion below, we would be very happy to elaborate further on that point if necessary, of course.
[How would these results translate to sigmoid or soft ReLU nonlinearities? Is this pathology specific to the hard ReLU?]
This pathology observed is by no means specific to hard ReLU. We can construct bad local minima for various activation functions.
In particular - the local minima for sigmoid-based networks are really fascinating geometrically. Their constructions are more technical, we are planning to devote another paper to the study of them.
In this paper we decided to present examples based on ReLU because they are the most commonly used activation functions, but the phenomenon of bad local minima is far more prominent than just being an effect of not receiving gradients from deactivated ReLU-s. Bad local minima are a part of the error surface landscape. We do not encounter them in practice - but we believe this is because we are dealing with special cases of data (similar claim was made recently by Lin and Tegmark).
We have modified the paper to emphasize the fragments regarding the above matters that as Remarks 2 and 3. We would like to thank you for bringing it to our attention that they were not highlighted sufficiently.
[The results showing that Adam underperforms GD on the Jellyfish problem]
The fact that the learning algorithm should be a part of the whole picture is a great point.
While most of the theory is based on simply following gradients, in practice using more complex optimizers might have a really strong effect on being able to find good solutions.
In particular, we believe that in case of Jellyfish, due to Adam’s aggressive behaviour (taking big steps) it simply jumps out of the basin of attraction of the global solution and ends up in the degenerate case.
In other words - it seems like Jellyfish has two main basins of attraction - one converging to a perfect separation and one to the degenerate linear model. The way we constructed the example was with the purpose of enlarging the second basin vs. the first basin. Apparently, Adam’s estimates of the curvature make it even more prone to fall into this suboptimal solution.
In addition, an important role in understanding this behaviour is not only the algorithm, but also the approximations it makes on the way. E.g., Adam does not use the actual curvature, but rather a crude diagonal approximation. The numerical imprecision also plays a big role. The effect of these approximations can be harder to understand mathematically, but one can try to understand how noise affects the dynamics of learning (which we always have in our models). Those are some reasons why empirically, an algorithm that should be better, can underperform.
[Doesn't a complete characterization of the error surface cover all parameter points, and hence all possible initializations?]
This question highlights the key goal of our programme: to characterize completely - or at least to the best of the scientific ability - the error surface. The previous results do provide very insightful intuitions, but in our opinion this is just the beginning of the journey to the full understanding of the error surface. It is an extremely complicated object that we are only starting to try to understand. The results obtained so far, rely on assumptions that do not hold in practice and we are convinced that those assumptions cannot be simply removed. We believe that the further advancements in our understanding of the error surface will not rely on finding an “interface” to an existing off-the-shelf theorem, should it be from statistical physics, general differential geometry or PDE-s etc. Those provide great intuitions about what mechanisms may possibly lie behind the good behavior of the models we train, but we do not think they can lead to proving anything in general about training of error surfaces of Neural Networks.
They are simply too general and they do not take into account the particular structures we are leveraging when training NN-s. Whatever theorem really pertinent to the error surfaces can be proved - it will have to have some assumptions about the data, initialization etc. It will have to be custom-tailored, not ready-made. One of the main goals of this paper is precisely that - to highlight a point that in order to advance the understanding of the error surface we need to look at the data and find realistic assumptions, because we cannot hope for an easy general “training of NN-s works - always” theorem.
Coming back to the specific question: “Doesn't a complete characterization of the error surface cover all parameter points, and hence all possible initializations?” - yes, that is the very point we are making. If the “complete characterization” was indeed universally valid, we would not be able to break the learning with the initialization. We were, so the theorem needs some assumptions, what we have so far is a very insightful intuition rooted in a solid analysis and supported by many practical examples, but it is not yet a proven theorem.
[Scaling factor / Figure 2 questions]
The scaling factor is global, hence Adam should be able to deal and correct for it easily.
In particular, for the linear case, Adam would easily correct the learning rate and result in roughly the same learning dynamics as for the unscaled gradients.
Furthermore, the number of steps each experiment is run for is sufficiently large (order of magnitude more than what it would be required to converge on the unscaled data).
Therefore one could not prove the contrary of what is stated in the question, we believe slowness to convergence not to be the main reason behind what we see.
[Are you attempting to separate saddle points from local minima?]
Yes,  throughout the whole paper, when we talk about minima we consistently mean the actual minima in a mathematical sense, not saddles.
In particular, Proposition 4 is based on neither of these two. What we mean by “never converges” is that the gradient is never zero, and the optimization simply never stops (assuming infinite precision arithmetics) and never increases the training accuracy. Due to this property this is clearly not a saddle nor a local minimum. The exact same construction and proof would lead to a (strict) local minimum (and consequently - convergence, but to a highly suboptimal solution) if one used L2 loss instead though. We updated the pdf to make it very clear in the proposition, which now says:
There exist an infinite amount of normalized (whitened) datasets, such that for any feed forward rectifier network architecture and an arbitrary $\epsilon \in [0, 1)$, there exists a normal distribution used to initialize the weights and biases initialized to $0$ such that with probability at least $1-\epsilon$ the gradient based techniques using log loss never achieve $0$ training error nor they ever converge (gradient is never zero).
Furthermore, this dataset can have a full rank covariance matrix and be linearly separable.
[Separatrix of the saddle]
Initializing at a separatrix is in our opinion an interesting issue in itself. We are convinced (see the answer above) that this is not what is happening in Proposition 4, but we would like to share our thoughts on that nevertheless.
Initializing at a separatrix can be looked at both from a theoretical and practical standpoint. Theoretically - separatrix is a zero-measure set, and the general agreement in the ODE community is that as such it can be safely ignored (like for example in the “transversality” framework of Arnold and his group).
Practically - we are in a finite subset of the parameter space already (due to the numeric representations computers use), so the above argument does not carry over directly.
Nevertheless, it is known that even for the most regular ODE-s the trajectories are usually very complicated. Even for polynomial vector fields the trajectories need not be even analytic, but only C^\infty.
In particular, as a separatrix of a saddle is a measure-zero repelling set, given that we are using an approximate methods in combination with numerical errors would make it close to impossible to stay on it in the process of training.
That could occur if our system was for example $\frac{\partial}{partial x} - \frac{partial}{partial y}$ initialized at (0,1), but unless the separatrix is a straight line, it is extremely unlikely that a gradient descent implemented as a discrete process could keep tracking it and not be affected by the push along the repelling subbundle.
In the proof of Proposition 4, the probability "1-epsilon" seems to be actually a very small quantity (a probability raised to the power of the number of weights in the network). Is it even an issue if the training procedure fails to converge with such low probability?
That is another excellent question, right to the very point of the paper. The idea behind Proposition 4 is to show the significance of taking into account the initialization of the network when proving convergence. It is correct - the epsilon depends exponentially on the number of weights.
However, the crucial bit here is that this exponent can be easily controlled by the parameters of the distribution used to initialize the weights.
In particular, Figure 6 shows that in order to have the probability of 95% of failing  all we have to do, when our network has 100 layers of 10,000 units, is to move the mean of the initializer to 0.24. Note, that it holds despite the fact it comes from the exponentially decaying probability with respect to the number of weights.
The claim here is not that this is something that in practice would easily kill the learning process. What our point is, it is to show that relatively small changes in typical initialization schemes lead to surprisingly high probability of failure and thus they have to be taken into account while trying to prove any convergence/local minima theorems.
The paper studies some special cases of neural networks and datasets where optimization fails. Most of the considered models and datasets are however highly constructed and do not follow the basic hyperparameters selection and parameter initialization heuristics. This reduces the practical relevance of the analysis.
The experiment "bad initialization on MNIST" shows that for very negative biases or weights drawn from a non-centered distribution, all ReLU activations are "off" for all data points, and thus, optimization is prevented. This never occurs in practice, because using proper initialization heuristics avoid these cases.
The "jellyfish" dataset constructed by the authors is demonstrated to be difficult to fit by a small model.
However, the size/depth of the considered model is unsuitable for this problem.
Proposition 4 assumes that we can choose the mean from which the weight parameters are initialized. This is typically not the case in practice as most initialization heuristics draw weight parameters from a distribution with mean 0.
Proposition 5 considers infinitely deep ReLU networks. Very deep networks would however preferably be of type ResNet.
The goal of this paper is to bridge the gap between the theoretical research and practical experience. On one hand a tremendous effort was made to come up with successful recipes for hyperparameter selection, initialization etc. The point made by the reviewer is 100% valid - yes, they work! They are the very founding stone of the development and the staggering success of deep learning.
Yet the very same time - they are to a big extent absent in the universe of theoretical research. We do not see them in the assumptions of the results trying to prove something about why deep learning works as well as it does.
We are not by any means trying to show that deep learning is not working.
On the contrary. What we are trying to say is: if one strays from the path laid out by the good practices and experience of thousands of successful experiments, things break badly on really simple examples.
Thus, it is important to narrow the gap between the theory and practice, in particular, for the theoretical research to embrace more closely what is known empirically to the practitioners.
We thought that ICLR, a forum bringing together both more practically-oriented and more theoretically focused communities was a good place to make this observation.
The material in the paper has been reorganized, to improve the presentation, reflecting the valuable feedback the authors received from the reviewers and readers.
This paper studies the error surface of deep rectifier networks, giving specific examples for which the error surface has local minima. Several experimental results show that learning can be trapped at apparent local minima by a variety of factors ranging from the nature of the dataset to the nature of the initializations. This paper develops a lot of good intuitions and useful examples of ways that training can go awry.
Even though the examples constructed in this paper are contrived, this does not necessarily remove their theoretical importance. It is very useful to have simple examples where things go wrong.
However the broader theoretical framing of the paper appears to be going after a strawman.
“The underlying easiness of optimizing deep networks does not simply rest just in the emerging structures due to high dimensional spaces, but is rather tightly connected to the intrinsic characteristics of the data these models are run on.” I believe this perspective is already contained in several of the works cited as not belonging to this perspective. Choromanska et al., for instance, analyze Gaussian inputs, and so clearly make claims based on characteristics of the data the models are run on. More broadly, the loss function is determined jointly by the dataset and the model parameters, and so no account of the error surface can be separated from dataset properties. It is not clear to me what ‘emerging structures due to high dimensional spaces’ are, or what they could be, that would make them independent of the dataset and initial model parameters. The emerging structure of the error surface is necessarily related to the dataset and model parameters.
Again, a key worry with this paper is that it is aiming at a strawman: replica methods characterize average behavior for infinite systems, so it is not surprising that specific finite sized systems might yield poor optimization landscapes. The paper seems surprised that training can be broken with a bad initialization, but initialization is known to be critical, even for linear networks: saddle points are not innocuous, with bad initializations dramatically slowing learning (e.g. Saxe et al. 2014).
It seems like the proof of proposition 5 may have an error. Suppose cdf_b(0) = 0 and cdf_W(0)=1/2. We have P(learning fails) >= 1 - 1/2^{h^2(k-1)}, meaning that the probability of failure _increases_ as the number of hidden units increases. It seems like it should rather be (ignoring the bias) p(fails) >= 1 - [ 1 - p(w<0)^h^2]^{k-1}. In this case the limit as k-> infinity depends on how h scales with k, so it is no longer necessarily true that “one does not have a globally good behaviour of learning regardless of the model size.”
The paper also appears to insufficiently distinguish between local minima and saddle points. Section 3.1 states it shows training being stuck in a local minimum, but this is based on training with a fixed budget of epochs. It is not possible to tell whether this result reflects a genuine local minimum or a saddle point based on simulation results.
It may also be the case that, while rectifiers suffer from genuine blind spots, sigmoid or soft rectifier nonlinearities may not. On the XOR problem with two hidden nodes, for instance, it was thought that were local minima but in fact there are none (e.g. L. Hamey, “Analysis of the error surface of the XOR network with two hidden nodes,” 1995).
If the desire is simply to show that training does not converge for particular finite problems, much simpler counterexamples can be constructed and would suffice: set all hidden unit weights to zero, for instance.
In the response to prereview questions, the authors write ‘If the “complete characterization” [of the error surface] was indeed universally valid, we would not be able to break the learning with the initialization’ but, as mentioned previously, the basic results for even deep linear networks show that a bad initialization (at or near a saddle point) will break learning. Again, it seems this paper is attacking a straw man along the lines of “nothing can possibly go wrong with neural network training.” No prior theoretical result claims this.
The Figure 2 explanation seems counterintuitive to me. Simply scaling the input, if the weight matrices are initialized with zero biases, will not change the regions over which each ReLU activates.
That is, this manipulation does not achieve the goal of concentrating “most of the data points in very few linear regions.” A far more likely explanation is that the much weaker scaling has not been compensated by the learning algorithm, but the algorithm would converge if run longer. The response notes that training has been conducted for an order of magnitude longer than required for the unscaled input to converge, but the scaling on the data is not one but five orders of magnitude—and indeed the training does converge without issue for scaling up to four orders of magnitude. The response notes that Adam should compensate for the scaling factor, but this depends on the details of the Adam implementation—the epsilon factor used to protect against division by zero, for example.
This paper contains many interesting results, but a variety of small technical concerns remain.
First of all, the authors would like to thank the reviewer for spotting the technical flaws that slipped through their quality control. Fortunately, they do not invalidate any of the results and they were easy to repair.
Namely:
- There was indeed an error in the proof Proposition 5 which now has been corrected. Fortunately, that had no influence on the formulation of the Proposition itself.
In the analysis provided, h is treated as a constant, only k (depth) is a variable, thus leading to an increasing probability of training failure. One could strengthen this result by providing the upper bound for the growth of h with respect to k for this results to hold (which roughly speaking would be logarithmic), which we omitted for simplicity.
- The claim in 3.1 about local minima got there due to an unfortunate error during the final organization of the material. 3.1 initially referred to the ReLU example, in which case we do have a local minimum indeed - which we prove. Thank you for spotting that. We could not agree more that just seeing empirically that the training slows down is not sufficient to make any claims about being stuck in a local minimum. We have made the necessary correction to the text. Especially in this paper that strives to improve our understanding of the basics of the landscape of the error surface the distinction needs to be clear - again, thank you for catching that.
The argument “If the desire is simply to show that training does not converge for particular finite problems, much simpler counterexamples can be constructed and would suffice: set all hidden unit weights to zero, for instance.” was a crucial object in a heated debate among the authors when they were planning this research. The authors were facing the dilemma:
1. Construct an example using ReLUs. Pros: These are the most commonly used activation functions. Cons: ...but it is trivial to deadlock them. Just put all the datapoints in the inactive region for example.
2. Use Sigmoids. Pros: the examples are highly nontrivial and have a fascinating geometry. Cons: The paper would get criticized for studying sigmoids that are so passe.
The final decision was to construct interesting ReLU based examples.
As much as the authors fully agree that it is extremely easy to construct a trivial example of a situation with no learning occurring, especially when using ReLUs, they believe that there is a value in constructing nontrivial ones, and that the examples presented fall into that category.
In particular “all weights set to zero” example is in the authors’ opinion a less interesting than the ones constructed in the paper, because it is a zero measure (and even a high-codimension isolated set) in a parameter space, and in most cases an unstable singularity of a training process.
Regarding Figure 2, and the number of linear regions on which the data lies by scaling down each examples: The code used to produce those results used an initialization where biases were non zero (which is the default initialization provided by torch), but sampled from the same Gaussian as the weights.
Hence scaling down the images has the desired effect.
Also, since the Gaussian from which we sample the biases has mean 0, the region on which all the data points end up is not the one where all hidden units are inactive (i.e. the biases are not large negative numbers).
The material in the paper has been reorganized, to improve the presentation, reflecting the valuable feedback the authors received from the reviewers and readers.
The main merit of this paper is to draw again attention to how crucial initialization of deep network *can* be; and to counter the popular impression that modern architectures and improved gradient descent techniques make optimization local minima and saddle points no longer a  problem.
While the paper provides interesting counter-examples that showcase how bad initialization mixed with particular data can lead the optimization to get stuck at a poor solution, these feel like contrived artificial constructs. More importantly the paper does not consider popular heuristics that likely help to avoid getting stuck, such as: non-saturating activation functions (e.g. leaky RELU), batch-norm, skip connections (resnet), that can all be thought of as contributing to keep the gradients flowing.
The paper puts up a big warning sign about potential initialization problems (with standard RELU nets), but without proposing new solutions or workarounds, nor carrying out a systematic analysis of how this picture is affected by most commonly used current heuristic techniques (in architecture, initialization and training). Such a broader scope analysis, especially if it did lead to insights of practical relevance, could much increase the value of the paper for the reader.
When constructing examples one often faces a tradeoff between simplicity and universality. What we decided to strive for in this paper was to present elegant examples with a nice, human-readable structure. Believing in the “one picture worth a thousand words” philosophy we decided to go with examples which actually could be drawn.
We agree that examples aimed at the common successful heuristics like batch norm or skip connections would be very interesting.
However, due to the very nature of those heuristics - they would need to be much more technically complicated.
Thus - in the authors minds - the paper benefits from simplicity, and sends a positive message: we really do need those techniques, things can go very wrong very quickly if we do not use them. The authors believed, that there was a value in demonstrating theoretical reasons for us needing the modern deep learning toolkit.
Of course, we have ResNets and batch normalization for good reasons, they were invented to deal with problems encountered in practice.
However, what the paper adds in the authors’ opinion is the observation, that the difficulties that gave birth to ResNets, skip  connections, momentum or dropouts are fundamental in nature.
Much of the theoretical research is aimed at proving good behaviour of learning under general weak assumptions not addressing the specifics of the architecture and the details of learning. This might give an impression that the heuristics the deep learning developed in practice are more of a gimmick, and that their purpose is is mostly to speed up the algorithms. What one of the goals of our paper was - was to bring to the attention the fact, that we need those heuristics for deep reasons and that they are often fundamental in order to learn.
The authors agree with the suggestion that proposing new solutions or workarounds would provide a lot of extra value.
However, that is a very challenging task. One of the practices of the scientific method is, when facing a big and important challenge that cannot be dealt with in “one blow”, is to attack it one step at a time. The authors believe that a good starting point is to first identify the “canonical” list of phenomena that present obstacles to the training, to then devise good strategies to make training work despite them. Coming up with better strategies is the authors’ ultimate endgoal. This paper is meant as “recognize what you are fighting against”  part of the strategy.
Final remark - the authors know examples that “deadlock” networks based on non-saturating activation functions as well. Being limited by the volume we decided to present ReLU based examples, as they are the most commonly used activations.
But the “deadlocking” is by no means caused only by the inactive regions.
The material in the paper has been reorganized, to improve the presentation, reflecting the valuable feedback the authors received from the reviewers and readers.
When constructing examples one often faces a tradeoff between simplicity and universality. What we decided to strive for in this paper was to present elegant examples with a nice, human-readable structure. Believing in the “one picture worth a thousand words” philosophy we decided to go with examples which actually could be drawn.
We agree that examples aimed at the common successful heuristics like batch norm or skip connections would be very interesting.
However, due to the very nature of those heuristics - they would need to be much more technically complicated.
Thus - in the authors minds - the paper benefits from simplicity, and sends a positive message: we really do need those techniques, things can go very wrong very quickly if we do not use them. The authors believed, that there was a value in demonstrating theoretical reasons for us needing the modern deep learning toolkit.
Of course, we have ResNets and batch normalization for good reasons, they were invented to deal with problems encountered in practice.
However, what the paper adds in the authors’ opinion is the observation, that the difficulties that gave birth to ResNets, skip  connections, momentum or dropouts are fundamental in nature.
Much of the theoretical research is aimed at proving good behaviour of learning under general weak assumptions not addressing the specifics of the architecture and the details of learning. This might give an impression that the heuristics the deep learning developed in practice are more of a gimmick, and that their purpose is is mostly to speed up the algorithms. What one of the goals of our paper was - was to bring to the attention the fact, that we need those heuristics for deep reasons and that they are often fundamental in order to learn.
The authors agree with the suggestion that proposing new solutions or workarounds would provide a lot of extra value.
However, that is a very challenging task. One of the practices of the scientific method is, when facing a big and important challenge that cannot be dealt with in “one blow”, is to attack it one step at a time. The authors believe that a good starting point is to first identify the “canonical” list of phenomena that present obstacles to the training, to then devise good strategies to make training work despite them. Coming up with better strategies is the authors’ ultimate endgoal. This paper is meant as “recognize what you are fighting against”  part of the strategy.
Final remark - the authors know examples that “deadlock” networks based on non-saturating activation functions as well. Being limited by the volume we decided to present ReLU based examples, as they are the most commonly used activations.
But the “deadlocking” is by no means caused only by the inactive regions.
The material in the paper has been reorganized, to improve the presentation, reflecting the valuable feedback the authors received from the reviewers and readers.

Meta-review:
role: PC
data: comment: This paper analyzes under which circumstances bad local optima prevent effective training of deep neural networks. The contribution is real, but the gap between the proposed scenarios and real training scenarios diminishes its importance.

Decision: Reject