Title: LOCAL MINIMA IN TRAINING OF DEEP NETWORKS

Abstract: There has been a lot of recent interest in trying to characterize the error surface of deep models. This stems from a long standing question. Given that deep networks are highly nonlinear systems optimized by local gradient methods, why do they not seem to be affected by bad local minima? It is widely believed that training of deep models using gradient methods works so well because the error surface either has no local minima, or if they exist they need to be close in value to the global minimum. It is known that such results hold under very strong assumptions which are not satisfied by real models. In this paper we present examples showing that for such theorem to be true additional assumptions on the data, initialization schemes and/or the model classes have to be made. We look at the particular case of finite size datasets. We demonstrate that in this scenario one can construct counter-examples (datasets or initialization schemes) when the network does become susceptible to bad local minima over the weight space.

1. INTRODUCTION:

Deep Learning (LeCun et al., 2015; Schmidhuber, 2015) is a fast growing subfield of machine learning, with many impressive results. One particular criticism often brought up against this family of models is the fact that it relies on non-convex functions which are optimized using local gradient descent methods. This means one has no guarantee that the optimization algorithm will converge to a meaningful minimum or even that it will converge at all. However, this theoretical concern seems to have little bearing in practice.

In Dauphin et al. (2013) a conjecture had been put forward for this based on insights from statistical physics which point to the scale of neural networks as a possible answer. The claim is that the error structure of neural networks might follow the same structure as that of random Gaussian fields which have been recently understood and studied in Fyodorov & Williams (2007) ; Bray & Dean (2007) . The critical points of these functions, as the dimensionality of the problem increases, seem to have a particularly friendly behaviour where local minima align nicely close to the global minimum of the function. Choromanska et al. (2015) provides a study of the conjecture by mapping deep neural models onto spin-glass ones for whom the above structure holds. These work has been extended further (see Section 2 for a review of the topic).

We believe many of these results do not trivially extend to the case of finite size datasets/finite size models. The learning dynamics of the neural network in this particular case can be arbitrarily bad. Our assertions are based on constructions of counterexamples that exploit particular architectures, the full domain of the parameters and particular datasets.

2. LITERATURE REVIEW:

One view, that can be dated back to Baldi & Hornik (1989) , about why the error surface of neural networks seems well behaved is the one stated in Dauphin et al. (2013) . We would refer to this hypothesis as the "no bad local minima" hypothesis. In Baldi & Hornik (1989) it is shown that an MLP with a single linear intermediate layer has no local minima, only saddle points and a global minimum. This intuition is carried further by Saxe et al. (2014; 2013) , where deep linear models are studied. While, from a representational perspective, deep linear models are not useful, the hope is that the learning dynamics of such models can be mathematically understood while still being rich enough to mirror the dynamics of nonlinear networks. The findings of these works are aligned with Baldi & Hornik (1989) and suggest that one has only to go through several saddles to reach a global minimum.

These intuitions are expressed clearly for generic deep networks in Dauphin et al. (2013) . The key observation of this work is that intuitions from low-dimensional spaces are usually misleading when moving to high-dimensional spaces. The work makes a connection with deep results obtained in statistical physics. In particular Fyodorov & Williams (2007) ; Bray & Dean (2007) showed, using the Replica Theory (Parisi, 2007) , that random Gaussian error functions have a particular friendly structure. Namely, if one looks at all the critical points of the function and plots error versus the (Morse) index of the critical point (the number of negative eigenvalues of the Hessian), these points align nicely on a monotonically increasing curve. That is, all points with a low index (note that every minimum has this index equal to 0) have roughly the same performance, while critical points of high error implicitly have a large number of negative eigenvalue which means they are saddle points.

These observations align also with the theory of random matrices (Wigner, 1958) which predicts the same behaviour for the eigenvalues of a random matrix as the size of the matrix grows. The claim of Dauphin et al. (2013) is that same structure holds for neural network as well, when they become large enough. Similar claim is put forward in Sagun et al. (2014) . The conjecture is very appealing as it provides a strong argument why deep networks end up performing not only well, but also reliably so. Choromanska et al. (2015) provides a study of the conjecture that rests on recasting a neural network as a spin-glass model.To obtain this result several assumptions need to be made, which the authors of the work, at that time, acknowledged that were not realistic in practice. The same line of attack is taken by Kawaguchi (2016) . Goodfellow et al. (2016) argues and provides empirical evidence that while moving from the original initialization of the model along a straight line to the solution (found via gradient descent) the loss seems to be only monotonically decreasing, which speaks towards the apparent convexity of the problem. Soudry & Carmon (2016) ; Safran & Shamir (2015) also look at the error surface of the neural network, providing theoretical arguments for the error surface becoming well-behaved in the case of over-parametrized models.

A different view, presented in Lin & Tegmark (2016) ; Shamir (2016), aligned with this work, is that the underlying easiness of optimizing deep networks does not simply rest just in the emerging structures due to high-dimensional spaces, but is rather tightly connected to the intrinsic characteristics of the data these models are run on.

3. THEORETICAL EXAMPLES AND RESULTS:

We propose to analyze the error surface of rectified MLPs on finite datasets. The approach we take is a construction one. We build examples of datasets and model initializations that result in bad learning dynamics. In most examples we use ReLU units, as they are the most commonly used activation functions for both classification and regression tasks (e.g. in deep reinforcement learning (Mnih et al., 2015; 2016) ). It is worth noting, however, that the phenomena we are demonstrating are not limited in nature to ReLU setup and they manifest themselves also for non-saturating activation functions like sigmoids.

3.1. LOCAL MINIMA IN A RECTIFIER-BASED REGRESSION:

In Figure 1 we present 3 examples of local minima for regression using a single layer with 1, 2 and 3 hidden rectifier units on 1-dimensional data. For the sake of simplicity of our presentation we will describe in detail the case with 1 hidden neuron, the other two cases can be treated similarly. In case of one hidden neuron the regression problem becomes

EQUATION

)

Consider a dataset D 1 (see Figure 1 (a)):

(x 1 , y 1 ) = (5, 2), (x 2 , y 2 ) = (4, 1), (x 3 , y 3 ) = (3, 0), (x 4 , y 4 ) = (1, -3), (x 5 , y 5 ) = (-1, 3).

Proposition 1. For the dataset D 1 and L defined in Equation (1) the point v = 1, b = -3, w = 1, c = 0 is a local minimum of L, which is not a global minimum.

Proof. See Appendix B.4.

Remark 1. The point (1, -3, 1, 0) is a minimum, but it is not a "strict" minimum -it is not isolated, but lies on a 1-dimensional manifold at which L ≡ 18 instead. Proof. Analogous to the previous one.

Maybe surprisingly, the global solution has a blind spot -all neurons deactivate in x 3 . Nevertheless, the network still has a 0 training error. This shows that even though blind spots were used previously to construct very bad examples for neural nets, sometimes they are actually needed to fit the dataset. Proposition 3. Let us consider a dataset D 3 with d = 1, given by points (x 1 , y 1 ) = (-1, 3), (x 2 , y 2 ) = (0, 0), (x 3 , y 3 ) = (1, -1), (x 4 , y 4 ) = (10, -3), (x 5 , y 5 ) = (11, -4), (x 6 , y 6 ) = (12, -6) (Figure 1 (c) ). Then, for a rectifier network with m = 3 hidden units and a squared error loss the set of weights w = (-1.5, -1.5,

1.5), b = (1, 0, -13 -1 6 ), v = (1, 1, -1), c = -1 is a bet- ter local minimum than the local minimum obtained for w = (-2, 1, 1), b = (3+ 2 3 , -10, -11), v = (1, -1, -1), c = -3.

Proof. Completely analogous, using the fact that in each part of the space linear models are either optimal linear regression fits (if there is just one neuron active) or perfect (0 error) fit when two neurons are active and combined.

Note that again that the above construction is not relying on the blind spot phenomenon. The idea behind this example is that if, due to initial conditions, the model partitions the input space in a suboptimal way, it might become impossible to find the optimal partitioning using gradient descent. Let us call (-∞, 6) the region I, and [6, ∞) region II. Both solutions in Proposition 3 are constructed in such way that each one has the best fit for the points assigned to any given region, the only difference being the number of hidden units used to describe each of them. In the local optimum two neurons are used to describe region II, while only one describes region I. Symmetrically, the better solution assigns two neurons to region I (which is more complex) and only one to region II.

We believe that the core idea behind this construction can be generalized (in a non-trivial way) to high-dimensional problems. We plan to extend the construction as future work.

Remark 3. In the examples we used only ReLU (and in one case a sigmoid) activation functions, as they are the most common used in practice. The similar examples can be constructed for different activation functions, however the constructions need some modifications and get more technically complicated.

3.2. BAD INITIALIZATION:

In this section we prove some general results regarding bad initialization phenomenon.

Proposition 4. There exist an infinite amount of normalized (whitened) datasets, such that for any feed forward rectifier network architecture and an arbitrary ∈ [0, 1), there exists a normal distribution used to initialize the weights and biases initialized to 0 such that with probability at least 1the gradient based techniques using log loss never achieve 0 training error nor they ever converge (gradient is never zero). Furthermore, this dataset can have a full rank covariance matrix and be linearly separable.

Proof. See Appendix B.1.

Even though the above construction requires control over the means of the normal distributions the weights are drawn from, as one can see in Figure 2 , they do not have to be very large in practice.

In particular, if one uses an initialization with σ as prescribed by LeCun et al. (1998) or Glorot & Bengio (2010) then the value of µ = 0.24 is sufficient to break the learning, even if we have 10, 000 hidden units in each of 100 hidden layers. Using fixed σ = 0.01 instead fails even with µ = 0.07. It is worth noting that even though this observations is about the existence of such dataset, our proof is actually done by construction, meaning that we show a way to build infinite amount of such datasets (as opposed to purely existential proofs). We would like to remark that it was well known that the initialization was important for the behaviour of learning (Glorot & Bengio, 2010; LeCun et al., 1998; Sutskever et al., 2013; Pascanu et al., 2013) . Here we are exploiting these ideas in order to better understand the error surface of the model.

If we do not care about the lack of convergence, and we are simply interested in learning failure, we can prove an even stronger proposition, which works for every single dataset: Proposition 5. For every dataset, every feed forward rectifier network built for it, and every distribution used to initialize both weights and biases such that

E[w] = 0, E[b] = 0, Var[w] > 0, Var[b] ≥ 0,

the probability that the gradient based training of any loss function will lead to a trivial model (predicting the same label for all datapoints) goes to 1 as the number of hidden layers goes to infinity.

Proof. See Appendix B.2.

We can extend the previous proposition to show that for any regression dataset a rectifier model has at least one local minimum with a large basin of attraction (over the parameter space). Again, we rely on the blind spots of the rectified models. We show that there exists such blind spot that corresponds to a region in parameter space of same dimensionality (codimension 0). The construction relies on the fact that the dataset is finite. As such, it is bounded, and one can compute conditions for the weights of any given layer of the model such that for any datapoint all the units of that layer are deactivated. Furthermore, we show that one can obtain a better solution than the one reached from such a state. The formalization of this result is as follows.

We consider a k-layer deep regression model using m ReLU units ReLU(x) = max(0, x). Our dataset is a collection

(x i , y i ) ∈ R d × R, i = 1, . . . , N . We denote h n (x i ) = ReLU(W n h n-1 (x i ) + b n )

where the the ReLU functions are applied component-wise to the vector W n h n-1 (x i ) and h 0 (x i ) = x i . We also denote the final output of the model by

EQUATION

)

Let us state two simple yet in our opinion useful Lemmata.

Lemma 1 (Constant input). If x 1 = . . . = x N , then the solution to regression (2) has a constant output M ≡ y1+...+y N N (the mean of the values in data).

Proof. Obvious from the definitions and the fact, that y1+...+y

N N = arg min c N i=1 (c -y i ) 2 .

Lemma 2. If there holds W 1 x i < -b 1 for all i-s, then the model M has a constant output. Moreover, applying local optimization does not change the values of W 1 , b 1 .

Proof. Straightforward from the definitions.

Combining these two lemmata yields:

Corollary 1. If for any 1 ≤ j ≤ k there holds W n h n-1 < -b n for all i-s then, after the training, the model M will output y1+...+y N N .

We will denote M ({a 1 , . . . , a L }) = a1+...+a L L the mean of the numbers a 1 , . . . , a L .

Definition 1. We say that the dataset (x i , y i ) is decent if there exists r such that M ({y p :

x p = x r } = M ({y p : p = 1, . . . , N }). Theorem 1. Let θ = ((W n ) k n=1 , (b n ) k n=1

) be any point in the parameter space satisfying W n h n (x i ) < -b n (coordinate-wise) for all i-s. Then i) θ is a local minimum of the error surface, ii) if the first layer contains at least 3 neurons and if the dataset (x i , y i ) is decent, then θ is not a global minimum.

Proof. See Appendix B.3.

4. EMPIRICAL EXAMPLES:

We start our examples with experiments using MNIST dataset which show that bad initialization can lead to significant obstacles in the training process.

4.1. BAD INITIALIZATION ON MNIST:

Figure 3 shows the training error of rectified MLP on the MNIST dataset for different seeds and different model sizes. The learning algorithm used is Adam (Kingma & Ba, 2014) and everything except initialization, when specifically stated, follows an accepted protocol (see Appendix A). The results show that models that are not initialized in a good interval do not seem to converge to a good solution of the problem even after 1,000,000 updates. Depth does not seem to be able to resolve the bad initialization of the model. The bottom row experiments are similar to those presented in Zhang et al. (2017) , though more limited in their scope. They explore the correlation between the structure in the data and learning, and, at least in appearance, they do not seem to support our working hypothesis that the structure is essential. It is worth noticing though that the initialization is even more important in that setting; destroying the structure makes the model significantly more susceptible to bad initializations than when trained on the data with unpermuted labels (second column of Figure 3 , the network requires at least 400 units to be able to achieve 0 training error). ,000,000 updates. Each point is a single neural net (blue triangles -5 layer models with number of hidden units in each layer, red triangles -2 layer models with same number of hidden units in each layer). The title of each column shows the distribution used to initialize weights (w) and biases (b). Top row shows results on MNIST, bottom row shows results when the labels of MNIST had been randomly permuted. The number of hidden units per layer is indicated on x-axis.

The bad initializations used in these experiments are meant to target the blind spots of the rectifiers.

The main idea is that by changing the initialization of the model (the mean of the normal distribution used to sample weights) one can force all hidden units to be deactivated for the most or for all examples in the training set. This prevents said examples from being learned, even though the task might be linearly separable. The construction may seem contrived, but it has important theoretical consequences. It shows that one can not prove well behaved learning for finite sized neural networks applied to finite sized data, without taking into account the initialization or data. We formalize this idea in the Proposition 4, making the observation that the effect can be achieved by either changing the initialization of the model, or the data. In particular, by introducing specific outliers, one can force most of the data examples in the dataset to be in the blind spot of the neural network, despite being whitened.

Details of the experimental setup are given in Appendix A. The results presented in Figure 3 (bottom row), suggest that (from an optimization perspective) the important relationship is not only the one between the inputs and targets, but also between the inputs and the way the model partitions the input space (in here we focus on rectifier models which are, from a mathematical perspective, piece-wise linear functions). To empirically test if this is a viable hypothesis we consider the MNIST dataset, where we scale the inputs by a factor τ . The intuition is not to force the datasets into the blind spot of the model, but rather to concentrate most of the datapoints in very few linear regions (given by the initialization of the MLP). While these results do not necessarily point towards the model being locked in a bad minimum, they suggest that learning becomes less well behaved (see Fig. 4 ).

Additional results on a simple Zig-Zag regression task are given Figure 5 . The dataset itself is in the left panel, the results are visualized in the right panel. Similarly as in the MNIST case, the experiments suggest that as data becomes more concentrated in the same linear regions (of the freshly initialized model) the learning becomes really hard, even if the model has close to 3000 units. 

4.2. THE JELLYFISH -SUBOPTIMAL MODELS IN CLASSIFICATION USING RELU AND:



SIGMOIDS:

To improve our understanding of learning dynamics beyond exploiting blind spots, we look at one of the most theoretically well-studied datasets, the XOR problem. We analyze the dataset using a single hidden layer network (with either ReLU or sigmoid units).

A first observation is that while SGD can solve the task with only 2 hidden units, full batch methods do not always succeed. Replacing gradient descent with more aggressive optimizers like Adam does not seem to help, but rather tends to make it more likely to get stuck in suboptimal solutions (Table 1 ). By exploiting observations made in the failure modes observed for the XOR problem, we were able to construct a similar dataset, the Jellyfish, that results in suboptimal learning dynamics. is formed of four datapoints, where the positive class is given by [1.0, 0.0], [0.2, 0.6] and the negative one by [0.0, 1.0], [0.6, 0.2]. The datapoints can be seen in the Figure 6 .

Compared to the XOR problem it seems the Jellyfish problem poses even more issues, especially for ReLU units, where with 4 hidden units one still only gets 2 out of 3 runs to end with 0 training error (when using GD). One particular observation (see Figure 6 ) is that in contrast with good solutions, when the model fails on this dataset, its behaviour close to the datapoints is almost linear. We argue hence, that the failure mode might come from having most datapoints concentrated in the same linear region of the model (in ReLU case), hence forcing the model to suboptimally fit these points.

5. DISCUSSION:

Previous results (Dauphin et al., 2013; Saxe et al., 2014; Choromanska et al., 2015) provide insightful description of the error surface of deep models under general assumptions divorced from the specifics of the architecture. While such analysis is very valuable not only for building up the intuition but also for the development of the tools for studying neural networks, it only provides one facade of the problem. In this work we move from the generic to the specific. We show that for finite sized models/finite sized datasets one does not have a globally good behaviour of learning regardless of the model size (and even of the ratio of model size to the dataset size).

The overwhelming amount of empirical evidence points towards learning being well behaved in practice. We argue that the way to reconcile these observations is to show that the well-behaved learning dynamics are local and conditioned on the data structure, initialization and perhaps on other architectural choices. One can imagine a continuum ranging from the very specific, where every detail of the setup is important to attain good learning dynamics, to the generic, where learning is globally well behaved regardless of dataset or initialization. We believe that an important step forward in the theoretical study of the neural networks can be made by identifying where this class of models falls on that continuum. In particular, what are the most generic sets of constraints that need to be respected in order to attain the good behaviour. Our results focus on constructing counterexamples which result in a bad learning dynamics. While this does not lead directly to sufficient conditions for well-behaved systems, we hope that by carving out the space of possible conditions we are moving forward towards that goal.

Similar to Lin & Tegmark (2016) we put forward a hypothesis that the learning is only well behaved conditioned on the structure of the data. We point out, that for the purpose of learning, this structure can not be divorced from the particular initialization of the model. We postulate that learning becomes difficult if the data is structured such that there exist regions with a high density of datapoints (that belong to different classes) and the initialization results in models that assign these points to very few linear regions. While constraining the density per region alone might not be sufficient, it can provide a good starting point to understand learning for rectifier models. Another interesting question arising in that regard is what are the consequences on overfitting for enforcing a relatively low density of points per linear regions? Understanding of the structure of the error surface is an extremely challenging problem. We believe that as such, in agreement with a scientific tradition, it should be approached by gradually building up a related knowledge base, both by trying to obtain positive results (possibly under weakened assumptions, as it was done so far) and by studying the obstacles and limitations arising in concrete examples.

A EXPERIMENTAL DETAILS:

For the experiments depicted in Figure 3 we used MNIST dataset with data divided by 255 and one-hot encoded labels and we:

• ran 100 jobs, with the number of hidden units h sampled from [100, 2000] jointly for all the hidden layers, meaning that the model was 784 -h -h -... -h -10,

• used Adam as an optimizer, with learning rate of 1e -4 (and other arguments default),

• each job was ran for 1,000,000 updates,

• batch size used was 200.

For the experiments depicted in Figure 4 we used MNIST dataset with data divided by 255 and one-hot encoded labels and we:

• ran 400 jobs, with the number of hidden units h sampled from [100, 2000] for MNIST experiments and [100, 3000] for the Zig-Zag problem,

• we used Adam as an optimizer with learning rate randomly sampled to be either 1e -4 or 1e -3 (and other arguments default),

• each job was ran for 2,000,000 updates for the Zig-Zag problem and 1,200,000 updates in the MNIST case,

• batch size used was 10 for the Zig-Zag problem (dataset size is 20 points) and 50 for MNIST experiment.

• Initialization of weights and biases was from a Gaussian with 0 mean and standard deviation 1 √ fanin .

B PROOFS B.1 PROOF OF PROPOSITION 4:

For simplicity, let us consider a network with only one hidden layer of h units and a binary classification task, implying we have a single output neuron. Analogous result holds for any number of classes and for any arbitrary depth of the network as well. We use the following notation: h(x i ) is a vector of activations of hidden units when presented with i-th training sample, M(x i ) is the activation of the output neuron given the same sample, W is matrix of hidden weights, b is a vector of biases in the hidden layer and finally v, c are the weights and bias in the output layer. The whole classification becomes M(x i ) = vReLU(Wx i + b) + c. Let us now consider a dataset where N -1 points have all features negative, and a single point, denoted x i * (with positive label) which has all the features positive. We can always find such points that their coordinate-wise mean is equal to 0 and their standard deviation equals 1, since we can place all N -1 points very close to the origin, and the point x i * arbitrary far away in the positive part of the input space. Our dataset is therefore normalized (whitened) and it can have a full rank covariance matrix (since the construction does not depend on nothing besides signs of the features). We want to compute

P (∀ i =i * h(x i ) = 0 ∧ h(x i * ) > 0)

. Since, by construction ∀ i =i * x i < 0 and x i * > 0, it is also true that if all the weights W are nonnegative (and at least one is positive), then all the activations (after ReLU) of the hidden units will be 0 besides the one positive activation of h(x i * ) which comes directly from the assumption that biases are initialized to 0 1 . Consequently

P (∀ i =i * h(x i ) = 0 ∧ h(x i * ) > 0) ≥ P (W > 0),

and given that weights initializations are independent samples from the same distribution we get

P (W > 0) = dh i=1 ∞ 0 N (µ, σ 2 ) = ∞ 0 N (µ, σ 2 ) dh ,

where µ, σ are parameters of the distribution we use to initialize weights and d is input space dimensionality. All that is left is to show that during any gradient based optimization these weights will not be corrected, which requires one more assumption -that the output weights are positive as well. If this is true, then ∀ i =i * M(x i ) = 0 (again using that the output bias is zero) and M(x i * ) > 0 and of course P (v > 0) = ( ∞ 0 N (µ, σ 2 )) h . Now we have a fully initialized model which maps all the samples to 0, and one positive sample to some positive value. Consequently, given the fact that we use log loss, there holds ∂L/∂v k > 0 and ∂L/∂w kl ≥ 0 for all k, l. Indeed, since these changes are all increasing the probability of good classification of x i * and all remaining points are in inactive part of ReLUs thus they cannot contribute to partial derivatives. Therefore, during any gradient based (as well as stochastic and mini-batch based) learning the projection of samples mapped to 0 will not change, and the projection of x i * will grow to infinity (so the sigmoid approaches 1). Consequently we constructed an initialization scheme which with probability at least

EQUATION

gives the initial conditions of the net, where despite learning with log loss, we always classify all, arbitrary labeled N -1 points to the same label (since they are all mapped to the same output value) and we classify the unique positive sample with the valid label. Furthermore -the optimization never finishes, and there exists a network with a better accuracy, as we can label N -1 points in any manner, including making it linearly separable by labeling according to the first feature only.

In order to generalize to arbitrary number of layers, we would similarly force all the weights to be positive, thus with parametrization of k-layer deep network θ = (W n ) k n=1 we would simply get

EQUATION

We can still construct similar proof for biases taken from normal distributions as well.

and finally, if the biases are not zero, but have some some arbitrary values (fixed or sampled) we simply adjust the size of the weights accordingly. Instead of having them bigger than 0 we would compute probability of having them big enough to make the whole first layer to produce 0s for every point besides x i * and analogously for the remaining layers.

Furthermore, it is possible to significantly increase the probability of failure if we do not mind the situation in which the learning process is not even starting. Proposition 5 addresses this situation.

B.2 PROOF OF PROPOSITION 5

Proof. Let us notice, that since we are using ReLU activations, the activations on jth hidden layer h j are non-negative. Consequently, if the following layer has only negative weights and non-positive biases, then h j+1 ≡ b j+1 (as all the ReLUs are inactive), so the network has to output exactly the same value for every single point. Furthermore, during the gradient based learning we will never change those weights as gradients going through ReLUs will be equal to zero.

Let us now consider a deep rectifier network with k hidden layers with h neurons each. If only k > 2 we can use the above observation to compute for every j > 1:

P (All neurons inactive in hidden layer j) ≥ P (b j ≤ 0, W j < 0) = P (b j ≤ 0)P (W j < 0), and consequently, due to the assumptions about expected values P (Learning fails with k hidden layers) ≥ P (∃ j>1 All neurons inactive in hidden layer j)

≥ P (∃ j>1 ∀ l,i b jl ≤ 0 ∧ W ji < 0) = 1 -P (∀ j>1 ∃ l b jl > 0 ∨ ∃ i W ji ≥ 0) = 1 - k-1 j=1 P (∃ l b jl > 0 ∨ ∃ i W ji ≥ 0)

As every layer has the same number h of neurons,the values (∃ l b jl > 0 ∨ ∃ i W ji ≥ 0) are equal for every j. Therefore

1 - k-1 j=1 P (∃ l b jl > 0 ∨ ∃ i W ji ≥ 0) = 1 -P (∃ l b 1l > 0 ∨ ∃ i W 1i ≥ 0) k-1 = 1 -(1 -P (∀ i b 1i ≤ 0 ∧ W 1i < 0)) k-1 = 1 -[1 -cdf b (0) h cdf W (0) h 2 ] k-1 .

Due to assumptions about distributions of biases and weights we know that

0 < cdf W (0) < 1, 0 ≤ cdf b (0) < 1 so lim k→∞ 1 -[1 -cdf b (0) h cdf W (0) h 2 ] k-1 = 1.

B.3 PROOF OF THEOREM 1:

Claim i) is a direct consequence of Corollary 1. It remains to prove ii). For that it is sufficient to show an example of a set of weighs θ = (( Ŵn ) k n=1 , ( bn ) k n=1 ) such that L((W n ) k n=1 , (b n ) k n=1 ) > L(( Ŵn ) k n=1 , ( bn ) k n=1 ). Let r be such that M ({y p : x p = x r }) = M ({y p : p = 1, . . . , N }). Such point exists by assumption that the dataset is decent. Let H be a hyperplane passing through x r such that none of the points x s = x r lies on H. Then there exists a vector v such that |v T (x s -x r )| > 2 for all x s = x r . Let γ = v T x r . We define W 1 in such a way that the first row of W 1 is v , the second row is 2v and the third one is v again, and if the first layer has more than 3 neurons, we put all the remaining rows of W 1 to be equal zero. We choose the first three biases of b 1 to be -γ + 1, -2γ and -γ -1 respectively. We denote µ = M ({y p : x p = x r }) and ν = M ({y p : x p = x r }). We then choose W 2 to be a matrix whose first row is (ν -µ, µ -ν, ν -µ, 0, . . . , 0) and the other

:



Figures and Tables:

One could ask whether blind spots are the only reasons for bad behaviour of rectifier nets. The answer is actually negative, and as following examples show -they can be completely absent in local optima, at the same time exisiting in a global solution! (a) Two local minima for 1 hidden neuron. (b) Two local minima for 2 hidden neurons. (c) Two local minima for 3 hidden neurons.

Figure 1: Local minima for ReLU-based regression. Both lines represent local optima, where the blue one is better than the red one. Proposition 2. Let us consider a dataset D 2 with d = 1, given by points (x 1 , y 1 ) = (-1, 5), (x 2 , y 2 ) = (0, 0), (x 3 , y 3 ) = (1, -1), (x 4 , y 4 ) = (10, -3), (x 5 , y 5 ) = (11, -4), (x 6 , y 6 ) = (12, -5) (Figure 1 (b)). Then, for a rectifier network with m = 2 hidden units and a squared error loss the set of weights w = (-5, -1), b = (1, -8), v = (1, -1), c = -1 is a global minimum (with perfect fit) and the set of weights w = (-3, -1), b = (4 + 1 3 , -10), v = (1, -1), c = -3 is a suboptimal local minimum.

Figure 2: On the left: exemplary dataset constructed in Proposition 4, color denotes label. Two middle ones: how big the mean of the normal distribution N (µ, σ 2 ) has to be in order to have at least 99% probability of the effect (very bad local minima) described in the Proposition 4, as a function of number of hidden units in 2 -h -... -h -1 classification network. By LeCun'98 initialization we mean taking weights from N (µ, 1 h ) and by Xavier'10 from N (µ, 2 hin+hout ). In both cases the original papers used µ = 0. Rightmost one: Proposition 5, probability of learning failing with increasing number of layers when the initialization is fully correct.

Figure3: Plots of final training accuracy on MNIST dataset after 1,000,000 updates. Each point is a single neural net (blue triangles -5 layer models with number of hidden units in each layer, red triangles -2 layer models with same number of hidden units in each layer). The title of each column shows the distribution used to initialize weights (w) and biases (b). Top row shows results on MNIST, bottom row shows results when the labels of MNIST had been randomly permuted. The number of hidden units per layer is indicated on x-axis.

Figure 4: Plots of the final train accuracy on scaled MNIST dataset after 1,200,000 updates of a single hidden layer neural net. The title of each column shows the scaling factor applied to the data.

Figure 5: Plots of training MSE error on the Zig-Zag regression task after 2,000,000 updates. See caption of Figure 4 for more details. The left panel depicts the Zig-Zag regression task with three found solutions for τ = 0.01. The actual datapoints are shown by the diamond shaped dots.

(a) Optimally converged net for Jellyfish.(b) Stuck net for Jellyfish.

Figure 6: Examples of different outcomes of learning on the Jellyfish dataset.

Peer review (iclr2017):
The paper studies some special cases of neural networks and datasets where optimization fails. Most of the considered models and datasets are however highly constructed and do not follow the basic hyperparameters selection and parameter initialization heuristics. This reduces the practical relevance of the analysis. The experiment *bad initialization on MNIST* shows that for very negative biases or weights drawn from a non-centered distribution, all ReLU activations are *off* for all data points, and thus, optimization is prevented. This never occurs in practice, because using proper initialization heuristics avoid these cases. The *jellyfish* dataset constructed by the authors is demonstrated to be difficult to fit by a small model. However, the size/depth of the considered model is unsuitable for this problem. Proposition 4 assumes that we can choose the mean from which the weight parameters are initialized. This is typically not the case in practice as most initialization heuristics draw weight parameters from a distribution with mean 0. Proposition 5 considers infinitely deep ReLU networks. Very deep networks would however preferably be of type ResNet.