Title: AN EMPIRICAL ANALYSIS OF DEEP NETWORK LOSS SURFACES

Abstract: The training of deep neural networks is a high-dimension optimization problem with respect to the loss function of a model. Unfortunately, these functions are of high dimension and non-convex and hence difficult to characterize. In this paper, we empirically investigate the geometry of the loss functions for state-of-the-art networks with multiple stochastic optimization methods. We do this through several experiments that are visualized on polygons to understand how and when these stochastic optimization methods find local minima.

1. INTRODUCTION:

Deep neural networks are trained by optimizing an extremely high-dimensional loss function with respect to the weights of the network's linear layers. The objective function minimized is some measure of the error of the network's predictions based on these weights compared to training data. This loss function is non-convex and has many local minima. These loss functions are usually minimized using first-order gradient descent (Robbins & Monro, 1951; Polyak, 1964) algorithms such as stochastic gradient descent (SGD) (Bottou, 1991) . The success of deep learning critically depends on how well we can minimize this loss function, both in terms of the quality of the local minima found and the time to find them. Understanding the geometry of this loss function and how well optimization algorithms can find good local minima is thus of vital importance.

Several works have theoretically analyzed and characterized the geometry of deep network loss functions. However, to make these analyses tractible, they have relied on simplifications of the network structures, including that the networks are linear (Saxe et al., 2014) , or assuming the path and variable independence of the neural networks (Choromanska et al., 2015) . Orthogonally, the performance of various gradient descent algorithms has been theoretically characterized (Nesterov, 1983) . Again, these analyses make simplifying assumptions, in particular that the loss function is strictly convex, i.e. there is only a single local minimum.

In this work, we empirically investigated the geometry of the real loss functions for state-of-the-art networks and data sets. In addition, we investigated how popular optimization algorithms interact with these real loss surfaces. To do this, we plotted low-dimensional projections of the loss function in subspaces chosen to investigate properties of the local minima selected by different algorithms. We chose these subspaces to address the following questions:

• What types of changes to the optimization procedure result in different local minima?

• Do different optimization algorithms find qualitatively different types of local minima?

2. RELATED WORK:



2.1. LOSS SURFACES:

There have been several attempts to understand the loss surfaces of deep neural networks. Some have studied the critical points of the deep linear neural networks (Baldi, 1989; Baldi & Hornik, 1989; Baldi & Lu, 2012) . Others further investigated the learning dynamics of the deep linear neural networks (Saxe et al., 2014) . More recently, several others have attempted to study the loss surfaces of deep non-linear neural networks (Choromanska et al., 2015; Kawaguchi, 2016; Soudry & Carmon, 2016) .

One approach is to analogize the states of neurons as the magnetics dipoles used in spherical spinglass Ising models from statistical physics (Parisi, 2016; Fyodorov & Williams, 2007; Bray & Dean, 2007) . Choromanska et al. (2015) attempted to understand the loss function of neural networks through studying the random Gaussian error functions of Ising models. Recent results (Kawaguchi, 2016; Soudry & Carmon, 2016) have provided cursory evidence in agreement with the theory provided by Choromanska et al. (2015) in that they found that that there are no "poor" local minima in neural networks still with strong assumptions.

There is some potential disconnect between these theoretical results and what is found in practice due to several strong assumptions such as the activation of the hidden units and output being independent of the previous hidden units and input data. The work of Dauphin et al. (2014) empirically investigated properties of the critical points of neural network loss functions and demonstrated that their critical points behave similarly to the critical points of random Gaussian error functions in high dimensional space. We will expose further evidence along this trajectory.

2.2. OPTIMIZATION:

In practice, the local minima of deep network loss functions are for the most part decent. This implies that we probably do not need to take many precautions to avoid bad local minima in practice. If all local minima are decent, then the task of finding a decent local minimum quickly is reduced to the task of finding any local minimum quickly. From an optimization perspective this implies that solely focusing on designing fast methods are of key importance for training deep networks.

In the literature the common method for measuring performance of optimization methods is to analyze them on nice convex quadratic functions (Polyak, 1964; Broyden, 1970; Nesterov, 1983; Martens, 2010; Erdogdu & Montanari, 2015) even though the problems are applied to non-convex problems. For non-convex problems, however, if two methods converge to different local minima their performance will be dictated on how those methods solve those two convex subproblems. It is challenging to show that one method will beat another without knowledge of the sort of convex subproblems, which is generally not known apriori. What we will explore is whether indeed are some characteristics that can found experimentally. If so, perhaps one could validate where these analytical results are valid or even improve methods for training neural networks.

2.2.1. LEARNING PHASES:



Slowing decaying Fast decaying:



Figure 1: An example of learning curve of neural network:

One of the interesting empirical observation is that we often observe is that the incremental improvement of optimization methods decreases rapidly even in non-convex problems. This behavior has been discussed as a "transient" phase followed by a "minimization" phase (Sutskever et al., 2013) where the former finds the neighborhood of a decent local minima and the latter finds the local minima within that neighborhood. The existence of these phases implies that if certain methods are better at different phases one could create novel methods that schedule when to apply each method.

3. EXPERIMENTAL SETUP AND TOOLS:



3.1. NETWORK ARCHITECTURES AND DATA SETS:

We conducted experiments on three state-of-the-art neural network architectures. Network-in-Network (NIN) (Lin et al., 2014) and the VGG (Simonyan & Zisserman, 2015) network are feedforward convolutional networks developed for image classification, and have excellent performance on the Imagenet (Russakovsky et al., 2014) and CIFAR10 (Krizhevsky, 2009) data sets. The long short-term memory network (LSTM) (Hochreiter & Schmidhuber, 1997 ) is a recurrent neural network that has been successful in tasks that take variable-length sequences as input and/or produce variable-length sequences as output, such as speech recognition and image caption generation. These are large networks currently used in many machine vision and learning tasks, and the loss functions minimized by each are highly non-convex.

All results using the feed-forward convolutional networks (NIN and VGG) are on the CIFAR10 image classification data set, while the LSTM was tested on the Penn Treebank next-word prediction data set.

3.2. OPTIMIZATION METHODS:

We analyzed the performance of five popular gradient-descent optimization methods for these learning frameworks: Stochastic gradient descent (SGD) (Robbins & Monro, 1951) , stochastic gradient descent with momentum (SGDM), RMSprop (Tieleman & Hinton, 2012) , Adadelta (Zeiler et al., 2011) , and ADAM (Kingma & Ba, 2014) . These are all first-order gradient descent algorithms that estimate the gradients based on randomly-grouped minibatches of training examples. One of the major differences between these algorithms is how they select the weight-update step-size at each iteration, with SGD and SGDM using fixed schedules, and RMSprop, Adadelta, and ADAM using adaptive, per-parameter step-sizes. Details are provided in Section A.2.

In addition to these five existing optimization methods, we compare to a new gradient descent method we developed based on the family of Runge Kutta integrators. In our experiments, we tested a second-order Runge-Kutta integrator in combination with SGD (RK2) and in combination with ADAM (ADAM&RK2). Details are provided in Section A.3).

3.3. ANALYSIS METHODS:

Several of our empirical analyses are based on the technique of Goodfellow et al. (Goodfellow et al., 2015) . They visualize the loss function by projecting it down to one carefully chosen dimension. They plot the value of the loss function along a set of samples along this dimension. The projection space is chosen based on important weight configurations, thus they plot the value of the loss function at linear interpolations between two weight configurations. They perform two such analyses: one in which they interpolate between the initialization weights and the final learned weights, and one in which they interpolate between two sets of final weights, each learned from different initializations.

In this work, we use a similar visualization technique, but choose different low-dimensional subspaces for the projection of the loss function. These subspaces are based on the initial weights as well as the final weights learned using the different optimization algorithms and combinations of them, and are chosen to answer a variety of questions about the loss function and how the different optimization algorithms interact with this loss function. In contrast, Goodfellow et al. only looked at SGDM. In addition, we explore the use of two-dimensional projections of the loss function, allowing us to better visualize the space between local minima. We do this via barycentric and bilinar interpolation for triplets and quartets of points respectively (details in Section A.1).

We refer to the critical points found using these variants of SGD, for which the gradient is approximately 0, as local minima. Our evidence that these are local minima as opposed to saddle points is similar to that presented in Goodfellow et al. (Goodfellow et al., 2015) . If we interpolate beyond the critical point, in this one-dimensional projection, the loss increases (Fig. 10 ).

3.4. TECHNICAL DETAILS:

We used the VGG and NIN implementations from https://github.com/szagoruyko/cifar.torch.git.

The batch size was set to 128 and the number of epochs was set to 200. The learning rate was chosen from the discrete range between [0.2, 0.1, 0.05, 0.01] for SGD and [0.002, 0.001, 0.0005, 0.0001] for adaptive learning methods. We doubled the learning rates when we ran our augmented versions with Runge-Kutta because they required two stochastic gradient computations per epoch. We used batchnormalization and dropout to regularize our networks. All experiments were run on a 6-core Intel(R) Xeon(R) CPU @ 2.40GHz with a TITAN X.

4. EXPERIMENTAL RESULTS:



4.1. DIFFERENT OPTIMIZATION METHODS FIND DIFFERENT LOCAL MINIMA:

We trained the neural networks described above using each optimization method starting from the same initial weights and with the same minibatching. We computed the value of the loss function for weight vectors interpolated between the initial weights, the final weights for one algorithm, and the final weights for a second algorithm for several pairings of algorithms. The results are shown in the lower triangle of Table 1 .

For every pair of optimization algorithms, we observe that the training loss between the final weights for different algorithms shows a sharp increase along the interpolated path. This suggests that each optimization algorithm found a different critical point, despite starting at the same initialization. We investigated the space between other triplets and quadruples of weight vectors (Figure 2 and 3), and even in these projections of the loss function, we still see that the local minima returned by different algorithms are separated by high loss weight parameters.

Deep networks are overparameterized. For example, if we switch all corresponding weights for a pair of nodes in our network, we will obtain effectively the same network, with both the original and permuted networks outputting the same prediction for a given input. To ensure that the weight vectors returned by the different algorithms were functionally different, we compared the outpts of the networks on each example in a validation data set:

dist(θ 1 , θ 2 ) = 1 N test Ntest i=1 F (x i , θ 1 ) -F (x i , θ 2 ) 2 ,

where θ 1 and θ 2 are the weights learned by two different optimization algorithms, x i is the input for a validation example, and F (x, θ) is the output of the network for weights θ on input x. We found that, for all pairs of algorithms, the average distance between the outputs of the networks (Equation 4.1) was approximately 0.16, corresponding to a label disagreement of about 8% (upper triangle of Table 1 ). Given the generalization error of these networks (approximately 11%, Figure 4 ), the maximum disagreement we could see was 22%. Thus, these networks disagreed on a large fraction of these test examples -over 1 3 rd. Thus, the local minima found by different algorithms correspond to effectively different networks, not trivial reparameterizations of the same one. 

4.2. DIFFERENT OPTIMIZATION ALGORITHMS FIND DIFFERENT TYPES OF LOCAL MINIMA:

Next, we investigated whether the local minima found by the different optimization algorithms had distinguishing properties. To do this, we trained the networks with each optimization algorithm using different initial parameters. We then compared differences between runs of the same algorithm but different initializations to differences between different algorithms.

As shown in Figure 4 (a), in terms of training accuracy, we do see some stereotypy for the optima found by different algorithms, with SGD finding local minima with the lowest training accuracy and ADAM, Rmsprop, and Adadelta finding local minima with the highest training accuracy. However, this could be attributed to SGD's asymtotically slow convergence near local minima due to the gradient diminishing near extrema. Despite this limitation, Figure 4 (b) shows that the generalization accuracy of these different local minima on validation data was not significantly different between algorithms. We also did not see a relationship between the weight initialization and the validation accuracy. Thus, while these algorithms fall into different local minima, they are not different in terms of their final quality.

We visualized the loss surface around each of the local minima for the multiple runs. To do this, we plotted the value of the loss function between the initial and final weights for each algorithm (Figure 5(a, c )) for each run of the algorithm from a different initialization. In addition, we plotted the value of the loss function between the final weights for selected pairs of algorithms for each run (Figure 5(b, d )). We see that the surfaces look strikingly similar for different runs of the same algorithm, but characteristically different for different algorithms. Thus, we found evidence that the different algorithms land in qualitatively different types of local minima.

In particular, we see in Figure 5(a, c ) that the size of the basins around the local minima found by ADAM and ADAM&RK2 are larger than those found by SGD and RK2, i.e. that the training loss is small for a wider range of α values. This is a relative measure, and the magnitude of the change in the weight vector is ∆α θ 1 -θ 0 for a change of size ∆α, where θ 0 is the initial weight vector θ 1 is the result found by a given optimization algorithm. In Figure 6 , we repeat this analysis, instead showing the loss as a function of the absolute distance in parameter space:

θ(λ) = θ 1 + λ θ 0 -θ 1 θ 0 -θ 1 (1)

We again see that the size of the basin around the local minima varies by optimization algorithm. Note that we evaluate the loss for weight vectors beyond the initial configuration, which had a loss of 2.4.

4.3. ANALYZING LEARNING AFTER "TRANSIENT PERIOD":

Recall that, during optimization, it has been observed that there is a short "transient" phase when the loss decreases rapidly and a "minimization" phase in which the loss decreases slowly (Section 2.2.1 and Figure 1 ). In this set of experiments, we investigated the effects of switching from one type of optimization method to another at various points during training, in particular at late stages of training when it is thought that a local minimum has been chosen and is only being localized. We switched from one optimization method to another 25%, 50%, and 75% of the way through training.

The results are plotted in Figure 7d . We emphasize that we are not switching methods to improve performance, but rather to investigate the shape of the loss function in regions explored during the "minimization" phase of optimization.

We found that, regardless of how late we switch optimization algorithms, as shown in the rightmost column of Figure 7 , the local minima found were all different. This directly disagrees with the notion that the local minimum has effectively been chosen before the "minimization" phase, but instead that which local minimum is found is still in flux this late in optimization. It appears that this switch from one local minimum to another happens almost immediately after the optimization method switches, with the training accuracy jumping to the characteristic accuracy for the given method within a few epochs (Figure 7 , left column). Interestingly, we also see the distance between the initial and current weight vectors changes drastically after switching from one optimization (c) The learning rate is set to 0.001 for ADAM , and then switched to Adadelta (learning rate is not required). method to another, and that this distance is characteristic per algorithm (Figure 7 , middle column). While distance increases with training epoch for any single optimization method, it actually starts to decrease when switching from ADAM to SGD.

4.4. EFFECTS OF BATCH-NORMALIZATION:

To understand how batch normalization affects the types of local minima found, we performed a set of experiments comparing loss surfaces near local minima found with and without batch normal-ization for each of the optimization methods. We visualized the surface near these local minima by interpolating between the initial weights and the final weights as well as between pairs of final weights found with different algorithms.

We observed clear qualitative differences between optimization with (Figure 5 ) and without (Figure 8 ) batch normalization. We see that, without batch normalization, the quality of local minimum found by a given algorithm is much more dependent on the initialization. In addition, the surfaces between different local minima are more complex in appearance: with batch normalization we see sharp unimodal jumps in performance but without batch normalization we obtain wide bumpy shapes that aren't necessarily unimodal.

The neural networks are typically initialized with very small parameter values (Glorot & Bengio, 2010; He et al., 2015) . Instead, we trained NIN with exotic intializations such as initial parameters drawn from N (-10.0, 0.01) or N (-1.0, 1.0) and observe the loss surface behaviours. The details of results are discussed in Appendix A.5.

5. CONCLUSIONS:

In this work, we performed a series of empirical analyses to understand the geometry of the loss functions corresponding to deep neural networks, and how different optimization methods minimize this loss to answer the two questions posed in the introduction.

What types of changes to the optimization procedure result in different local minima?:

We found that every type of change to the optimization procedure we tested resulted in a different local minimum. Different local minima were found using the different optimization algorithms from the same initialization (Section 4.1). Even switching the optimization algorithm to another very late in optimization -during the slow "mimimization" portion of learning -resulted in a different local minimum (Section 4.3). The quality of the local minima found, in terms of training and generalization error, is similar. These different local minima were not equivalent, and made mistakes on different test examples (Section 4.1). Thus, they were not trivially different local minima, as would occur if nodes in internal layers of the network were permuted. We observed that the quality of these local minima was only consistently good when we used batch normalization for regularization. Without batch normalization, the quality of the critical points found depended on the initialization, and some solutions found were not as good as others. Our observations are in contrast to the conclusions of Goodfellow et al., i.e . that local minima are not a problem in deep learning because, in the region of the loss function explored by SGD algorithms, the loss function is well-behaved (Goodfellow et al., 2015) . Instead, our observations are more consistent with the explanation that the local minima found by popular SGD optimization methods are almost all good (Choromanska et al., 2015; Kawaguchi, 2016; Soudry & Carmon, 2016) .

Do different optimization algorithms find qualitatively different types of local minima?:

Interestingly, we found that, while the local minima found by the same optimization algorithm from different initializations were different, the shape of the loss function around these local minima was strikingly similar, and was a characteristic of the optimization algorithm. In particular, we found that the size of the basin around ADAM-based optimization was larger than that around vanilla SGD (Section 4.2). A large basin is related to a large margin, as small changes in the weight vector will not affect the training error, and perhaps could have some implications for generalization error. In our experiments, however, we did not observe better generalization error for ADAM than SGD. Questions for potential future research are why the shapes of the loss functions around different local minima found by the same algorithm are so similar, and what the practical implications of this are.

where F t is some value changing over time and E[F ] t is the averaged quantity.

An easy scheme to apply this method to is to compute a rolling weighted average of gradients such as E[g] t = (1 -α)g t + αE[g] t-1 but there will be other uses in the future.

A.2.3 PERTINENT METHODS:

With the aforementioned tools there are a variety of methods that can be constructed. We choose to view these algorithms as implementations of Explicit Euler on a variety of different vector fields to remove the ambiguity between η and g t . We therefore can define a method by the vector field X t that explicit Euler is applied to with a single η that is never changed.

SGD with Momentum (SGDM) By simply applying momentum to g t one obtains this stabilized stochastic version of gradient descent:

X t = -E[g] t .

(12) This is the most fundamental method that is used in practice and the basis for everything that follows.

Adagrad Adagrad rescales X t by summing up the sqaures of all previous gradients in a coefficientwise fashion:

EQUATION

)

Here is simply set to some small positive value to prevent division-by-zero. In the future we will neglect this term in denominators because it is always necessary.

The concept is to accentuate variations in g t , but because the denominator is monotonically nondecreasing over time this method is doomed to retard its own progress over time. The denominator can also be seen as a form of momentum where α and β are both set to 1.

Rmsprop A simple generalization of ADAGrad is to simply allow for α and β to be changed from 1. In particular one usually chooses a β less than 1, and presumably α = 1 -β. Thus one arrives at a method where the effects of the distance history are diminished:

EQUATION

)

Adadelta Adadelta adds another term to RMSprop in order to guarantee that the magnitude of X is balanced with g t Zeiler et al. (2011) . More precisely it maintains

EQUATION

which results in the following vector field:

EQUATION

)

and η is set to 1.

ADAM By applying momentum to both g t and g 2 t one arrives at what is called ADAM. This is often considered a combination of SGDM + RMSprop,

EQUATION

)

c t = √ 1-β t 2 1-β t 1

is the initialization bias correction term with β 1 , β 2 ∈ [0, 1) being the β parameters used in momentum for g and g 2 respectively. Initialization bias is caused by the history of the momentum variable being initially set to zero.

A.3 RUNGE KUTTA:

Runge-Kutta methods Butcher (1963) are a broad class of numerical integrators categorized by their truncation error. Because the ordinary differential equations Runge-Kutta methods solve generalize gradient descent, our augmentation is quite straightforward. Although our method applies to all explicit Runge-Kutta methods we will only describe second order methods for simplicity.

Table 3 : The coefficients of various second order Runge-Kutta methods Hairer et al. (1987) Method Name a 1 a 2 q 1 Midpoint 0 1

1 2 Heun 1 2 1 2 1 Ralston 1 3 2 3 3 4

The general form of second-order explicit Runge-Kutta on a time-independent vector field is

θ t+1 = θ t + (a 1 k 1 + a 2 k 2 )h (18) k 1 = X (θ t ) (19) k 2 = X (θ t + q 1 hk 1 )

(20) where a 1 , a 2 , and q 1 are parameters that define a given Runge-Kutta method. Table 3 refers to the parameters used for the different Runge-Kutta variants we use in our experiments.

A.3.1 AUGMENTING OPTIMIZATION WITH RUNGE KUTTA:

For a given timestep, explicit integrators can be seen as a morphism over vector fields X → X h . For a gradient g t = ∇ θ U we can solve a modified RK2 gradient ḡt in the following fashion:

EQUATION

= (a 1 k 1 + a 2 k 2 ).

(24) If we simply substitute the gradient g t with ḡt one obtains an RK2-augmented optimization technique.

A.4 EXPERIMENTS WITH RUNGE-KUTTA INTEGRATOR:

The results in Figure 9 illustrates that, with the exception of the Midpoint method, stochastic Runge-Kutta methods outperform SGD. "SGD x2" is the stochastic gradient descent with twice of the learning rate of "SGD". From the figure, we observe that the Runge-Kutta methods perform even better with half the number of gradient computed by SGD. The reason is because SGD has the accumulated truncated error of O(h) while second-order Runge-Kutta methods have the accumulated truncated error of O(h 2 ).

Unfortunately, ADAM outperforms ADAM+RK2 methods. We speculate that this is because the way how ADAM's renormalization of input gradients in conjunction with momentum eliminates the value added by using our RK-based descent directions.

A.5 EFFECTS OF BATCH-NORMALIZATION AND EXTREME INITIALIZATIONS:

The neural networks are typically initialized with very small parameter values (Glorot & Bengio, 2010; He et al., 2015) . Instead, we trained NIN with exotic intializations such as initial parameters drawn from N (-10.0, 0.01) or N (-1.0, 1.0) and observe the loss surface behaviours. The results are shown in Figure 11 . We can see that NIN without BN does not train at all with any of these initializations. Swirszcz et al. (2016) mentioned that bad performance of neural networks trained with these initializations are due to finding a bad local minima. However, we see that loss surface region around these initializations are plateau1 rather than a bad local minima as shown in Figure 11b . On the other hand, NIN with BN does train slowly over time but finds a local minima. This implies that BN redeems the ill-posed loss surface (plateau region). Nevertheless, the local minima it found was not good as when the parameters were initialized with small values. However, it is not totally clear whether this is due to difficulty of training or due to falling in a bad local minima. Loss A200->A100-S100 S200->S100-A100 S200->S50-A150 A200->A50-S150 S50-A150->S100-A100 A50-S150->A100-S100 (a) Learning rate is not required for Adadelta. Learning rate is set to 0.05 for SGD in the beginning, and then switched it to 0.1. Loss A200->A100-S100 S200->S100-A100 S200->S50-A150 A200->A50-S150 S50-A150->S100-A100 A50-S150->A100-S100 (a) Learning rate is not required for Adadelta. Learning rate is set to 0.05 for SGD in the beginning, and then switched it to 0.1.

Figure 15 : VGG -Switching methods from ADAM to Adadelta and Adadelta to ADAM at epoch 50 and 100. Zoomed in version (Left) . Distance between initial weights to weights at each epoch (Middle) . The interpolation between different convergence parameters (Right). Each figure shows the results of switching methods at different learning rate. We label the switch of methods in terms of ratio. For instance, S50-A50 as trained with SGD in the first 100 epoch and swithced to Adadelta for the rest of the epoch.

:

We used same initializations as(Swirszcz et al.,

2016) but we trained different neural networks with SGD on a different dataset. We usedNIN and CIFAR10 and Swirszcz et al. (2016) used smaller neural network and MNIST.

Figures and Tables:

Figure 2: Visualization of the loss surface at weights interpolated between two initial configurations and the final weight vectors learned using SGD from these initializations.

Figure 4: (a) Training accuracy and (b) test accuracy for each of the optimization methods. Colors correspond to different initializations.

(d) VGG -Final to Final.

Figure 5: Loss function value near local minima found by multiple restarts of each algorithm.

Figure 6: Observing the absolute size of basin for different local minimas found by different optimization methods.

The learning rate is set to 0.001 for ADAM , and then switched to SGD with learning rate 0.01. The learning rate is set to 0.1 for SGD, and then switched to ADAM with learning rate 0.0001.

The learning rate is not required for Adadelta, and then switched to ADAM with learning rate 0.0001.

Figure 7: Switching methods from one method to another method at epoch 50 and 100 and 150. Accuracy curve (Left two columns). Distance between initial weights to weights at each epoch (Middle). The interpolation between different convergence parameters (Right). For instance, S100-A100 as trained with SGD in the first 100 epoch and switched to ADAM for the rest of the epoch. (Best viewed in zoom)

Figure 8: (Without batch normalization) Loss function with parameter interporlated between Initial to Final. (Bottom) Loss function with parameters interporlated between Final to Final. Each column of the plots are from different initializations.

Figure 9: Training accuracy curve

Figure 11: NIN trained from different initializaitons.

Learning rate is set to 0.05 for SGD in the beginning, and then switched it to 0.01.

Figure 14: VGG -Switching methods from SGD to Adadelta and Adadelta to SGD at epoch 50 and 100. Zoomed in version (Left). Distance between initial weights to weights at each epoch (Middle).The interpolation between different convergence parameters (Right). Each figure shows the results of switching methods at different learning rate. We label the switch of methods in terms of ratio. For instance, S50-A50 as trained with SGD in the first 100 epoch and swithced to Adadelta for the rest of the epoch.

Visualization of the Loss Surface with and without batch-normalization.

Table content: <table/>

Peer review (iclr2017):
First of all, I would like to thank the authors for putting this much work into a necessary but somewhat tedious topic. While I think the paper is somewhat below the standard of a conference paper (see detailed comments below), I would definitely love to see a version of this paper published with some of the issues ironed out. I also agree with many of the points raised by other reviewers and will not repeat them here. Major points: -- *As we saw in the previous section, the minima of deep network loss functions are for the most part decent.* All you said in the previous section was that theory shows that there are no bad minima under *strong assumptions*. There is no practical proof that minima do not vary in quality. -- *This implies that we probably do not need to take many precautions to avoid bad minima in practice. If all minima are decent, then the task of finding a *decent minima quickly* is reduced to the task of finding any minima quickly.* First of all, as one of the reviewers pointed out, we are never guaranteed in practice to actually reach a local minimum. We could always hit a region of the objective function where the algorithm makes essentially no further progress. The final error level, in practice, actually does depend significantly on many factors such as (i) optimization algorithm (ii) learning rate schedule (iii) initialization of weights (iv) presence of unsupervised pretraining (v) whether neurons are added or eliminated during training etc. etc. Therefore, the task of optimizing neural networks is far from being *reduced to finding any minima quickly*. -- Figure 1 I don*t like Figure 1, because it suggests to me that you diagnosed exactly where the transition between the two phases happened, which I don*t think you did. Also, the concept of having a fast-decaying error followed by a slow-decaying error is simple enough for readers to understand without a dedicated graph. Minor point on presentation: The red brace is positioned lower in the figure than the blue brace and the braces don*t join up horizontally. Please be more careful. -- Misuse of the transient phase / minimization phase concept In section 4.3, you talk about the transient and minimization phase of optimization. However, you have no way of diagnosing when or if your algorithm reaches the minimization phase. You seem to think that the minimization phase is simply the part of the optimization process where the error decreases slowly. AFAIK, this is not the case. The minimization phase is where the optimization algorithm enters the vicinity of the local minimum that can be approximated by the second-order Taylor expansion. For this to even occur, one would have to verify, for example, that the learning rate is small enough. You change the algorithm after 25%, 50% and 75% of training, but these points seem arbitrary. What is the minimization phase was reached at 99%, or 10 epochs after you decided to stop training? -- Only 1 dataset You run most experiments on only 1 dataset (CIFAR). Please replicate with at least one more dataset. -- Many figures are unclear For each figure, the following information are relevant: network used; dataset used; learning rate used; batch norm yes / no; whether figure shows train, test, or validation error. It should be easy for the reader to ascertain this information for all figures, not just for some. -- You say at the beginning of section 4.1 that each algorithm finds a different minimum as if this is a significant finding. However, this is obvious because the updates taken by these algorithms vary wildly. Keep in mind that there is an exponentially large number of minima. The probability of different algorithms choosing the same minimum is essentially zero because of their sheer number. The same would be true if you even shift the learning rate slightly or use a different random seed for minibatch generation etc. etc. -- Lack of confidence intervals The value of Figures 1, 2, 3 and 6 is limited is because it is unclear how these plots would change if the random seed were changed. We only get information for a single weight initialization and a single minibatch sequence. While figures 5 and 7 can be used to try and infer what confidence intervals around plots in figures 1, 2, 3 and 6 might look like, I think those confidence intervals should still be shown for at least a subset of the configurations presented. -- Lack of information regarding learning rate There is big question mark left open regarding how all your results would change if different learning rates were used. You don*t even tell us how you chose the learning rates from the intervals you gave in section 3.4. -- Lack of information regarding the absolute distance of interpolated points In most figures, you interpolate between two or three trained weight configuration. However, you do not say how far the interpolated points are apart. This is highly significant, because if points are close together and there is a big *hump* between them, it means that those points are more *brittle* than if they are far apart and there is a big *hump* between them. Minor points: -- LSTM is not a fixed network architecture like NiN or VGG, but a layer type. LSTM would be equivalent to CNN. Also, the VGG paper has multiple versions of VGG. You should specify which one you used. -- The font size for the legends in the upper triangle of Table 1 is too small. You can*t just write *best viewed in zoom* in the table caption and pretend that somehow fixes the problem. Personally, I prefer no legend over an unreadable legend.