Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: This paper refers to "local minima" and "minima" without demonstrating a support for their presence. 
Please describe the conditions of the latter and demonstrate whether they are satisfied. 

review_title: local minima

final_score: rating: 4: Ok but not good enough - rejection
review_content: First of all, I would like to thank the authors for putting this much work into a necessary but somewhat tedious topic. While I think the paper is somewhat below the standard of a conference paper (see detailed comments below), I would definitely love to see a version of this paper published with some of the issues ironed out. I also agree with many of the points raised by other reviewers and will not repeat them here.

Major points:

-- "As we saw in the previous section, the minima of deep network loss functions are for the most part decent."

All you said in the previous section was that theory shows that there are no bad minima under "strong assumptions". There is no practical proof that minima do not vary in quality.

-- "This implies that we probably do not need to take many precautions to avoid bad minima in practice. If all minima are decent, then the task of finding a "decent minima quickly" is reduced to the task of finding any minima quickly."

First of all, as one of the reviewers pointed out, we are never guaranteed in practice to actually reach a local minimum. We could always hit a region of the objective function where the algorithm makes essentially no further progress. The final error level, in practice, actually does depend significantly on many factors such as (i) optimization algorithm (ii) learning rate schedule (iii) initialization of weights (iv) presence of unsupervised pretraining (v) whether neurons are added or eliminated during training etc. etc. Therefore, the task of optimizing neural networks is far from being "reduced to finding any minima quickly".

-- Figure 1
 
I don't like Figure 1, because it suggests to me that you diagnosed exactly where the transition between the two phases happened, which I don't think you did.
 
Also, the concept of having a fast-decaying error followed by a slow-decaying error is simple enough for readers to understand without a dedicated graph. Minor point on presentation: The red brace is positioned lower in the figure than the blue brace and the braces don't join up horizontally. Please be more careful.

-- Misuse of the transient phase / minimization phase concept

In section 4.3, you talk about the transient and minimization phase of optimization. However, you have no way of diagnosing when or if your algorithm reaches the minimization phase. You seem to think that the minimization phase is simply the part of the optimization process where the error decreases slowly. AFAIK, this is not the case. The minimization phase is where the optimization algorithm enters the vicinity of the local minimum that can be approximated by the second-order Taylor expansion. For this to even occur, one would have to verify, for example, that the learning rate is small enough. You change the algorithm after 25%, 50% and 75% of training, but these points seem arbitrary. What is the minimization phase was reached at 99%, or 10 epochs after you decided to stop training?

-- Only 1 dataset
 
You run most experiments on only 1 dataset (CIFAR). Please replicate with at least one more dataset.
 
-- Many figures are unclear
 
For each figure, the following information are relevant: network used; dataset used; learning rate used; batch norm yes / no; whether figure shows train, test, or validation error. It should be easy for the reader to ascertain this information for all figures, not just for some.
 
-- You say at the beginning of section 4.1 that each algorithm finds a different minimum as if this is a significant finding. However, this is obvious because the updates taken by these algorithms vary wildly. Keep in mind that there is an exponentially large number of minima. The probability of different algorithms choosing the same minimum is essentially zero because of their sheer number. The same would be true if you even shift the learning rate slightly or use a different random seed for minibatch generation etc. etc.
  
-- Lack of confidence intervals

The value of Figures 1, 2, 3 and 6 is limited is because it is unclear how these plots would change if the random seed were changed. We only get information for a single weight initialization and a single minibatch sequence. While figures 5 and 7 can be used to try and infer what confidence intervals around plots in figures 1, 2, 3 and 6 might look like, I think those confidence intervals should still be shown for at least a subset of the configurations presented.

-- Lack of information regarding learning rate

There is big question mark left open regarding how all your results would change if different learning rates were used. You don't even tell us how you chose the learning rates from the intervals you gave in section 3.4.

-- Lack of information regarding the absolute distance of interpolated points

In most figures, you interpolate between two or three trained weight configuration. However, you do not say how far the interpolated points are apart. This is highly significant, because if points are close together and there is a big "hump" between them, it means that those points are more "brittle" than if they are far apart and there is a big "hump" between them.
 
Minor points:
 
-- LSTM is not a fixed network architecture like NiN or VGG, but a layer type. LSTM would be equivalent to CNN. Also, the VGG paper has multiple versions of VGG. You should specify which one you used.
  
-- The font size for the legends in the upper triangle of Table 1 is too small. You can't just write "best viewed in zoom" in the table caption and pretend that somehow fixes the problem. Personally, I prefer no legend over an unreadable legend.
review_title: From an interested reader: many issues to be ironed out

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper provides an extensive analysis of the error loss function for different optimization methods. The presentation is well done and informative. The experimental procedure is clarified sufficiently well. Theoretical evaluations like this are crucial for a wide range of applications and help to better understand and improve the convergence behavior for a given system.

Pros:

- Important analysis
- Good visualizations

Cons:

- The paper describes mostly the observation that the optima vary for different methods, however doesn't attempt to explain why it happens and how to solve it (aside from batch-norm)
- Some fonts are very small (e.g. Fig. 5)

review_title: A good evaluation of optimization methods

final_score: rating: 4: Ok but not good enough - rejection
review_content: I appreciate the work but I do not think the paper is clear enough. 
Moreover, the authors say "local minimia" ~70 times but do not show (except for Figure 11?) that the solutions found are not necessarily local minima. 
The authors do not talk about that fact that slices of a non-convex problem can look like the ones they show. 
It is well-known that the first-order methods may just fail to deal with certain non-convex ill-conditioned problems even in low-dimensional noiseless cases, the place/solution where they fail to make progress is not necessarily a local minimum. 
Some sentences like the one given below suggest that the study is too superficial:  
"One of the interesting empirical observation is that we often observe is that the incremental improvement
of optimization methods decreases rapidly even in non-convex problems."
review_title: null

final_score: rating: null
review_content: This is a comprehensive and useful evaluation. Some remarks:

- conclusions are somewhat missing and are not entirely clear
- there are some minor typos, e.g. page 8 on bottom (swithced) and inconsistencies in the capitalization of Adam
review_title: conclusion

final_score: rating: null
review_content: Dear Authors,

1) What network regularizations were used in the experiments?
2) The paper seems incremental over Goodfellow et al. 2015 (it extends this work though). Could you clarify more the significant novelty of this paper and the message it conveys? What future research directions it gives rise to? Could you discuss what are the algorithmic/other consequences of the findings in the paper?
3) The sentence starting at "For every pair of optimization algorithms, we..." on page 4 seems to have a missing word.
4) Similarly, sentence on page 4: "This suggests to use that..." seems to have a missing part.

review_title: Clarifications

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper is dedicated to better understanding the optimization landscape in deep learning, in particular when explored with different optimization algorithms, and thus it also characterizes the behavior of these algorithms. It heavily re-uses the approach of Goodfellow et al. (2015). I find it hard to understand the contributions of the paper, for example: is it surprising that different algorithms reach different solutions when starting from the same initialization? It would be useful if the authors build such basic intuition in the paper. I also did not receive a clear answer to the question I posed to reviewers regarding clarifying how does the findings of the paper can contribute to future works on optimization in deep learning. And this is what I find fundamentally missing. So for example, there are probably plenty of ways to modify approach of Goodfellow et al. (2015), and similar works, and come up with interesting visualization methods for deep learning - but the question is: how is this helpful in terms of designing better algorithms, gaining more intuition how the optimization surface looks like in general, etc.? This is an interesting paper, though I am fairly confident it is a better fit for the journal than this conference. 

It would be interesting and instructive, even for sanity check, to plot the eigenspectra of the solutions recovered by the algorithms to see the order of critical points recovered.
review_title: Interesting paper, but the message is not clear

Meta-review: The paper proposes an empirical investigation of the energy landscape of deep neural networks using several stochastic optimization algorithms.
 
 The extensive experiments conducted by the authors are interesting and inspiring. However, several reviewers expressed major concerns pointing out the limitations of a experimental investigation on real-world datasets. The paper would benefit from additional experiments on simulated datasets. This would allow to complement the experimental analysis. Furthermore, theoretical analysis and analytical derivations, consistent with the simulated datasets, could shed light on the experimental results and allow to make more precise claims. 
 
 A revised version, following the reviewers' suggestions, will result in a stronger submission to a future venue.
Decision: Reject