Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper explores momentum SGD and an adaptive version of momentum SGD which the authors name YF (Yellow Fin). They compare YF to hand tuned momentumSGD and to Adam in several deep learning applications.


I found the first part which discusses the theoretical motivation behind YF to be very confusing and misleading:
Based on the analysis of 1-dimensional problems, the authors design a framework and an algorithm that  supposedly ensures accelerated convergence. There are two major problems with this approach:

-First: Exploring 1-dim functions is indeed a nice way to get some intuition. Yet,  algorithms that work in the 1-dim case do not trivially generalize to high dimensions, and such reasoning might lead to very bad solutions.

-Second: Accelerated GD does not benefit over GD in the 1-dim case. And therefore, this is not an appropriate setting to explore acceleration.
Concretely, the definition of the generalized condition number $\nu$, and relating it to the standard definition of the condition number $\kappa$, is very misleading. This is since $\kappa =1$ for 1-dim problems, and therefore accelerated GD does not have any benefits over non accelerated GD in this case.
However, $\nu$ might be much larger than 1 even in the 1-dim case.


Regarding the algorithm itself: there are too many hyper-parameters (which depend on each other) that are tuned (per-dimension).
And as I have mentioned, the design of the algorithm is inspired by the analysis of 1-dim quadratic functions.
Thus, it is very hard for me to believe that this algorithm works in practice unless very careful fine tuning is employed.
The authors mention that their experiments were done without tuning or with very little tuning, which is very mysterious for me.

In contrast to the theoretical part, the experiments seems very encouraging. Showing YF to perform very well on several deep learning tasks without (or with very little) tuning. Again, this seems a bit magical or even too good to be truth. I suggest the authors to perform a experiment with say a qaudratic high dimensional function, which is not aligned with the axes in order to illustrate how their method behaves and try to give intuition.

review_title: Misleading and shaky theoretical motivation/approach

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a method to automatically tuning the momentum parameter in momentum SGD methods, which achieves better results and fast convergence speed than state-of-the-art Adam algorithm.

Although the results are promising, I found the presentation of this paper almost inaccessible to me.

First, though a minor point, but where does the name *YellowFin* come from?

For the presentation, the motivation in introduction is fine, but the following section about momentum operator is hard to follow. There are a lot of undefined notation. For example, what does the *convergence rate* mean (what is the measurement for convergence)? And is the *optimal accelerated rate* the same as *convergence rate* mentioned above? Also, what do you mean by *all directions* in the sentence below eq.2?

Then the paper talks about robustness properties of the momentum operator. But: first, I am not sure why the derivative of f(x) is defined as in eq.3, how is that related to the original definition of derivative?

In the following paragraph, what is *contraction*? Does it have anything to do with the paper as I didn't see it in the remaining text?

Lemma 2 seems to use the spectral radius of the momentum operator as the *robustness*. But how can it describe the robustness? More details are needed to understand this.

What it comes to Section 3, it seems to me that the authors try to use a local quadratic approximation for the original function f(x), and use the results in last section to find the optimal momentum parameter. I got confused in this section because eq.9 defines f(x) as a quadratic function. Is this f(x) the original function (non quadratic) or just the local quadratic approximation? If it is the local quadratic approximation, how is it correlated to the original function? It seems to me that the authors try to say if h and C are calculated from the original function, then this f(x) is a local quadratic approximation? If what I think is correct, I think it would be important to show this.

Also, the objective function in SingleStep algorithm seems to come from eq.13, but I failed to get the exact reasoning.

Overall, I think this is an interesting paper, but the presentation is too fuzzy to get it evaluated.
review_title: YellowFin and the Art of Momentum Tuning

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Apologies for short review, I got called in late. Marking my review as "educated guess" since i didn't have time for a detailed review]

The paper proposes an algorithm to tune the momentum and learning rate for SGD. While the algorithm does not have a theory for general non-quadratic functions, experimental validation is extensive, making it a worthy contribution in my opinion. I have personally tried the algorithm when the paper came out and can vouch for the empirical results presented here.
review_title: Worthy contribution

Meta-review: This paper asks when SGD+M can beat adaptive methods such as Adam, and then suggests a variant of SGD+M with an adaptive controller for a single learning rate and momentum parameter.  There is are comparisons with some popular alternatives.  However, the bulk of the paper is concerned with a motivation that didn't convince any of the reviewers.
Decision: Reject