Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I am not sure about my assessment of this paper. 
The authors propose their approach as a mixture of two other approaches. Then, they proceed with an illustrative 2-D example where they apply the same hyperparameters for all tested methods. This does not seem appropriate especially given that later in the paper they show that SGD with momentum and Adam use very different hyperparameter settings such as learning rate and weight decay. Basically, SGD and NovoGrad perform very similar in Figure 6 and the difference might be due to a too large learning rate for SGD (and other approaches). 
Increasing it further does not help as Figure 7 suggests. It is not the case for NovoGrad because of its gradient normalization. However, what the authors don't show us is a different experimental setup with some trivial objective function where it would take NovoGrad an enormous amount of steps to converge to the optimum IF the algorithm is initialized far enough from the optimum. Since NovoGrad is based on normalized gradients, its steps are pretty much just signs of the original gradient multiplied by learning rate, especially in 2D. Thus, with some initial learning rate of 0.1 and when initialized in say (1, 10^10), it would take NovoGrad about 10^9 steps to converge independently on the scale of the objective function. This is not the case for SGD whose gradients are not normalized. This is actually a strong counterargument to the main claim of the paper that "NovoGrad is robust to the choice of learning rate and weight initialization". This argument in the paper is based solely on the results of the toy problem where other methods used large learning rates. 

The paper includes a set of experiments containing different methods and their hyperparameters. 
When the authors use some cosine function to schedule learning rate in Table 1, they do it for AdamW and NovoGrad but they use polynomial schedule for SGD. Why? Weight decay and learning rate values for SGD and NovoGrad are very different in the same table, why? Do the authors explain why this difference may happen, they don't. I believe that there is a better answer than just "this is the result of hyperparameter tuning". 

The authors note that when beta=0, NovoGrad becomes layer-wise NGD with decoupled weight decay. They suggest beta to be in [0.2, 0.5] and I am wondering which beta were used in different experiment (the default suggested value is not given). 

The experimental results suggest that NovoGrad performs slightly better than SGD with momentum but it has more hyperparameters and we don't know whether the results are due to different computation efforts in hyperparameter tuning. Figure 5 compares Adam and Novograd on WikiTex-103 and shows that Adam converges faster in terms training perplexity. However, Adam's test perplexity is worse than the one of NovoGrad. Interestingly, NovoGrad's test perplexity is better than its training perplexity especially in the beginning. It seems that NovoGrad is good here because it does not converge well, i.e., the results are likely due to regularization which is problematic in the original Adam. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In the paper, the authors propose a novel optimization method for training deep learning models. The idea is from the LARS and AdamW. The authors then test the proposed method on multiple experiments, results showing that the proposed method works better than other compared methods.  The following are my concerns:

1) No convergence guarantee in the paper. There are too many papers claiming faster convergence these days, proof of convergence guarantee is always preferred. 
2) The proposed method is straightforward and easy to understand. It is just a combination of AdamW and LARS. I am worried about the novelty of this paper. 
3) In the experiments, why the compared methods are usually different. For example, compared methods are Adam, SGD, and NovoGrad in table 4 and compared methods are Adam, AdamW, and NovoGrad in table 6.  Why not compare all these methods?
4) When the batch size varies,  is it required to tune beta_2 accordingly? I didn't find it clearly mentioned in the paper, could authors explain how to set it? 
5)  I am confusing that NovoGrad method works much better than Adam or AdamW in Table 6 with no weight decay, more explanations are required. 
6) It is unclear why NovoGrad is better than LARS. LARS normalizes learning rate through |w|_2/|g|_2. The authors should explain why normalizes using layerwise |g|_2 is better.

Although the idea is straightforward, the proposed method may be helpful for the community.   I will consider increasing the score if authors can address my concerns.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors present a variation of Adam that combines layerwise normalization and weight decay decoupling. They test quite extensively their algorithm against other optimization methods on varied tasks.

From a theoretical point of view, the contribution seems very incremental. The authors acknowledge that weight decay decoupling is already present in the literature, so it appears that the only contribution is the layer-wise normalization. No justification is proposed as to why this kind of normalization would either accelerate the convergence or leads to better generalization. Proof of convergence even in a deterministic convex setting are missing, so the reader has to extrapolate correctness from previous work on adaptive gradient descent.
On the other hand, the proposed algorithm is tested on a great variety of tasks, using state of the art models. The reported performance for the benchmarks (checked for Resnet and Transformer-XL) are on par with what can be found in the literature. The proposed method outperform consistently the other optimizers in terms of generalization performance.

I have some concerns regarding section 4:
 “SGD converges nicely toward (1, 1) but its trajectory is still slightly off of the optimal solution”. It is unclear to me what the reader should understand. Does it converge to the optimal solution? If yes, why should we expect the trajectory to follow the hyperbola?
Using the same learning rate for all methods is a bit odd. Why not search for the best learning rate for each optimizer and report its performance? It seems that the oscillation of some of the optimizers could be fixed by using a smaller learning rate. Also, it can be seen in section 5 that Adam consistently needs a smaller learning rate than Novograd.
If the difference between NovoGrad and AdamW lies in the layer-wise second moment, it is unclear to me why their performance should differ on this task, as each layer has only one weight. It would be great if the authors could clarify this point.

As a conclusion, I am a bit conflicted regarding this paper. The motivation for this modified version of AdamW are unclear, but the empirical results are convincing and rigorous. The authors made a great effort in testing in a variety of different settings. I’m leaning toward accepting this paper, to give the community a chance of testing and, maybe, adopting it.

review_title: Official Blind Review #3

Meta-review: The paper presented an adaptive stochastic gradient descent method with layer-wise normalization and decoupled weight decay and justified it on a variety of tasks. The main concern for this paper is the novelty is not sufficient. The method is a combination of LARS and AdamW with slight modifications. Although the paper has good empirically evaluations, theoretical convergence proof would make the paper more convincing. 
Decision: Reject