Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new step size adaptation in first-order gradient methods. The proposed method establishes a new optimization problem with the first-order expansion of loss function and the regularization, where the step size is treated as a variable.  ADMM is adopted to solve the optimization problem.

This paper should be rejected because (1) the proposed method does not show the convergence rate improvement of the gradient method with other step sizes adaptation methods. (2) the linearization of the objective function leads the step size to be small ($0<\eta<\epsilon$), which could slow down the convergence in some cases. (3) the experiments generally do not support a significant contribution. In table 1, the results of the competitor are not with the optimal step sizes. The limit grid search range could not verify the empirical superiority of the proposed method.


Minor comments:
The y-axis label of (a) panel in each figure is wrong. I guess it should be "Training loss ".
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: First, I would like to point out that there has not been a conclusion or discussion section included, therefore the paper appears to be incomplete.
Aside from this the main contribution of the paper is a study on optimising the step size in gradient methods. They achieve this through the use of alternating direction method of multipliers. Given all the formulations provided, it appears as if this method does not rely on second order information or any probabilistic method.
An extension of the proposed method covers stochastic environments.
The results demonstrate some promising properties, including convergence and improvements on MNIST, SVHN, Cifar-10 and Cifar-100, albeit marginal improvements.
Although the results appear to be promising the overall structure of the paper and the method presented are based upon established techniques, therefore the technical contribution is rather limited.
I have read the rebuttal and answered to some of the concerns of the authors.
review_title: Official Blind Review #2

Meta-review: The paper is rejected based on unanimous reviews.
Decision: Reject