Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes an optimization method with the preconditioning in the framework of supervised learning. The ideal preconditioning is given by the Fenchel conjugate of the optimization function. This paper uses a supervised scenario to find the ideal preconditioning. The authors point out the importance of the sampling distribution then and propose a sampling scheme using the uniform distribution on the space of gradient descents. The samples are used to train neural networks that imitate the mapping of the Fenchel conjugate. The training network is incorporated into the Dual space Preconditioned Gradient Descent (DPGD). Some numerical experiments show the effectiveness of the proposed method comparing to the standard gradient descent method. 

The authors proposed an interesting approach to the preconditioning in optimization problems. However, the paper is not well-written. In particular, the optimization algorithm is not clearly described. Numerical experiments with some toy problems are not very convincing to show the benefit of the proposed method. Though this paper may have some interesting ideas, more intensive analysis would be required. 

other comments:
- The optimization algorithm is not explicitly described. Is the neural network trained as the batch learning? Is it possible to use the learning of the preconditioning in an online manner? 
- It is not sure whether the ideal distribution \mu over the domain D(f) presented in Proposition 3 is computationally tractable. 
- In Proposition 3: I think that the uniform property of the sampling does not directly mean the optimality in the sense of the learning accuracy. The authors need to investigate the more detailed relationship between the distribution mu and the prediction accuracy of the Fenchel conjugate. 
- The proposed method requires the learning of neural networks, which will be computationally demanding. Please report the overall computational cost of the optimization algorithm in numerical experiments.

review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: This paper attempts to learn a preconditioner for optimization, specifically for the Dual space preconditioned descent (DPGD). 
- The techniques used to learn the preconditioner are heuristic, not scalable and without justification or ablation studies. 
- It does not compare against "standard" optimization techniques that construct data-driven preconditioners such as Adam or Adagrad or even to more Newton, natural gradient methods that use the Hessian or the Fisher information matrix as preconditioners. It shows ad-hoc synthetic experiments in dimensions 1 and 50. This is clearly not enough. 
Detailed review below:
- Section 2: Please explain why Legendre functions are useful in ML. For assumption 1, 2; it needs to be explained why these hold for a given f*. What constraints do you need on f? What functions satisfy these? Please explain this explicitly. 
- Section 3: What is the number of points x_i needed in high dimensions to learn? Is it even possible to scale up this method to high dimensions?
- Constructing \mu requires computing the determinant of the Jacobian. What is the computational complexity? Moreover, it seems that we need access to the \nabla f(x) for all x in D(f)? 
- Please state all the assumptions in the beginning rather than introducing one at a time in the propositions. 
- Remark 1: It is unclear that the cost of an inverse Hessian matrix is more than the procedure proposed in this paper. 
- Section 3.5: Please explain what is the advantage of this learned optimizer compared to other methods? Note that there is literature on non-smooth optimization and methods like sub-gradient descent can be used in this case. 
- What is the justification for the selection of the loss function and log-rescaling?
- The result of Lemma 1 is standard. Please acknowledge this. 
-  Section 4: "The step-size is set to 1". It seems that the optimizer has been overfit and engineered to work on this specific problem. Either these decisions need to be justified, there needs to be an ablation study or there needs to be a larger set of experiments. 



review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: [Update after rebuttal period]
I have read the response,  my confusion in the original reviews cannot be answered satisfactorily. Therefore, I keep my initial scores.


[Original reviews]
Firstly, the motivation of the proposed method is not convincing for me. The authors want to propose a general methodology for learning precondition by supervised learning setting. However the method in practice, the x is a complex distribution, it is difficult to handle the map between the gradient and the x. This method proposes log-scaling, but it needs to be stored with a precision of approximately 15 decimal places and the regressed model will be a piecewise constant function, which is very computationally time-consuming.

Secondly, the experimental results are not sufficient for evaluation. This paper shows two
The experimental result which includes the result of power function and the logistic function. But
It is not clear that the whole process of dual space preconditioned method with the model of computation of precondition given. And without quantitative results given, it is not convincing the “dramatic” speedups” of these methods, because the surprising training process is off-line and time-consuming. On the other hand, because of the different forms of the convex objective function, the network will train for the specific convex objective functions. In my opinion, it is not a general method to lead to dramatically speed up.

Finally，the function of x and the gradient is complex, it is difficult to predict the relationship by using a simple network,

review_title: Official Blind Review #2

Meta-review: Thanks for the detailed replies to the reviewers, which significantly helped us understand your paper better.
However, after all, we decided not to accept your paper due to weak justification and limited experimental validation. Writing should also be improved significantly. We hope that the feedback from the reviewers help you improve your paper for potential future submission.

Decision: Reject