Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a first-order preconditioning (FOP) method to generalize previous work on hypergradient descent to learn a preconditioning matrix that only makes use of first-order information.

Pros:
This paper extends the idea of hypergradient descent in [Almeida et al., 1998; Maclaurin et al., 2015; Baydin et al., 2017] with a preconditioning method. A low-rank FOP is further proposed to lighten the computation burden for the preconditioning matrix. 

Cons:
1-	The novelty and contribution is not clear.
2-	The ideas of approximating the preconditioning matrix or factorized approximate inverse have been well studied in the literature, which are not sufficiently cited in the paper, such as Adagrad (Duchi et al. 2011), review in Bottou et al. 2016, etc. 
3-	Derivation of Eq.(4) seems to be missing. 
4-	Typo errors such as “is can” in page 5. 
5-	A mistaken derivation in A.1 Eq.(20). “k” should be “k+1”.

Therefore, I tend to give this paper a Weak Reject score. 
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper studies hypergradient descent for precondition matrices. The goal is to learn an adaptable preconditioning for the task while training. Specifically, they take the gradient of the loss wrt the precondition matrix and update the precondition matrix to decrease the loss. They reparametrize the precondition matrix to ensure it is positive-definite and provide low-rank approximations and they provide cheap approximations for CNNs.

Pros:
- Figure 3 and 4 show promising results on cifar10 with a 9-layer cnn.
- Figure 4 shows FOP can improve the accuracy for particular hyper-parameters. In cases improving by 2%.

Cons:
- Results on imagnet are not particularly good. The improvement is not significant.
- Why positive-definite precondition matrix rather than positive-semi-definite?
- Section 5: why is a degenerate precondition matrix bad? Fisher and Hessian for deep networks can be highly ill-conditioned.
- Theo 1 seems to have errors. The term M_t in the update rule should show up in the bound on P as an exponential term in the first upper bound.
- Figure 2: On mnist after 20 epochs the model has not reached 1% test error. Not clear if we can make any conclusions from this figure.

After rebuttal:
I keep my rating as weak reject. I reiterate that results look promising. However, the quality and accuracy of the writing are not acceptable for a paper on optimization. In my original review I only named a few problematic statements. I have to clarify that I do not think fixing only those few is enough.

I am also not convinced about the proof of Theorem 1. Basically, section 6 looks very much like section 5 from Baydin et al. 2018. Even the wording is mostly the same. Theorem 5.1 in Baydin et al. 2018 is based on their update rule in Eq 6 in the form of alpha_t = alpha_{t-1} - beta nabla^T nabla, where alpha does not appear in the second term. However, in this paper, the update rule on line 7 in Algorithm 1 is M_t = M_{t-1} + rho * eps *(.) M_{t-1}, where M_t appears in the second term. Hence, the first bound in Theorem 1 in this paper cannot simply be the same as in Baydin et al. 2018.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes an interesting optimization algorithm called first-order preconditioning (FOP). 
The basic idea of FOP is updating the preconditioned matrix by its gradient, which avoid calculating or approximating the Hessian directly. To make the algorithms more practical, the authors also conduct the low-rank FOP and the momentum-type version. The empirical studies on CIFAR-10 and ImageNet validate the effectives of the proposed algorithms.

Major comments:

1. Section 2.1 says “we follow the example of Almeida et al. (1998) and assume that J does not dramatically”. However, the goal of FOP is to encourage J reduce faster. Is there any conflict?

2. In low-rank FOP, the initial preconditioner P contains the term I_m which does not exist in standard FOP (section 2.1). How does this term affect the update procedure? Can you provide some details?

3. Theorem 2 provide a linear convergence of FOP under convex, Lipschitz and PL condition. The proof relaxes the preconditioner P into its minimum and maximum eigenvalues. Since P changes over the course of training, it is difficult to check weather the result of Theorem 2 is stronger than gradient descent method.

4. Why the experimental results not include the other second order optimization algorithms such as K-FAC and KFC?

Minor comment:

The notations M in (1) (2) and (5) are ambiguous. It is prefer to use another letter to present the preconditioner in (1).

review_title: Official Blind Review #2

Meta-review: This paper has been assessed by three reviewers who scored it as 3/3/3, and they did not increase their scores after the rebuttal. The main criticism lies in novelty of the paper, lack of justification for MM^T formulation, speed compared to gradient descent (i.e. theoretical analysis plus timing). Other concerns point to overlaps with Baydin et al. 2015 and the question about the validity of Theorem 1. On balance, this paper requires further work and it cannot be accepted to ICLR2020.
Decision: Reject