Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper studies a problem of learning step-size policy for L-BFGS algorithm. This paper falls into a general category of meta-learning algorithms that try to derive a data-driven approach to learn one of the parameters of the learning algorithm. In this case, it is the learning rate of L-BFGS. The paper is very similar in nature to the papers of Ravi & Larochelle, MAML and Andrychowicz.

My biggest issue with this paper is the evaluation. The paper itself cites in the introduction:  "...for large-dimensional problems this procedure is likely to become the bottle-neck of the optimization task". However, the paper doesn't not provide necessary evaluation to help resolving this issue. In fact, it is not very clear at all that the proposed method would work on a more general scenario when the evaluation dataset is wildly different from training the dataset.

I'm also a bit surprised to see this paper tackling specifically L-BFGS algorithm, instead of more general case of learning rate parameter for any gradient basis algorithm. I would be curious to learn what is so special about L-BFGS that made the authors chose it. After all, the paper (and L-BFGS) deals with deterministic optimization problems on a full batch which limits the applicability of the paper.





review_title: Learn to learn approach for L-BFGS step size parametrization

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Summary**:

The paper presents a novel steps-size adaptation for the L-BFGS algorithm inspired by the learning-to-learn idea. The step-size policy is determined by two linear layers which compare a higher dimensional mapping of curvature information which is trained to adapt the step-size.


**Reasons for score**:

While the idea of learning to predict a suitable step-size is intriguing and is definitely worth pursuing I am not convinced that the proposed algorithm results in an active policy that usefully adapts the step-size. There are too many concerns that I think needs to be addressed and it is not clear if the speed up improves over the reliability of a line search. I therefore vote to reject the paper in its current form.

**Pros**:

- It is clear that a lot of thought has gone into the project to come up with the policy. I think it might have merit but requires additional tests.

- The figures were first difficult to understand but once the content had been explained in the text the benefit of the chosen presentation became clear.



**Concerns**:

- My main concern is best visualized in Figure 3. Both the learned policy and the BTLS seem to mostly favour a step of 1 which raises several questions. 
	1) What would be the results of using no adaptation and rely on a step of 1 (or 0.9) constantly as a baseline? 
	2) The $\pi$-algorithm mostly uses a step-size of 1 which happens top be the upper boundary $\tau_M$, which means it is not clear if the policy network has learned that 1 is a good step or if it has not learned at all and the results are just due to the clipping. What would happen if $\tau_M>1$ for example? 
	3) Given that both the BTLS and $\pi$ mostly use $t_k=1$, is there any intuitive explanation as to why the results between the two algorithms differ by so much in figure 3? Are there additional figures where the BTLS similarly outperforms the competition (it did reach $10^{-5}$ first in ~60% of the tasks according to table 1, column 1)? This despite the fact that BTLS is at least 1 forward pass more expensive per iteration than the policy (for a fully connected network I think that is ~50% of the iteration cost).

- The benefit of using the double network for the policy is not clear to me. What would be the result of using a single linear layer instead or a recurrent network that monitors temporal changes to the curvature information that is used as input?

- Given that the input and output dimensionality of the policy network is of low dimension it would be interesting to see what the weights and biases look like for respective policy layers. By looking at the weights it would be possible to see what curvature information makes the network decide to adjust the step-length. Does the policy learn a cosine behaviour similar to the proposed possibility in the appendix? 

- Could the policy be used for another optimization algorithm for which $-g_t^\intercal d_k>0$, such as RMSprop or GD? It might be easier to understand the influence of the policy in such a setting.


Comparably minor points:

- Section 3 first paragraph ends with a statement regarding $\rho_i$ to keep the Hessian approximation s.p.d by ignoring iterations. Is this used in the implementation? 
- Table 1: According to what metric is "first" defined (iteration, time, function evaluations)? It would be good to mention the average value of this metric for each optimizer at the end of $K=800$ inner steps.
- In Section 6 it says that the learning rate of Adam and RMSprop was chosen to yield fast convergence. I understand this as quickly reaching a threshold, not achieving best performance for allocated time. Is this correct? That could help explain why so many settings in table 1 and figure 4 fail to converge. Personally I think the first-order algorithms should be allowed to change the learning rate between problems to prevent them from not converging (ex. RMSprop).
- Eq.7 s.t. -> with, or is the outer problem actually constrained and I missed something?


-------------------


**Post Rebuttal**

I have considered the revised article, additional reviews and rebuttal and decided to slightly raise my score but I am still in favor of rejecting the paper. Below is a summary of my reasoning.

--------

The authors have provided a good rebuttal and I am overall pleased with the detailed response, additional experiments and figures, and overall exhibited transparency. 
Unfortunately my assumption about $t_k$ seemed correct when considering the additional L-BFGS-B results, which indicate that using standard $t_k=1$ is a really strong baseline that proved difficult to beat.

I would suggest finding another set of problems where $t_k=1$ is not so good for L-BFGS or consider adapting another first-order algorithm for which it is clear that the step-length needs to change between tasks and architectures.
review_title: Official Review #2

final_score: rating: 5: Marginally below acceptance threshold
review_content: ** Description

This paper makes two separate contributions to machine learning: (1) It proposes a neural network to learn the step-size policy of L-BFGS and (2) it uses the algorithm for training a deep neural network.

** Pros

In practical terms it provides a solution to the problem of finding an appropriate step-size policy.  It also demonstrates that the policy out performs ADAM and RMSprop on a rather strangely chosen problem.

** Cons

Learning how to optimise by examining related problems is often a fruitless endeavour because small changes in a problem can drastically change the behaviour of optimisation algorithms.  It would have been nice to see more convincing evidence that learning a step-size polices generalises across at few more problem domains.

For me, the evaluation of their algorithm for training a neural network was slightly unconvincing.  Possibly this was just chosen as an example optimisation problem and the application shouldn't be taken too seriously (a comment to that effect would be useful).  Obviously the use of full-batch training is not that practical for most problems.  For neural network training, robustness to the noise produced by mini-batch training is important to understand.  Although ADAM and RMSProp are state-of-the-art optimisers for minibatch training when compared on full-batch it would be useful to compare with other standard approaches such as conjugate gradient, etc.  The choice of problem was puzzling.  Clearly an MLP on MNIST is not representative of modern machine learning.  It left the question of whether a small network was deliberately chosen because the algorithm does not scale to really large networks.  Again a comment about this would have strengthened the paper.

I was left with the impression that the authors were being slightly selective in their choice of problems for showing the utility of their method.  I would have liked to see more conclusive evidence in this direction and a clearer discussion of the regime where this method is likely to perform well.

** Typos

The paper is comprehensible, but would gain from being proof read by a native speaker.  This is not a consideration that affects the valuation.  Examples of rewordings that would make the paper flow slightly better are

p2 l4: good <-- well
p2 l8: exempts <-- frees
p2 l23: seek <-- search
p3 section 3 paragraph 2 line 2: it does ... <-- there does ...

review_title: Review on Step Size Policy Network for L-BFGS

final_score: rating: 4: Ok but not good enough - rejection
review_content: 1. Paper contribution summary
    This paper proposes a neural network architecture with local gradient/quasi gradient as input to learn the step-length policy from data for the specific L-BFGS algorithm. The step-length policy can be learned from the data with similar optimization problems. The designed neural network architecture for step-length policy is using inner product as input to allow the policy to be independent of the problem size, and the logarithm operation is used to enable products and division among powers of the inputs.
    The numerical example shows that the learned policy is comparable to L-BFGS with backtracking line search and potentially better generalization to the same problem with higher dimension.

2.  Strong and weak points of the paper
    Strong part: 1) The network architecture for the step-length policy is very interesting, and may be useful for other problems with similar needs. 2) The numerical experiment shows that the learned step-length policy is comparable to the same algorithm with backtracking line searched step-length, which looks promising.
    Week part: 1) The paper's goal is limited to design the step-length policy for one specific optimization algorithm L-BFGS. If the original L-BFGS algorithm is not effective for certain problems, then the learned step-length policy may be un-useful as well. 2) Based on Table 1's metric of gradient length smaller than \epsilon, the learned step-length policy seems not better than Adam even in the higher dimension setup in which the Adam is using the pre-tuned learning rate. The result in Fig. 5 seems better for the learned step-length policy, but that uses the smallest value over training as compared value, which may not be 'fair'. Because Adam, RMSProp, and L-BFGS are different algorithms, which may result in different (local) minima in different problems. 3) The performance test setup is not realistic in neural network model training. The setup for training the model (inner optimization) along with step-length policy model (outer optimization) is with a fixed 1000 images (randomly selected from 60 batches of data) over T = 16 outer iteration and 10 epochs. When comparing different algorithms' performance, it also splits the test dataset into 10 batches of 1000 images. And record performance for each fixed batch dataset, which is not realistic in usual neural network model training. Usually, different batches are randomly selected at each time step as opposed to being fixed. This makes the claimed performance doubtful in realistic setting.

3. Based on the strong and week points of this paper, I tend to reject it.

4. Supporting arguments
    Please refer to my above arguments.

5. Questions
    1) Can author/s please explain the first two plots in Figure 4? It seems to me that both Adam and RMSProp are faster reaching to the required gradient norm precision. The precision seems different between these two algorithms and the author/s claimed one. Why not use the same precision requirement so that we can compare these algorithms directly?
    2) Can author/s provide any feedback on the third point of "weak part"?
    3) Can this proposed architecture/idea be used as a step-size policy for not just L-BFGS?
review_title: Interesting submission about step-length policy learning

Meta-review: The paper presents a novel procedure to set the steps-size for the L-BFGS algorithm using a neural network.
Overall, the reviewers found the paper interesting and the main idea well-thought. However, a baseline that was proposed by one of the reviewers seems to be basically on par with the performance of the proposed algorithm, at least in the experiments of the paper. For this reason, it is difficult to understand if the new procedure has merit or not. Also, the reviewers would have liked to see the same approach applied to different optimization algorithms.

For the reasons, the paper cannot be accepted in the current form. Yet, the idea might have potential, so I encourage the authors to take into account the reviewers' comments and resubmit the paper to another venue.
Decision: Reject