Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper aims to speed up solving of QPs by using RL to learn adaptive parameters of the OSQP solver. The authors develop and implement an RL algorithm based on TD3 to obtain optimal parameters for OSQP that significantly speed up solution times for several classes of problems.
main_review: The paper is well-motivated as quickly solving QPs is important for a variety of applications, and the approach is described clearly. However, I had difficulty understanding some of the results/experiments, and thus how much the proposed approach improves on existing work and where it falls short. In particular:

(1) In section 5.3, it’s mentioned that OSQP’s handcrafted policy beats RLQP on class-specific problems. Is this even with a class-specific policy? This seems to be concerning if RLQP cannot learn to outperform the handcrafted policy on specific problem classes.

(2) The results in Table 1 for QPLIB suggest that RLQP can improve solve times, but when it does not improve solve times, it often fails (timeout). This may suggest that RLQP learns parameters that also make the solver very fragile. This should be discussed in the Limitations section, and the RL policy may need to be trained to be more robust.

Overall the paper/idea are very interesting, but issues still exist with the results. The above mentioned issues should be clarified, and the fragility of the resulting policy seems to be a natural problem that would arise from using RL in this domain, and should be dealt with for wider applicability of the approach. 

Minor points:

(1) RL is known to have significant variance. Did you try running TD3 several times, and observing the  performance across different policies? Or was the policy trained once and the results are just taken from this run?

(2) The scale on Figure 3 makes it difficult to read, especially for low problem sizes – perhaps the x-axis could be log scale?. Also, should there be a dashed green curve that is missing?
limitations_and_societal_impact: As mentioned above, discussion of robustness of the resulting policy and when/why it can be expected to fail should be included.
review_title: null

final_score: rating: 1: Trivial or wrong
review_content: summary: Applied RL to the problem of 'auto' tuning of the ADMM class of optimization algorithms for QPs.
main_review: The paper is rather light in terms of novelty. Indeed, the idea of using RL to tune ADMM algorithms is not new and has been introduced in [44]. The difference w.r.t. [44] is marginal, if any at all.

Besides, the paper is not well written; the convex optimization section is rather amateurish, same to be said for the RL part, etc.

As far as the general idea goes, I am not convinced that this line of work brings much to the optimization algorithmic area, since to tune the ADMM algorithm, or any algorithm, the authors are solving yet another optimization problem, much more complex, since non-convex and potentially large-scale, with many other hyper-parameters which need to be tuned !!! so what's the gain really ? do you use another RL algorithm to tune the optimization algorithm (e.g. ADAM) that is used to solve the first RL algorithm, etc. ? This line of research is merely transferring the problem of tuning one optimization algorithm to another optimization algorithm, and so, in my opinion, not useful as it is proposed in this paper.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Summary

The paper uses RL methods to learn how to adapt the internal parameters of a QP solver to minimize solve time. The paper demonstrates superior performance compared to current standard solvers across a large class of QP problems. 

main_review: 
Pros

The motivation of applying RL to minimize solve time across a distribution of optimization problems seems natural to me, since solve time is non-differentiable with respect to the internal parameters of the solver. The paper also shows superior performance compared to standard solvers across a variety of QP problem classes, and therefore increases my confidence in the utility of the approach.

Cons 

Given the successes of RL in solving problems where sample efficiency is not a concern, perhaps the results presented in the paper are not surprising. That being said, given that solving QP has many practical uses, I think this method and line of work will be very practically useful.

The paper demonstrates the benefit of using RL to reduce solve time, but if I am not mistaken, does not offer any analysis on why the updates by the RL policy to the internal parameters of the solver reduces solve time. It would be interesting if the authors can offer some analysis on this. For example, how different are the updates as computed by the RL policy compared to the heuristic update rule?

Limitations as a reviewer 

I am not an expert in the area of using RL to speed up optimization, therefore I might have missed crucial background context in assessing the paper.

limitations_and_societal_impact: Yes, I read the checklist at the end of the pdf and agree with the authors' assessment
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method to improve QP solving speed via reinforcement learning.  A regularization parameter used in OSQP solver is being updated via a RL policy. The approach is shown to improve OSQP performance on various benchmarks. 
main_review: Pro:

1. using RL to adapt QP solver parameters is novel to me.

2. approach seems to be effective across various benchmarks.

3. detailed ablation on generalization to problem sizes, problem classes and warm start problems.

4. code provided.

Con:

1. The choice of state space might need more discussion, more on this in questions.


Questions:

1. The state spaces of the RLQP(scalar) and RLQP(vector) are very different, as RLQP(scalar) only has the primal/dual residual while RLQP(vector) has other stuff.  If one provide the additional information of RLQP(vector) to RLQP(scalar), will performance of RLQP(scalar) improve?

2. Why not just use a big policy that takes all the state and output all the action for RLQP? My understanding is that the setup in the current paper allows for solving QP with different sizes, but suppose I only want to solve problem of particular size, will the "big policy" approach generate more efficient policy?
limitations_and_societal_impact: Yes, limitations are discussed.
review_title: null

Meta-review: This is a difficult one to assess. There were some strong opinions among the reviewers, and one reviewer that did seem a bit excessively harsh - perhaps expecting the paper to have the same sort of rigor one might find in a pure optimization paper, rather than an RL application paper.

Evaluated as an RL application paper, a key question is whether the results generalize enough to be interesting. The word "enough" is important because it's not reasonable to expect RL to find a silver bullet set of parameters that will make everything better. One might hope that RL could find useful parameters within related classes of problems. I do think that the authors provide some reasonable evidence to support the claim that they are making significant progress in this direction, so my overall assessment is on the positive side, albeit with some reservations that reflect that reality that we have seen decades of efforts to tweak optimization through RL (see, e.g., Justin Boyan's thesis from the late 90's) but not much practical impact despite some nice tables and graphs in papers.
Decision: Accept (Poster)