Accelerating Quadratic Optimization with Reinforcement Learning

1 Introduction . Solving quadratic programs ( QPs ) efficiently is critical to applications in finance , robotic control and operations research . While state-of-the-art interior-point methods scale poorly with problem dimensions , first-order methods for solving QPs typically require thousands of iterations . Moreover , real-time control applications have tight latency constraints for solvers [ 33 ] . Therefore , it is important to develop efficient heuristics to solve QPs in fewer iterations . The Alternating Direction Method of Multipliers ( ADMM ) [ 6 , 15 , 18 ] is an efficient first-order optimization algorithm , and is the basis for the widely used and state-of-the art Operator-Splitting QP ( OSQP ) solver [ 44 ] . ADMM performs a linear solve on a matrix based on the optimality conditions of the QP to generate a step direction , and then projects the step onto the constraint bounds . While state-of-the-art , the ADMM algorithm has numerous hyperparameters that must be tuned with heuristics to regularize and control optimization . Most importantly , the step size parameter ρ has considerable impact on the convergence rate . However , is still unclear how to select ρ before attempting the QP solution . While some theoretical works compute the optimal ρ [ 17 ] , they rely on solving semidefinite optimization problems which are much harder than solving the QP itself . Alternatively , some heuristics introduce “ feedback ” by adapting ρ throughout optimization in order to balance primal and dual residuals [ 44 , 6 , 22 ] . We propose RLQP ( see Fig . 1 ) , an accelerated QP solver based on OSQP that uses reinforcement learning ( RL ) to adapt the internal parameters of the ADMM algorithm between iterations to minimize solve times . An RL algorithm learns a policy πθ : S → A , parameterized by θ ( e.g. , the weights of a neural network ) , that maps states in a set S to actions in setA such that the selected action maximizes the accumulated reward r. To train the policy for RLQP , we define S to be the internal state of the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . QP solver ( e.g. , the constraint bounds , the primal and dual estimates ) , A to be the adaptation to the internal parameter ( ρ ) vector , and r to minimize the number of ADMM iterations taken . RLQP ’ s policy can be trained either jointly across general classes of QPs or with respect to a specific class . The general version of RLQP is trained once on a broad class of QPs and can be used out-ofthe-box on new problems . The specialized version of RLQP is trained on a specific class of problems that the solver will repeatedly encounter . While this requires additional setup and training time , it is useful when QPs will be repeatedly solved in application ( e.g. , in a 100 Hz control loop ) . In experiments , we train RLQP on a set of randomized QPs , and compare convergence rates of RLQP to non-adaptive and heuristic adaptive policies . To compare generalization and specialization , we investigate RLQP ’ s performance in the settings where 1 ) the train and test sets of QPs come from the same class of problems , 2 ) the train set contains problems from a superset of classes contained in the test set , 3 ) the train set contains a subset , and 4 ) when the train and test sets are from distinct classes . In the results section we show that RLQP outperforms OSQP by up to 3x . The contributions of this paper are : • Two RL formulations to train policies that provide coarse ( scalar ) and fine ( vector ) grain updates to the internal parameters of a QP solver for faster convergence times • Policies trained jointly across QP problem classes or to specialize to specific classes • Experimental results showing that RLQP reduces convergence times by up to 3x and generalizes to different problem classes and outperform existing methods 2 Related Work . This work touches a number of related research areas , including convex optimization , ML-accelerated optimization , learning in 1st-order methods and RL . We summarize related work in Table 1 . Convex optimization Many researchers have proposed algorithms for quadratic programs , which generally fall into three classes : active set [ 47 ] , interior point [ 37 ] , and first-order methods . Of the active set and interior point solvers , perhaps the most well-known are Gurobi [ 20 ] and MOSEK [ 36 ] . Active-set solvers operate by iteratively adapting an active set of constraints based on the cost function gradient and dual variables [ 39 ] . Interior-point solvers iteratively introduce and vary barrier functions to represent constraints and solve unconstrained convex problems . We instead base this work on a first-order method solver , OSQP [ 44 ] . One of the advantages of OSQP over interior points solvers , is that it can readily be warm started from a near-by solution , as is common in many applications such as solving a sequential quadratic program [ 41 ] and solving QPs for model-predictive control . ML-accelerated combinatorial optimization Accelerating combinatorial optimization problems with deep learning has been explored with wide application [ 4 , 5 ] , including branch-and-bound for mixed-integer linear programming [ 1 , 27 ] , graph algorithms [ 9 ] and boolean satisfiability problems ( SAT ) [ 7 ] . Many combinatorial optimization problems have exponential search spaces and are NP-hard in a general setting . However , learning-augmented combinatorial algorithms utilize very different methods to RLQP as combinatorial problems have discrete search spaces . Learning in first-order methods Accelerating first-order methods with machine learning has gained considerable recent interest . Li and Malik [ 28 ] demonstrate a learned optimization algorithm outperforms common first-order methods for several convex problems and a small non-convex problem . Metz et al . [ 34 ] show a learned policy outperforms first-order methods when optimizing neural networks , but finds that directly learning parameter update values can be sensitive to exploding gradient problems . We avoid this instability during optimization by learning a policy to adapt parameters of the ADMM algorithm . Wei et al . [ 46 ] recently proposed an RL agent to tune parameters for an ADMM-based inverse imaging solver . Reinforcement Learning Overview Reinforcement learning ( RL ) algorithms include both onpolicy algorithms , such as Proximal Policy Optimization [ 42 ] , REINFORCE [ 45 ] , and IMPALA [ 11 ] , and off-policy algorithms , such as DQN [ 35 ] and Soft Actor Critic [ 21 ] . RLQP extends the offpolicy Twin-Delayed DDPG ( TD3 ) [ 13 ] , an actor-critic framework with an exploration policy for continuous action spaces that extends Deep Deterministic Policy Gradient ( DDPG ) algorithm [ 29 ] while addressing approximation errors . Furthermore , in one formulation of RLQP , we train a shared policy for multiple agents following an RL approach proposed by Huang et al . [ 23 ] . With this single policy , RLQP updates multiple parameters using state associated with each constraint of a QP . 3 Background . In this section , we summarize QPs , the OSQP solver , and an MDP formalization . 3.1 Quadratic Programs . A quadratic program with n variables and m constraints takes the form : minimize ( 1/2 ) xTPx+ qTx subject to l ≤ Ax ≤ u , where x ∈ Rn is the optimization variable , P is an n× n symmetric positive semi-definite matrix that defines the quadratic cost , q ∈ Rn defines the linear cost , A is an m × n matrix that defines the m linear constraints , and l , u ∈ Rm are the constraint ’ s lower and upper bounds . Here , ≤ is an element-wise less-than-or-equal-to operator . In this form , to specify an equality constraint , the lower and upper bounds are set to the same value , and to specify a constraint unbounded from one side , a sufficiently large value ( or ±∞ ) is specified for the other side . 3.2 First-Order QP Solver Algorithm . The solver we speed up is OSQP , which uses a first-order ADMM method to solve QPs . We summarize OSQP here . Given a QP , OSQP first forms a KKT matrix ( below ) , then iteratively refines a solution from an initialization point for vectors x ( 0 ) ∈ Rn , y ( 0 ) ∈ Rm , and z ( 0 ) ∈ Rm , where the superscript in parenthesis refers to the iteration . Each iteration computes the values for the k + 1 iterates by solving following linear system ( e.g. , with an LDLT solver ) : [ P + σI AT A diag ( ρ ) −1 ] ︸ ︷︷ ︸ KKT matrix [ x ( k+1 ) v ( k+1 ) ] = [ σx ( k ) − q z ( k ) − diag ( ρ ) −1y ( k ) ] ( 1 ) and then performing the following updates : z̃ ( k+1 ) ← z ( k ) + diag ( ρ ) −1 ( v ( k+1 ) − y ( k ) ) z ( k+1 ) ← Π [ l , u ] ( z̃ ( k+1 ) + diag ( ρ ) −1y ( k ) ) y ( k+1 ) ← y ( k ) + diag ( ρ ) ( z̃ ( k+1 ) − z ( k+1 ) ) , where σ ∈ R+ and ρ ∈ Rm+ are regularization and step-size parameters , and Π [ l , u ] : Rm → Rm projects its argument on the constraint bounds . We use the notation diag : Rm → Sm to denote the operator that maps a vector to a diagonal matrix . We define the primal and the dual residual vectors as ξ ( k ) primal = Ax ( k ) − z ( k ) , and ξ ( k ) dual = Px ( k ) + q +AT y ( k ) . When the primal and dual residual vectors are small enough in norm after k iterations , x ( k+1 ) and y ( k+1 ) are primal and dual ( approximate ) solutions to the QP . Internally , OSQP has a single scalar ρ̄ that it uses to form ρ according to the following formula : ρ = ρ̄ h1 ... hm , where hi = { 1 if li ̸= ui ( inequality constraints ) 103 if li = ui ( equality constraints ) , ( 2 ) where the subscript i denotes the i-th coefficient of h , and the bounds l and u . Periodically , between ADMM iterations , OSQP will adapt the value of ρ̄ . The existing handcrafted formula for adapting ρ attempts to balance between primal and dual residuals , by setting ρ̄ ( k+1 ) ← ρ̄ ( k ) √ ∥ξprimal∥∞/∥ξdual∥∞ . Empirically , adapting ρ between iterations can speed up the convergence rate . 3.3 Multi-Agent Single-Policy MDP . In a Markov Decision Process ( MDP ) , an agent can be in any state s ∈ S , take an action a ∈ A , and with the transition dynamics function , T ( · | s , a ) , transitions from state s to state s′ after taking action a . The agent receives a reward R : S ×A → R for transitioning from s to s′ by taking action a . Given a tuple ( S , A , T , R , γ ) , the MDP optimization objective is to find a policy πθ : S → A , parameterized by θ , that maximizes the expected cumulative reward E [ ∑∞ t=0 γ trt ] , where rt is the reward at time t and γ ∈ [ 0 , 1 ) is a discount factor . We also formulate a multi-agent single-policy MDP setting in which m agents collaborate in a shared environment in state senv ∈ Senv . At each time step , each collaborating agent ( CA ) i has its own state si ∈ Sca , action ai ∈ Aca , and observations oi ∈ O , but , for computation feasibility , share a single policy πθ : Sca → Aca . State transitions for the environment and all m agents occur simultaneously according to a state transition function T : Senv × Smca ×Amca → Senv × Sca m and result in a single shared reward R : Senv×Smca×Amca → R and discount factor . The objective is to find a single shared policy πθ that maximizes the expected cumulative reward . This can be thought of as a special case of a multi-agent MDP [ 31 ] or Markov game [ 30 ] , and we adapt a formulation from Huang et al . [ 23 ] .

Summary: The paper introduces a RL formulation and training pipeline to accelerate quadratic programming. The main idea is based on learning the multi-dimensional multiplier for ADMM. The paper shows some empirical performance gains over a number of setups.  ====== updated post rebuttal =====  Thanks the reviewers for the replies. Aggregating all information so far, I think the major weakness of the paper is the following:  It does not provide a clear reason why formulating the whole problem as a multi-agent Markov game is useful (I am not fully convinced by the rebuttal). The algorithm I can see is still essentially a single agent policy but with a specialized architecture.  Now that after exchanging messages with the author a few times, I have understood how the detailed architecture works, and I am convinced about my point above. I would suggest that the authors present their architecture (such as mapping from R^6 to R^1) in a clear way, so as to make the architecture easier to understand. Also, I would like to see less stress on the Markov games, because it does not provide much gains -- once again, I'd like to frame it as a form of specialized architecture for single agent problem.  The paper is overall an application of RL to optimization problem. The gains are ok though the idea-wise contributions are not very novel (no extremely novel architecture or formulation).  Still, overall I think the paper provides a solid contribution and I am willing to raise my eval to 6.