Accelerating Quadratic Optimization with Reinforcement Learning

1 Introduction . Solving quadratic programs ( QPs ) efficiently is critical to applications in finance , robotic control and operations research . While state-of-the-art interior-point methods scale poorly with problem dimensions , first-order methods for solving QPs typically require thousands of iterations . Moreover , real-time control applications have tight latency constraints for solvers [ 33 ] . Therefore , it is important to develop efficient heuristics to solve QPs in fewer iterations . The Alternating Direction Method of Multipliers ( ADMM ) [ 6 , 15 , 18 ] is an efficient first-order optimization algorithm , and is the basis for the widely used and state-of-the art Operator-Splitting QP ( OSQP ) solver [ 44 ] . ADMM performs a linear solve on a matrix based on the optimality conditions of the QP to generate a step direction , and then projects the step onto the constraint bounds . While state-of-the-art , the ADMM algorithm has numerous hyperparameters that must be tuned with heuristics to regularize and control optimization . Most importantly , the step size parameter ρ has considerable impact on the convergence rate . However , is still unclear how to select ρ before attempting the QP solution . While some theoretical works compute the optimal ρ [ 17 ] , they rely on solving semidefinite optimization problems which are much harder than solving the QP itself . Alternatively , some heuristics introduce “ feedback ” by adapting ρ throughout optimization in order to balance primal and dual residuals [ 44 , 6 , 22 ] . We propose RLQP ( see Fig . 1 ) , an accelerated QP solver based on OSQP that uses reinforcement learning ( RL ) to adapt the internal parameters of the ADMM algorithm between iterations to minimize solve times . An RL algorithm learns a policy πθ : S → A , parameterized by θ ( e.g. , the weights of a neural network ) , that maps states in a set S to actions in setA such that the selected action maximizes the accumulated reward r. To train the policy for RLQP , we define S to be the internal state of the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . QP solver ( e.g. , the constraint bounds , the primal and dual estimates ) , A to be the adaptation to the internal parameter ( ρ ) vector , and r to minimize the number of ADMM iterations taken . RLQP ’ s policy can be trained either jointly across general classes of QPs or with respect to a specific class . The general version of RLQP is trained once on a broad class of QPs and can be used out-ofthe-box on new problems . The specialized version of RLQP is trained on a specific class of problems that the solver will repeatedly encounter . While this requires additional setup and training time , it is useful when QPs will be repeatedly solved in application ( e.g. , in a 100 Hz control loop ) . In experiments , we train RLQP on a set of randomized QPs , and compare convergence rates of RLQP to non-adaptive and heuristic adaptive policies . To compare generalization and specialization , we investigate RLQP ’ s performance in the settings where 1 ) the train and test sets of QPs come from the same class of problems , 2 ) the train set contains problems from a superset of classes contained in the test set , 3 ) the train set contains a subset , and 4 ) when the train and test sets are from distinct classes . In the results section we show that RLQP outperforms OSQP by up to 3x . The contributions of this paper are : • Two RL formulations to train policies that provide coarse ( scalar ) and fine ( vector ) grain updates to the internal parameters of a QP solver for faster convergence times • Policies trained jointly across QP problem classes or to specialize to specific classes • Experimental results showing that RLQP reduces convergence times by up to 3x and generalizes to different problem classes and outperform existing methods 2 Related Work . This work touches a number of related research areas , including convex optimization , ML-accelerated optimization , learning in 1st-order methods and RL . We summarize related work in Table 1 . Convex optimization Many researchers have proposed algorithms for quadratic programs , which generally fall into three classes : active set [ 47 ] , interior point [ 37 ] , and first-order methods . Of the active set and interior point solvers , perhaps the most well-known are Gurobi [ 20 ] and MOSEK [ 36 ] . Active-set solvers operate by iteratively adapting an active set of constraints based on the cost function gradient and dual variables [ 39 ] . Interior-point solvers iteratively introduce and vary barrier functions to represent constraints and solve unconstrained convex problems . We instead base this work on a first-order method solver , OSQP [ 44 ] . One of the advantages of OSQP over interior points solvers , is that it can readily be warm started from a near-by solution , as is common in many applications such as solving a sequential quadratic program [ 41 ] and solving QPs for model-predictive control . ML-accelerated combinatorial optimization Accelerating combinatorial optimization problems with deep learning has been explored with wide application [ 4 , 5 ] , including branch-and-bound for mixed-integer linear programming [ 1 , 27 ] , graph algorithms [ 9 ] and boolean satisfiability problems ( SAT ) [ 7 ] . Many combinatorial optimization problems have exponential search spaces and are NP-hard in a general setting . However , learning-augmented combinatorial algorithms utilize very different methods to RLQP as combinatorial problems have discrete search spaces . Learning in first-order methods Accelerating first-order methods with machine learning has gained considerable recent interest . Li and Malik [ 28 ] demonstrate a learned optimization algorithm outperforms common first-order methods for several convex problems and a small non-convex problem . Metz et al . [ 34 ] show a learned policy outperforms first-order methods when optimizing neural networks , but finds that directly learning parameter update values can be sensitive to exploding gradient problems . We avoid this instability during optimization by learning a policy to adapt parameters of the ADMM algorithm . Wei et al . [ 46 ] recently proposed an RL agent to tune parameters for an ADMM-based inverse imaging solver . Reinforcement Learning Overview Reinforcement learning ( RL ) algorithms include both onpolicy algorithms , such as Proximal Policy Optimization [ 42 ] , REINFORCE [ 45 ] , and IMPALA [ 11 ] , and off-policy algorithms , such as DQN [ 35 ] and Soft Actor Critic [ 21 ] . RLQP extends the offpolicy Twin-Delayed DDPG ( TD3 ) [ 13 ] , an actor-critic framework with an exploration policy for continuous action spaces that extends Deep Deterministic Policy Gradient ( DDPG ) algorithm [ 29 ] while addressing approximation errors . Furthermore , in one formulation of RLQP , we train a shared policy for multiple agents following an RL approach proposed by Huang et al . [ 23 ] . With this single policy , RLQP updates multiple parameters using state associated with each constraint of a QP . 3 Background . In this section , we summarize QPs , the OSQP solver , and an MDP formalization . 3.1 Quadratic Programs . A quadratic program with n variables and m constraints takes the form : minimize ( 1/2 ) xTPx+ qTx subject to l ≤ Ax ≤ u , where x ∈ Rn is the optimization variable , P is an n× n symmetric positive semi-definite matrix that defines the quadratic cost , q ∈ Rn defines the linear cost , A is an m × n matrix that defines the m linear constraints , and l , u ∈ Rm are the constraint ’ s lower and upper bounds . Here , ≤ is an element-wise less-than-or-equal-to operator . In this form , to specify an equality constraint , the lower and upper bounds are set to the same value , and to specify a constraint unbounded from one side , a sufficiently large value ( or ±∞ ) is specified for the other side . 3.2 First-Order QP Solver Algorithm . The solver we speed up is OSQP , which uses a first-order ADMM method to solve QPs . We summarize OSQP here . Given a QP , OSQP first forms a KKT matrix ( below ) , then iteratively refines a solution from an initialization point for vectors x ( 0 ) ∈ Rn , y ( 0 ) ∈ Rm , and z ( 0 ) ∈ Rm , where the superscript in parenthesis refers to the iteration . Each iteration computes the values for the k + 1 iterates by solving following linear system ( e.g. , with an LDLT solver ) : [ P + σI AT A diag ( ρ ) −1 ] ︸ ︷︷ ︸ KKT matrix [ x ( k+1 ) v ( k+1 ) ] = [ σx ( k ) − q z ( k ) − diag ( ρ ) −1y ( k ) ] ( 1 ) and then performing the following updates : z̃ ( k+1 ) ← z ( k ) + diag ( ρ ) −1 ( v ( k+1 ) − y ( k ) ) z ( k+1 ) ← Π [ l , u ] ( z̃ ( k+1 ) + diag ( ρ ) −1y ( k ) ) y ( k+1 ) ← y ( k ) + diag ( ρ ) ( z̃ ( k+1 ) − z ( k+1 ) ) , where σ ∈ R+ and ρ ∈ Rm+ are regularization and step-size parameters , and Π [ l , u ] : Rm → Rm projects its argument on the constraint bounds . We use the notation diag : Rm → Sm to denote the operator that maps a vector to a diagonal matrix . We define the primal and the dual residual vectors as ξ ( k ) primal = Ax ( k ) − z ( k ) , and ξ ( k ) dual = Px ( k ) + q +AT y ( k ) . When the primal and dual residual vectors are small enough in norm after k iterations , x ( k+1 ) and y ( k+1 ) are primal and dual ( approximate ) solutions to the QP . Internally , OSQP has a single scalar ρ̄ that it uses to form ρ according to the following formula : ρ = ρ̄ h1 ... hm , where hi = { 1 if li ̸= ui ( inequality constraints ) 103 if li = ui ( equality constraints ) , ( 2 ) where the subscript i denotes the i-th coefficient of h , and the bounds l and u . Periodically , between ADMM iterations , OSQP will adapt the value of ρ̄ . The existing handcrafted formula for adapting ρ attempts to balance between primal and dual residuals , by setting ρ̄ ( k+1 ) ← ρ̄ ( k ) √ ∥ξprimal∥∞/∥ξdual∥∞ . Empirically , adapting ρ between iterations can speed up the convergence rate . 3.3 Multi-Agent Single-Policy MDP . In a Markov Decision Process ( MDP ) , an agent can be in any state s ∈ S , take an action a ∈ A , and with the transition dynamics function , T ( · | s , a ) , transitions from state s to state s′ after taking action a . The agent receives a reward R : S ×A → R for transitioning from s to s′ by taking action a . Given a tuple ( S , A , T , R , γ ) , the MDP optimization objective is to find a policy πθ : S → A , parameterized by θ , that maximizes the expected cumulative reward E [ ∑∞ t=0 γ trt ] , where rt is the reward at time t and γ ∈ [ 0 , 1 ) is a discount factor . We also formulate a multi-agent single-policy MDP setting in which m agents collaborate in a shared environment in state senv ∈ Senv . At each time step , each collaborating agent ( CA ) i has its own state si ∈ Sca , action ai ∈ Aca , and observations oi ∈ O , but , for computation feasibility , share a single policy πθ : Sca → Aca . State transitions for the environment and all m agents occur simultaneously according to a state transition function T : Senv × Smca ×Amca → Senv × Sca m and result in a single shared reward R : Senv×Smca×Amca → R and discount factor . The objective is to find a single shared policy πθ that maximizes the expected cumulative reward . This can be thought of as a special case of a multi-agent MDP [ 31 ] or Markov game [ 30 ] , and we adapt a formulation from Huang et al . [ 23 ] .

Summary: In this paper, the authors proposed a novel method to accelerate the QP  (quadratic programming) solving process using reinforcement learning. In their formulation, the state of the problem is the internal variables of the QP solver (in this case, OSQP's internal variables), the action space consists of the ADMM step size vector rho. Since the vector/matrix sizes of the QP can change between different problems, the authors used a multi-agent single-policy formulation, so each agent only observes the internal states that are relevant for a single coefficient in the step vector rho. The authors demonstrated that after training, their RLQP can converge faster for different classes of problems in the benchmark suite, especially if the problem is cold-started.