Infinite-Horizon Differentiable Model Predictive Control

1 INTRODUCTION . Imitation Learning ( IL , Osa et al. , 2018 ) aims at reproducing an existing control policy by means of a function approximator and can be used , for instance , to hot-start reinforcement learning . Effective learning and generalisation to unseen data are paramount to IL success , especially in safety critical applications . Model Predictive Control ( MPC , Maciejowski , 2000 ; Camacho & Bordons , 2007 ; Rawlings & Mayne , 2009 ; Kouvaritakis & Cannon , 2015 ; Gallieri , 2016 ; Borrelli et al. , 2017 ; Raković & Levine , 2019 ) is the most successful advanced control methodology for systems with hard safety constraints . At each time step , a finite horizon forecast is made from a predictive model of the system and the optimal actions are computed , generally relying on convex constrained Quadratic Programming ( QP , Boyd & Vandenberghe , 2004 ; Bemporad et al. , 2000 ) . Stability of the MPC in closed loop with the physical system requires the solution of a simpler unconstrained infinite horizon control problem ( Mayne et al. , 2000 ) which results in a value function ( terminal cost and constraint ) and a candidate terminal controller to be accounted for in the MPC forecasting . For Linear Time Invariant ( LTI ) models and quadratic costs , this means solving ( offline ) a Riccati equation ( Kalman , 2001 ) or a linear matrix inequality ( Boyd et al. , 1994 ) . Under these conditions , an MPC controller will effectively control a system , up to a certain accuracy , provided that uncertainties in the model dynamics are limited ( Limon et al. , 2009 ) . Inaccuracies in the MPC predictions can reduce its effectiveness ( and robustness ) as the forecast diverges from the physical system trajectory over long horizons . This is particularly critical in applications with both short and long-term dynamics and it is generally addressed , for instance in robust MPC ( Richards , 2004 ; Raković et al. , 2012 ) , by using a controller to pre-stabilise the predictions . This paper presents an infinite-horizon differentiable linear quadratic MPC that can be learned using gradient-based methods . In particular , the learning method uses an MPC controller where the terminal cost and terminal policy are the solution of an unconstrained infinite-horizon Linear Quadratic Regulator ( LQR ) . A closed-form solution for the derivative of the Discrete-time Algebraic Riccati Equation ( DARE ) associated with the LQR is presented so that the stationary solution of the forward pass is fully differentiable . This method allows analytical results from control theory ∗Corresponding author . to be used to determine the stabilizing properties of the learned controller when implemented in closed-loop . Once the unconstrained LQR is computed , the predictive model is pre-stabilised using a linear state-feedback controller to improve the conditioning of the QP and the numerical accuracy of the MPC solution and gradients . The proposed algorithm successfully learns an MPC with both local stability and intrinsic robustness guarantees under small model uncertainties . Contributions This paper provides a framework for correctly learning an infinite-horizon , LTI quadratic MPC using recent developments in differentiable QPs ( Amos & Kolter , 2017 ) and principles from optimal control ( Blanchini & Miani , 2007 ) . A primary contribution is that the Discrete-time Algebraic Riccati Equation ( DARE ) is used to provide infinite-horizon optimality and stability , and an analytical derivative of the solution of the DARE is derived so that differentiation-based optimization can be used for training . This connects known results on MPC stability ( Limon et al. , 2003 ; 2009 ) and on infinite-horizon optimality ( Scokaert & Rawlings , 1998 ) to imitation learning ( Osa et al. , 2018 ) . A further contribution is the MPC control formulation : a pre-stabilizing linear state-feedback controller is implemented from the solution of the DARE , and then the total control input is obtained as a perturbation of the feedback control law from the solution of a convex QP . The pre-stabilizing controller ensures that the QP is well conditioned and promotes a highly accurate global solution , which in turn ensures that the gradients calculated in the backwards pass are accurate . Additionally , an augmented Lagrangian penalty method is used to enforce constraints on state and control input . This approach ensures that the hard constraints are strictly enforced if the penalty term is sufficiently large , and also guarantees that the MPC problem is feasible throughout the training process . These contributions are in contrast to ( Amos et al. , 2018 ) which did not consider state constraints , and implemented a differential dynamic programming ( Tassa et al. , 2014 ) method to solve the MPC optimization for which convergence could not be guaranteed . The framework is implemented on a set of second order mass-spring-damper systems and a vehicle platooning model , where it is demonstrated that the infinite horizon cost can be learned and the hard constraints can be guaranteed using a short finite prediction horizon . Notation In : = n × n identity matrix . Om×n : = m × n matrix of zeros . 0n : = a vector of n zeros . 1n : = a vector of n ones . All inequalities ≤ and ≥ are considered element-wise in the context of vectors . ρ ( A ) : = largest absolute eigenvalue of given matrix A. vec : Rm×n 7→ Rmn is defined as vec ( [ c1 · · · cn ] ) : = ( c1 , · · · , cn ) , i.e . the columns of a matrix stacked into a vector . For a matrix A ∈ Rm×n , the Vm , n ∈ Rmn×mn permutation matrix is implicitly defined by Vm , nvecA : = vecA > . The Kronecker product , ⊗ , is defined as in ( Magnus & Neudecker , 1999 , pp . 440 ) . 2 DIFFERENTIABLE MPC . Linear quadratic MPC This paper considers linear time invariant systems of the form xt+dt = Axt +But , ( 1 ) where xt ∈ Rn is the system state , ut ∈ Rm is the control input , A ∈ Rn×n is the state transition matrix , B ∈ Rn×m is the input matrix , t ∈ R is the time , and dt ∈ R is the timestep ( assumed constant ) . The control problem for this system is to determine the sequence of values of ut that achieve a desired level of performance ( e.g . stability , frequency response , etc ... ) , and when the system is subject to hard constraints on control input , ut ∈ U , and state , xt ∈ X , ( or a combination of both ) , a well studied framework for controller synthesis is MPC . The principle of MPC is that the system ’ s control input and state are optimized over a finite prediction horizon , then the first element of the obtained control sequence is implemented at the current time step and the process is repeated ad infinitum . For linear MPC it is common to use a quadratic stage cost and box constraints on state and control ( x ≤ xk ≤ x and u ≤ uk ≤ u where u ≤ 0 ≤ u ) , so that at each time index t the vector of optimized control variables û ? is obtained from û ? 0 : N = argmin û 1 2 N−1∑ k=0 û > k Rûk + 1 2 N−1∑ k=1 x̂ > k Qx̂k + 1 2 x̂ > NQN x̂N + ku N−1∑ k=0 1 > mrk + kx N∑ k=1 1 > n sk s.t . x̂0 = xt , x̂k+1 = Ax̂k +Bûk , k ∈ { 0 , . . . , N − 1 } , u− rk ≤ ûk ≤ u+ rk and rk ≥ 0 , k ∈ { 0 , · · ·N − 1 } , x− sk ≤ x̂k ≤ x+ sk and sk ≥ 0 , k ∈ { 1 , . . . , N } , ( 2 ) where û0 : N is the predicted control trajectory , x̂ is the predicted state trajectory , R ∈ Rm×m 0 represents the stage cost on control input , Q ∈ Rn×n 0 represents the stage cost on state , QN ∈ Rn×n 0 represents the terminal cost on state , N ∈ N is the prediction horizon , rk ∈ Rm are slack variables for the control constraint , sk ∈ Rn are slack variables for the state constraint , and ku ∈ R > 0 and kx ∈ R > 0 represent the cost of control and state constraint violations . The variables s and r enforce the box constraints on state and control using the augmented Lagrangian method ( Nocedal & Wright , 2006 , §17.2 ) , and it can be shown that for sufficiently high kx and ku the constraints x ≤ xk ≤ x and u ≤ uk ≤ u can be exactly guaranteed ( Kerrigan & Maciejowski , 2000 ) ( i.e . s = r = 0 ) . The benefit of this approach is that it ensures that the MPC optimization is feasible at each iteration of the learning process , whilst still ensuring that the constraints are ‘ hard ’ . To close the MPC control loop , at each timestep , t , the first element of the optimized control sequence , û ? 0 , is implemented as ut . Pre-stabilised MPC If the control input is decomposed into ut = Kxt + δut , where K ∈ Rm×n is a stabilizing linear state-feedback matrix and δut is a perturbation to the feedback control , system ( 1 ) becomes xt+dt = ( A+BK ) xt +Bδut , ( 3 ) and problem ( 2 ) becomes δû ? 0 : N = argmin δû 1 2 N−1∑ k=0 ( Kx̂k + δûk ) > R ( Kx̂k + δûk ) + 1 2 N−1∑ k=1 x̂ > k Qx̂k + 1 2 x̂ > NQN x̂N + ku N−1∑ k=0 1 > mrk + kx N∑ k=1 1 > n sk s.t . x̂0 = xt , x̂k+1 = ( A+BK ) x̂k +Bδûk , k ∈ { 0 , . . . , N − 1 } , u− rk ≤ Kx̂k + δûk ≤ u+ rk and rk ≥ 0 , k ∈ { 0 , . . . , N − 1 } , x− sk ≤ x̂k ≤ x+ sk and sk ≥ 0 , k ∈ { 1 , . . . , N } , ( 4 ) so that û ? 0 = Kxt + δû ? 0 is implemented as ut . Using this decomposition , system ( 3 ) controlled with the solution of ( 4 ) is precisely equal to system ( 1 ) controlled with the solution of ( 2 ) , but problem ( 4 ) is preferable from a computational standpoint if A is open-loop unstable ( i.e . ρ ( A ) > 1 ) and N is ‘ large ’ , as this can lead to poor conditioning of the matrices defined in Appendix A . This is important in the context of differentiable MPC , as if A is being learned then there may be no bounds on its eigenvalues at any given iteration . MPC derivative . Problems ( 2 ) and ( 4 ) can be rearranged into the QP form ( details in Appendix A ) z ? = argmin z 1 2 z > Hz + q > z s.t . lb ≤Mz ≤ ub . ( 5 ) When z ? is uniquely defined by ( 5 ) , it can also be considered as the solution of an implicit function defined by the Karush-Kuhn-Tucker ( KKT ) conditions , and in Amos & Kolter ( 2017 ) it was demonstrated that it is possible to differentiate through this function to obtain the derivatives of z ? with respect to the parameters H , q , l , M , and u . 1 The MPC controller can then be used as a layer in a 1Note that ( 5 ) differs from the form presented in Amos & Kolter ( 2017 ) , and is instead the form of problem solved by the OSQP solver used in this paper . Appendix B demonstrates how to differentiate ( 5 ) using the solution returned by OSQP . neural network , and backpropagation can be used to determine the derivatives of an imitation cost function with respect to the MPC parameters Q , R , A , B , u , u , x , x , kx and ku . Imitation Learning . A possible use case of the derivative of a model predictive controller is imitation learning , where a subset of { cost function , system dynamics , constraints } are learned from observations of a system being controlled by an ‘ expert ’ . Imitation learning can be performed by minimizing the loss 1 T T∑ t=0 ‖ut : t+Ndt − û ? 0 : N ( xt ) ‖22 + β‖ŵt‖22 , ( 6 ) where ut is the measured control input , û ? 0 : N ( xt ) is the full MPC solution , and β ≥ 0 is a hyperparameter . It is assumed that both the learning algorithm and MPC controller have completely precise measurements of both the state and control input . The first term of ( 6 ) is the control imitation loss , and the second term penalises the one-step ahead prediction error ŵt = Axt + But − xt+dt . In practice , the prediction error loss might not be needed for the MPC to be learned correctly , however its use can be instrumental for stability , as discussed in the next section .

Summary: The paper shows how to use the Discrete-time Algebraic Riccati Equation (DARE) to provide infinite horizon stability & optimality to differentiable MPC learning.  The paper also shows how to use DARE to derive a pre-stabilizing (linear state-feedback) controller.  The paper provides a theoretical characterization of the problem setting, which shows that prior work on differentiable MPC learning may lead to unstable controllers without the proposed augmentations using DARE.