Global Convergence of Online Optimization for Nonlinear Model Predictive Control

1 Introduction . We consider the following time-varying nonlinear optimal control problem P ( x̄0 ) : min x , u N−1∑ k=0 gk ( xk , uk ) + gN ( xN ) , s.t . xk+1 = fk ( xk , uk ) , k ∈ [ N − 1 ] , ( 1 ) x0 = x̄0 , where xk ∈ Rnx are the state variables , uk ∈ Rnu are the control variables , gk : Rnx × Rnu → R+ ( gN : Rnx → R+ ) are the nonnegative cost functions , fk : Rnx × Rnu → Rnx are the dynamical constraint functions , x̄0 is the given initial state , and N is the horizon length . We suppose that gk , fk are twice continuously differentiable and , without loss of generality , that the origin is a steady state , i.e . fk ( 0,0 ) = 0 , and is also a stationary point with zero loss , i.e . gk ( 0,0 ) = 0 and ∇gk ( 0,0 ) = 0 . The well-known linear quadratic regulator ( LQR ) satisfies this setup and the same setup is commonly used in the literature [ 8 ] . Problem ( 1 ) is also called dynamic program in control community , which has a close relation to reinforcement learning ( RL ) ( see [ 6 ] ) . The main difference is that the dynamic fk in our paper is known , which is the case in many industrial applications and model-based RL studies . In modern applications such as energy and autonomous control , N tends to be extremely large or even infinity , which stimulates the interest of solving ( 1 ) in real time . Let fk ( xk , uk ) = fk ( xk , uk ; dk ) where dk is data at stage k , then a more realistic setting is to collect dk as a data stream . That is , instead of knowing f0 all the way to fN−1 and solving ( 1 ) offline , we only know fk sequentially and have to solve ( 1 ) online . Model predictive control ( MPC ) , also known as receding horizon 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . control , is an important feedback control technique that is used in online optimization of control problems . Given the current system state , MPC looks ahead multiple stages on the prediction horizon , solves the subproblem on the prediction horizon to optimality , and provides the first optimal control action to the system as the feedback . Then the system evolves to a new state based on the control action and the procedure is repeated . However , the classical nonlinear MPC is being regarded as a theoretical concept rather than a practical strategy . It is not as widely used as linear MPC , and is mostly investigated in slow dynamics . This is due to the intensive computations of solving the control subproblems when we have nonlinear models . Fortunately , the design of real-time iteration ( RTI ) and its predecessors [ 26 , 37 ] have enabled RTI-based nonlinear MPC to be applied on fast dynamics and achieve good performance [ 48 , 4 , 14 , 38 , 15 , 24 , 18 , 22 , 17 , 20 , 16 , 19 , 39 , 1 ] . Instead of using optimal feedback control , RTI schemes compute a cheap approximation of control action and provide back to the system as fast as possible . The existing RTI schemes perform a single , full Newton step for each subproblem , and the output iterate of the previous subproblem is used to warmly initialize the current subproblem [ 8 , 41 , 42 ] . Thus , RTI schemes significantly reduce the computations of each subproblem , which makes them succeed in modern industrial applications . RTI schemes exploit the fact that two successive subproblems are closely related though different , so that the iteration can achieve the convergence “ on the fly ” ( i.e . the error goes to zero as horizon moves forward ) . However , the RTI-based MPC is challenging to analyze . The main difficulty lies in the fact that the prediction horizon is shifted each time . Each subproblem is appended by a new stage that has never been scanned before and may introduce perturbation into the system . Different RTI-based MPC schemes have been studied [ 9 , 12 , 23 ] , with an improved local analysis established in [ 31 ] recently . Our paper complements the literature on RTI-based MPC by investigating its global convergence . We emphasize that by global convergence we mean the convergence to stationary points ( i.e . KKT points ) for any initialization , instead of the convergence to global optimality . The latter is not achievable for nonlinear problems without strong assumptions [ 34 ] . Instead of using a full Newton step , we design an adaptive scheme to involve the stepsize selection procedure . In fact , the stepsize selection has been suggested as a way to improve the global behavior of RTI in shrinking ( not moving ) horizons in an old paper [ 25 ] . The authors implemented a ` 1 penalized merit function1 with a watchdog line search . However , the global convergence theory remains open . Different from the suggestion of [ 25 ] , we use a differentiable exact augmented Lagrangian as the merit function . This function mainly has two differences to the ` 1 penalized merit function : ( i ) it depends on both primal and dual variables , so that dual variables are updated using the same stepsize as the primal variables ( ` 1 merit function only updates the primal variables ) ; ( ii ) a simple backtracking line search can effectively overcome Maratos effect [ 29 ] and accept the unit stepsize locally ( ` 1 merit function requires a more time-consuming line search method such as watchdog or second-order correction ) . We conduct experiments to show the superiority of augmented Lagrangian over ` 1 penalized function , especially for time-varying problems . Moreover , our algorithm adaptively selects the penalty parameters of augmented Lagrangian on the fly . When the unit stepsize is accepted , the algorithm simply boils down to the standard RTI-based MPC , and the existing local theories [ 8 , 31 ] apply seamlessly . We notice that the same merit function has been employed in a recent work [ 32 ] to achieve the global convergence for solving ( 1 ) in offline fashion , with a horizon decomposition strategy . In that work , the authors showed the unit stepsize is accepted even with an inexact Newton direction . Our paper is in online regime and does not investigate the local behavior . Related work and contribution : Our work is closely related to a growing literature on online optimization via RTI . The theories of RTI have been investigated over the past decade . [ 10 ] and [ 8 ] established the local convergence of RTI in shrinking horizons and moving horizons , respectively . When we have different models , [ 3 , 42 , 44 , 35 , 36 ] all showed similar results . A different view of RTI-based MPC is via the parametric optimization , where subproblems are parameterized by a continuous time index and a single Newton step is performed at each time instant . Similar stability results in this setup are shown in [ 11 , 45 , 13 ] . Recently , [ 31 ] studied a lag-L RTI-based MPC ( i.e . one shifts L stages each time ) , and proved that RTI iterates on each stage converge linearly with a rate decaying exponentially in the prediction horizon length . Their analyses rely on the sensitivity analysis of ( 1 ) in [ 30 , 33 ] . An important consequence of convergence of online iteration is the dynamic regret analysis [ 47 , 27 , 7 , 43 ] . For example , the convergence rate in [ 31 ] and the regret order in [ 27 ] are both exponential in prediction horizon length . This is due to the widely-used Lipschitz continuity assumption on the objective . 1The merit function refers to the function that is used in the line search . However , all above theories are local results . The algorithms always take a full ( Newton ) step for each subproblem , without involving any stepsize selection procedure . Thus , the analyses require to have an initial point that is sufficiently close to the local solution , or require that the first Newton step is sufficiently small . Obviously , for Problem ( 1 ) , when ‖x̄0‖ is small enough , the origin is a good initial point and RTI iterates stably track the origin . However , if ‖x̄0‖ is large or we may want to track a nontrivial MPC policy , the origin is not a reasonable choice for the initial point . Based on these considerations , the global convergence of RTI addressed in this paper complements the aforementioned rich local analyses . Again , by global convergence we mean that the KKT residuals of subproblems go to zero as horizon moves forward for any initialization . The known , local stability of tracking error of RTI is not addressed in this paper , but referred to [ 45 , Theorem 4.2 ] . The challenge of our analysis is that : the problem varies with time instants and only a single Newton step is performed at each time instant . Such a setup contrasts to online Newton ( with line search ) analysis , where a single Newton step targeting the same problem is performed . A key , technical step of our analysis is to show that the exact augmented Lagrangian merit function keeps decreasing as horizons shift . To our knowledge , this is the first paper studying the global behavior of online iteration under RTI setting , via a tool of the exact augmented Lagrangian . Another contribution to nonlinear MPC is that we consider time-varying dynamics , while most of the literature considered time-invariant dynamics ( e.g . [ 8 , 21 , 40 ] ) . The subproblems of time-invariant dynamics only differ in the initial state , which results in a much simpler analysis . We notice that [ 31 ] allowed time-varying dynamics , but required to use a larger lag when shifting the horizon . We allow lag one instead . Notations : For two vectors a , b , ( a ; b ) denotes a long column vector that stacks a and b together . For two scalars a , b , a ∨ b = max ( a , b ) . Without specification , ‖ · ‖ denotes the ` 2 norm for vectors and operator norm for matrices . We also reserve the following notations . 0 denotes zero vector or matrix , and I denotes the identity matrix . Their dimensions are clear from the context . M denotes the prediction horizon length , t indexes the sampling time , and zk = ( xk ; uk ) ∈ Rnz with nz = nx+nu denotes the state-control pair at stage k . 2 Preliminaries . We introduce MPC subproblems formulation , the Newton step , and the augmented Lagrangian merit function in this section . At time t , the prediction horizon shifts to [ t , Mt ] with Mt = t+M denoting the last stage , and the corresponding subproblem Pt ( x̄t ) is given by Pt ( x̄t ) : minxt : Mt ut : Mt−1 Mt−1∑ k=t gk ( xk , uk ) + gMt ( xMt ,0 ) + µ 2 ‖xMt‖2 , s.t . xk+1 = fk ( xk , uk ) , k ∈ [ t , Mt − 1 ] , ( 2 ) xt = x̄t . Here , µ is a quadratic penalty parameter independent from t ; x̄t is the current system state , which is given and determined by the control feedback provided to the system before t. The formulation ( 2 ) is used in [ 31 ] for local analysis , while alternative subproblem formulations are available as well [ 8 , 23 ] . Given subproblem ( 2 ) , RTI schemes perform a single Newton step for Pt ( x̄t ) and move to the next time instant . To ease notations , we let x̃t = xt : Mt and ũt = ut : Mt−1 denote the state and control variables of the t-th subproblem , respectively . We also let z̃t = zt : Mt = ( zt ; . . . ; zMt ) denote the whole ordered state-control variables , with slightly abusing the notation zMt to let zMt = xMt . We sometimes also express z̃t = ( x̃t , ũt ) when we specify each component of z̃t . Then , the Lagrangian function of ( 2 ) is Lt ( z̃t , λ̃t ; x̄t ) = Mt−1∑ k=t gk ( xk , uk ) + gMt ( xMt ,0 ) + µ 2 ‖xMt‖2 + Mt−1∑ k=t λTk ( xk+1 − fk ( xk , uk ) ) + λ T t−1 ( xt − x̄t ) , where λ̃t = λt−1 : Mt−1 is the dual vector . Here , λk ∈ Rnx is associated to the k-th constraint of ( 2 ) for k ∈ [ t , Mt − 1 ] , and λt−1 ∈ Rnx is associated to the initial state constraint . Given the input iterate ( z̃0t , λ̃ 0 t ) , the Newton direction ( ∆z̃t , ∆λ̃t ) is solved by the KKT system ( Bt ( Gt ) T Gt 0 ) ( ∆z̃t ∆λ̃t ) = − ( ∇z̃tLt ∇λ̃tL t ) , ( 3 ) where Bt is an approximation of the Hessian Ht = ∇2z̃tL t ; Gt = ∇λ̃tz̃tL t is the Jacobian matrix ; and all matrices in ( 3 ) are evaluated at ( z̃0t , λ̃ 0 t ) . The left hand side matrix is called the KKT matrix . In what follows , Ht , Bt , Gt are always evaluated at the input iterate ( z̃0t , λ̃ 0 t ) if no evaluation point is specified . Taking a closer look at ( 3 ) ( cf . Appendix A ) , we note that the KKT matrix and ∇z̃tLt are independent from the given initial state x̄t . This is a crucial observation . In reality , the realization of the system state may lag behind the sampling time , since the system has to apply the feedback control to evolve . Thus , x̄t may not be known as soon as the input iterate ( z̃0t , λ̃ 0 t ) . However , a promising property of RTI is that most of computations of ( 3 ) can be prepared without knowing x̄t . Given the Newton direction ( ∆z̃t , ∆λ̃t ) from ( 3 ) , we update the iterate by involving a stepsize αt ( z̃1t λ̃ 1 t ) = ( z̃0t λ̃ 0 t ) + αt ( ∆z̃t ∆λ̃t ) . ( 4 ) In our paper , αt is selected by forcing Armijo condition hold on a differentiable exact augmented Lagrangian , defined as Ltη ( z̃t , λ̃t ; x̄t ) =Lt ( z̃t , λ̃t ; x̄t ) + η1 2 ∥∥∥∇λ̃tLt ( z̃t , λ̃t ; x̄t ) ∥∥∥2 + η22 ∥∥∥∇z̃tLt ( z̃t , λ̃t ; x̄t ) ∥∥∥2 . ( 5 ) Here η = ( η1 , η2 ) are parameters of the quadratic penalties . ( 5 ) is called exact augmented Lagrangian since one can show that , for sufficiently large η1 and sufficiently small η2 , a strict local minimizer of Ltη ( · ) is also a strict local minimizer of Problem ( 2 ) and vice versa ( see [ 2 , Proposition 4.5 ] ) . When η2 = 0 , Ltη ( · ) reduces to standard augmented Lagrangian , while it does not enjoy exactness property anymore , and may cause issues when it is used as merit function . For simplicity , we sometimes suppress “ exact ” and call Ltη ( · ) augmented Lagrangian . Based on ( 5 ) , αt is selected by making the following Armijo condition , Ltη ( z̃1t , λ̃ 1 t ; x̄t ) ≤ Ltη ( z̃0t , λ̃ 0 t ; x̄t ) + αtβ ( ∇z̃tLtη ( z̃ 0 t , λ̃ 0 t ; x̄t ) ∇λ̃tL t η ( z̃ 0 t , λ̃ 0 t ; x̄t ) ) T ( ∆z̃t ∆λ̃t ) , ( 6 ) hold for a prespecified parameter β ∈ ( 0 , 1 ) . Note that , by mean value theory , αt can always be found by backtracking line search provided the inner product in ( 6 ) is negative ( cf . Theorem 4.5 ) .

Summary: This paper studies a real-time iteration scheme for solving nonlinear MPC. The paper provides an adaptive way of Newton stepsize selection and establishes global convergence. The paper also provides abundant numerical results.