Global Convergence of Online Optimization for Nonlinear Model Predictive Control

1 Introduction . We consider the following time-varying nonlinear optimal control problem P ( x̄0 ) : min x , u N−1∑ k=0 gk ( xk , uk ) + gN ( xN ) , s.t . xk+1 = fk ( xk , uk ) , k ∈ [ N − 1 ] , ( 1 ) x0 = x̄0 , where xk ∈ Rnx are the state variables , uk ∈ Rnu are the control variables , gk : Rnx × Rnu → R+ ( gN : Rnx → R+ ) are the nonnegative cost functions , fk : Rnx × Rnu → Rnx are the dynamical constraint functions , x̄0 is the given initial state , and N is the horizon length . We suppose that gk , fk are twice continuously differentiable and , without loss of generality , that the origin is a steady state , i.e . fk ( 0,0 ) = 0 , and is also a stationary point with zero loss , i.e . gk ( 0,0 ) = 0 and ∇gk ( 0,0 ) = 0 . The well-known linear quadratic regulator ( LQR ) satisfies this setup and the same setup is commonly used in the literature [ 8 ] . Problem ( 1 ) is also called dynamic program in control community , which has a close relation to reinforcement learning ( RL ) ( see [ 6 ] ) . The main difference is that the dynamic fk in our paper is known , which is the case in many industrial applications and model-based RL studies . In modern applications such as energy and autonomous control , N tends to be extremely large or even infinity , which stimulates the interest of solving ( 1 ) in real time . Let fk ( xk , uk ) = fk ( xk , uk ; dk ) where dk is data at stage k , then a more realistic setting is to collect dk as a data stream . That is , instead of knowing f0 all the way to fN−1 and solving ( 1 ) offline , we only know fk sequentially and have to solve ( 1 ) online . Model predictive control ( MPC ) , also known as receding horizon 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . control , is an important feedback control technique that is used in online optimization of control problems . Given the current system state , MPC looks ahead multiple stages on the prediction horizon , solves the subproblem on the prediction horizon to optimality , and provides the first optimal control action to the system as the feedback . Then the system evolves to a new state based on the control action and the procedure is repeated . However , the classical nonlinear MPC is being regarded as a theoretical concept rather than a practical strategy . It is not as widely used as linear MPC , and is mostly investigated in slow dynamics . This is due to the intensive computations of solving the control subproblems when we have nonlinear models . Fortunately , the design of real-time iteration ( RTI ) and its predecessors [ 26 , 37 ] have enabled RTI-based nonlinear MPC to be applied on fast dynamics and achieve good performance [ 48 , 4 , 14 , 38 , 15 , 24 , 18 , 22 , 17 , 20 , 16 , 19 , 39 , 1 ] . Instead of using optimal feedback control , RTI schemes compute a cheap approximation of control action and provide back to the system as fast as possible . The existing RTI schemes perform a single , full Newton step for each subproblem , and the output iterate of the previous subproblem is used to warmly initialize the current subproblem [ 8 , 41 , 42 ] . Thus , RTI schemes significantly reduce the computations of each subproblem , which makes them succeed in modern industrial applications . RTI schemes exploit the fact that two successive subproblems are closely related though different , so that the iteration can achieve the convergence “ on the fly ” ( i.e . the error goes to zero as horizon moves forward ) . However , the RTI-based MPC is challenging to analyze . The main difficulty lies in the fact that the prediction horizon is shifted each time . Each subproblem is appended by a new stage that has never been scanned before and may introduce perturbation into the system . Different RTI-based MPC schemes have been studied [ 9 , 12 , 23 ] , with an improved local analysis established in [ 31 ] recently . Our paper complements the literature on RTI-based MPC by investigating its global convergence . We emphasize that by global convergence we mean the convergence to stationary points ( i.e . KKT points ) for any initialization , instead of the convergence to global optimality . The latter is not achievable for nonlinear problems without strong assumptions [ 34 ] . Instead of using a full Newton step , we design an adaptive scheme to involve the stepsize selection procedure . In fact , the stepsize selection has been suggested as a way to improve the global behavior of RTI in shrinking ( not moving ) horizons in an old paper [ 25 ] . The authors implemented a ` 1 penalized merit function1 with a watchdog line search . However , the global convergence theory remains open . Different from the suggestion of [ 25 ] , we use a differentiable exact augmented Lagrangian as the merit function . This function mainly has two differences to the ` 1 penalized merit function : ( i ) it depends on both primal and dual variables , so that dual variables are updated using the same stepsize as the primal variables ( ` 1 merit function only updates the primal variables ) ; ( ii ) a simple backtracking line search can effectively overcome Maratos effect [ 29 ] and accept the unit stepsize locally ( ` 1 merit function requires a more time-consuming line search method such as watchdog or second-order correction ) . We conduct experiments to show the superiority of augmented Lagrangian over ` 1 penalized function , especially for time-varying problems . Moreover , our algorithm adaptively selects the penalty parameters of augmented Lagrangian on the fly . When the unit stepsize is accepted , the algorithm simply boils down to the standard RTI-based MPC , and the existing local theories [ 8 , 31 ] apply seamlessly . We notice that the same merit function has been employed in a recent work [ 32 ] to achieve the global convergence for solving ( 1 ) in offline fashion , with a horizon decomposition strategy . In that work , the authors showed the unit stepsize is accepted even with an inexact Newton direction . Our paper is in online regime and does not investigate the local behavior . Related work and contribution : Our work is closely related to a growing literature on online optimization via RTI . The theories of RTI have been investigated over the past decade . [ 10 ] and [ 8 ] established the local convergence of RTI in shrinking horizons and moving horizons , respectively . When we have different models , [ 3 , 42 , 44 , 35 , 36 ] all showed similar results . A different view of RTI-based MPC is via the parametric optimization , where subproblems are parameterized by a continuous time index and a single Newton step is performed at each time instant . Similar stability results in this setup are shown in [ 11 , 45 , 13 ] . Recently , [ 31 ] studied a lag-L RTI-based MPC ( i.e . one shifts L stages each time ) , and proved that RTI iterates on each stage converge linearly with a rate decaying exponentially in the prediction horizon length . Their analyses rely on the sensitivity analysis of ( 1 ) in [ 30 , 33 ] . An important consequence of convergence of online iteration is the dynamic regret analysis [ 47 , 27 , 7 , 43 ] . For example , the convergence rate in [ 31 ] and the regret order in [ 27 ] are both exponential in prediction horizon length . This is due to the widely-used Lipschitz continuity assumption on the objective . 1The merit function refers to the function that is used in the line search . However , all above theories are local results . The algorithms always take a full ( Newton ) step for each subproblem , without involving any stepsize selection procedure . Thus , the analyses require to have an initial point that is sufficiently close to the local solution , or require that the first Newton step is sufficiently small . Obviously , for Problem ( 1 ) , when ‖x̄0‖ is small enough , the origin is a good initial point and RTI iterates stably track the origin . However , if ‖x̄0‖ is large or we may want to track a nontrivial MPC policy , the origin is not a reasonable choice for the initial point . Based on these considerations , the global convergence of RTI addressed in this paper complements the aforementioned rich local analyses . Again , by global convergence we mean that the KKT residuals of subproblems go to zero as horizon moves forward for any initialization . The known , local stability of tracking error of RTI is not addressed in this paper , but referred to [ 45 , Theorem 4.2 ] . The challenge of our analysis is that : the problem varies with time instants and only a single Newton step is performed at each time instant . Such a setup contrasts to online Newton ( with line search ) analysis , where a single Newton step targeting the same problem is performed . A key , technical step of our analysis is to show that the exact augmented Lagrangian merit function keeps decreasing as horizons shift . To our knowledge , this is the first paper studying the global behavior of online iteration under RTI setting , via a tool of the exact augmented Lagrangian . Another contribution to nonlinear MPC is that we consider time-varying dynamics , while most of the literature considered time-invariant dynamics ( e.g . [ 8 , 21 , 40 ] ) . The subproblems of time-invariant dynamics only differ in the initial state , which results in a much simpler analysis . We notice that [ 31 ] allowed time-varying dynamics , but required to use a larger lag when shifting the horizon . We allow lag one instead . Notations : For two vectors a , b , ( a ; b ) denotes a long column vector that stacks a and b together . For two scalars a , b , a ∨ b = max ( a , b ) . Without specification , ‖ · ‖ denotes the ` 2 norm for vectors and operator norm for matrices . We also reserve the following notations . 0 denotes zero vector or matrix , and I denotes the identity matrix . Their dimensions are clear from the context . M denotes the prediction horizon length , t indexes the sampling time , and zk = ( xk ; uk ) ∈ Rnz with nz = nx+nu denotes the state-control pair at stage k . 2 Preliminaries . We introduce MPC subproblems formulation , the Newton step , and the augmented Lagrangian merit function in this section . At time t , the prediction horizon shifts to [ t , Mt ] with Mt = t+M denoting the last stage , and the corresponding subproblem Pt ( x̄t ) is given by Pt ( x̄t ) : minxt : Mt ut : Mt−1 Mt−1∑ k=t gk ( xk , uk ) + gMt ( xMt ,0 ) + µ 2 ‖xMt‖2 , s.t . xk+1 = fk ( xk , uk ) , k ∈ [ t , Mt − 1 ] , ( 2 ) xt = x̄t . Here , µ is a quadratic penalty parameter independent from t ; x̄t is the current system state , which is given and determined by the control feedback provided to the system before t. The formulation ( 2 ) is used in [ 31 ] for local analysis , while alternative subproblem formulations are available as well [ 8 , 23 ] . Given subproblem ( 2 ) , RTI schemes perform a single Newton step for Pt ( x̄t ) and move to the next time instant . To ease notations , we let x̃t = xt : Mt and ũt = ut : Mt−1 denote the state and control variables of the t-th subproblem , respectively . We also let z̃t = zt : Mt = ( zt ; . . . ; zMt ) denote the whole ordered state-control variables , with slightly abusing the notation zMt to let zMt = xMt . We sometimes also express z̃t = ( x̃t , ũt ) when we specify each component of z̃t . Then , the Lagrangian function of ( 2 ) is Lt ( z̃t , λ̃t ; x̄t ) = Mt−1∑ k=t gk ( xk , uk ) + gMt ( xMt ,0 ) + µ 2 ‖xMt‖2 + Mt−1∑ k=t λTk ( xk+1 − fk ( xk , uk ) ) + λ T t−1 ( xt − x̄t ) , where λ̃t = λt−1 : Mt−1 is the dual vector . Here , λk ∈ Rnx is associated to the k-th constraint of ( 2 ) for k ∈ [ t , Mt − 1 ] , and λt−1 ∈ Rnx is associated to the initial state constraint . Given the input iterate ( z̃0t , λ̃ 0 t ) , the Newton direction ( ∆z̃t , ∆λ̃t ) is solved by the KKT system ( Bt ( Gt ) T Gt 0 ) ( ∆z̃t ∆λ̃t ) = − ( ∇z̃tLt ∇λ̃tL t ) , ( 3 ) where Bt is an approximation of the Hessian Ht = ∇2z̃tL t ; Gt = ∇λ̃tz̃tL t is the Jacobian matrix ; and all matrices in ( 3 ) are evaluated at ( z̃0t , λ̃ 0 t ) . The left hand side matrix is called the KKT matrix . In what follows , Ht , Bt , Gt are always evaluated at the input iterate ( z̃0t , λ̃ 0 t ) if no evaluation point is specified . Taking a closer look at ( 3 ) ( cf . Appendix A ) , we note that the KKT matrix and ∇z̃tLt are independent from the given initial state x̄t . This is a crucial observation . In reality , the realization of the system state may lag behind the sampling time , since the system has to apply the feedback control to evolve . Thus , x̄t may not be known as soon as the input iterate ( z̃0t , λ̃ 0 t ) . However , a promising property of RTI is that most of computations of ( 3 ) can be prepared without knowing x̄t . Given the Newton direction ( ∆z̃t , ∆λ̃t ) from ( 3 ) , we update the iterate by involving a stepsize αt ( z̃1t λ̃ 1 t ) = ( z̃0t λ̃ 0 t ) + αt ( ∆z̃t ∆λ̃t ) . ( 4 ) In our paper , αt is selected by forcing Armijo condition hold on a differentiable exact augmented Lagrangian , defined as Ltη ( z̃t , λ̃t ; x̄t ) =Lt ( z̃t , λ̃t ; x̄t ) + η1 2 ∥∥∥∇λ̃tLt ( z̃t , λ̃t ; x̄t ) ∥∥∥2 + η22 ∥∥∥∇z̃tLt ( z̃t , λ̃t ; x̄t ) ∥∥∥2 . ( 5 ) Here η = ( η1 , η2 ) are parameters of the quadratic penalties . ( 5 ) is called exact augmented Lagrangian since one can show that , for sufficiently large η1 and sufficiently small η2 , a strict local minimizer of Ltη ( · ) is also a strict local minimizer of Problem ( 2 ) and vice versa ( see [ 2 , Proposition 4.5 ] ) . When η2 = 0 , Ltη ( · ) reduces to standard augmented Lagrangian , while it does not enjoy exactness property anymore , and may cause issues when it is used as merit function . For simplicity , we sometimes suppress “ exact ” and call Ltη ( · ) augmented Lagrangian . Based on ( 5 ) , αt is selected by making the following Armijo condition , Ltη ( z̃1t , λ̃ 1 t ; x̄t ) ≤ Ltη ( z̃0t , λ̃ 0 t ; x̄t ) + αtβ ( ∇z̃tLtη ( z̃ 0 t , λ̃ 0 t ; x̄t ) ∇λ̃tL t η ( z̃ 0 t , λ̃ 0 t ; x̄t ) ) T ( ∆z̃t ∆λ̃t ) , ( 6 ) hold for a prespecified parameter β ∈ ( 0 , 1 ) . Note that , by mean value theory , αt can always be found by backtracking line search provided the inner product in ( 6 ) is negative ( cf . Theorem 4.5 ) .

Summary: This paper gives an algorithm for nonlinear MPC based on augmented Lagrangians, and demonstrates that the iterates converge globally to a critical point of the KKT equations. The key insight is a rule for tuning the regularization parameters of the augmented Lagrangian that ensures that each algorithm step reduces the KKT residuals sufficiently to overcome any increase that would arise due to taking a step in the dynamics. As a result, after alternating between algorithm steps and dynamics steps for several iterations, all iterates produced by the algorithm will be near critical points.