Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexity

1 Introduction . Recent years have witnessed the rapid development of reinforcement learning ( RL ) methods in handling continuous control tasks [ 1 , 2 , 3 ] . Central to the success of RL are policy optimization ( PO ) methods , including policy gradient ( PG ) [ 4 , 5 , 6 ] , actor-critic [ 7 , 8 ] , and other variants [ 9 , 10 ] . Progress reported in the literature has clearly shown an increasing interest in understanding theoretical properties of PO methods for relatively simple baselines such as linear control problems [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] . However , the theory of model-free PO methods on risk-sensitive/robust control remains underdeveloped in the literature . Since risk-sensitivity and robustness are important issues for designing safety-critical systems , it is natural to bring up the questions of whether and how model-free PO methods would converge for these continuous control tasks . Our work in this paper is motivated by the above concern , and studies the sample complexity of model-free PG methods on two important baseline problems in risk-sensitive/robust control , namely the linear exponential quadratic Gaussian ( LEQG ) , and the linear quadratic ( LQ ) disturbance attenuation problems . The former covers a fundamental setting in risk-sensitive control , and the latter is an important baseline for robust control . Based on the well-known equivalence between these problems and LQ dynamic games [ 22 , 23 , 24 , 25 , 26 ] , we develop a unified PO perspective \Equal contribution . K. Zhang is with the LIDS and the CSAIL at the Massachusetts Institute of Technology . Email : kaiqing @ mit.edu . X. Zhang , B. Hu , and T. Başar are with the Department of ECE and the CSL at the University of Illinois at Urbana-Champaign . Emails : { xz7 , binhu7 , basar1 } @ illinois.edu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for both . A common feature for the above two problems is that their optimization landscapes are by nature more challenging than that of the linear quadratic regulator ( LQR ) problem , and existing proof techniques for model-free PG methods [ 11 , 17 , 18 , 19 ] are no longer effective due to lack of coercivity of the objective functions . Specifically , when applying PG methods to LQR , the feasible set for the resultant constrained optimization problem is the set of all linear state-feedback controllers that stabilize the closed-loop dynamics . The objective function of LQR is coercive on this feasible set and serves as a barrier function itself [ 14 ] , guaranteeing for the PG iterates to stay in the feasible set and converge to the globally optimal controller . For the LEQG and LQ disturbance attenuation problems , the risk-sensitivity/robustness conditions have reshaped the feasible set in a way that the objective function becomes non-coercive , i.e . the objective value can remain finite while approaching the boundary . The objective is no longer a barrier function , and new proof techniques are needed to show that model-free PG iterates will stay in the feasible set . This is significant for safety-critical control systems with model uncertainty , as the iterates ’ feasibility here is equivalent to risk-sensitivity/robustness of the controller ( cf . Remark A.8 ) , and the failure to preserve robustness during learning can cause catastrophic effects , e.g. , destabilizing the system in face of disturbances . The most relevant result was developed in [ 16 ] , which proposes implicit regularization ( IR ) arguments to show the convergence of several PG methods on the mixedH2/H1 control design problem ( which can be viewed as the infinite-horizon variant of the LQ disturbance attenuation problem studied in this paper ) . The main finding there is that two specific PG search directions ( with perfect model information ) are automatically biased towards the interior of the robustness-related feasible set . In [ 16 ] , it is emphasized that IR is a feature of both the problem and the algorithm , contrasting to that the stability-preserving nature of PG methods for LQR problems is based on the barrier function property of the objective and hence is algorithm-agnostic . Although the idea of IR is relevant to PO problems with non-coercive objective functions , the arguments in [ 16 ] only apply to the setting with a known model , since they rely on a specialized perturbation technique which may potentially generate arbitrarily small “ margins '' . However , in the model-free setting , a uniform margin is required for provable tolerance of statistical errors . See a detailed comparison of the literature in §A.1 . In this paper , for the LEQG and LQ disturbance attenuation problems , we overcome the above margin issue and obtain the first IR result in the model-free setting . This enables the first model-free PG method that provably solves these control problems with a finite number of samples . We highlight our contributions as follows . Contributions . We provide the first sample complexity results for model-free PG methods for solving linear control problems with risk-sensitivity/robustness concerns ( the LEQG and LQ disturbance attenuation problems ) , which were viewed as important open problems in the seminal work [ 11 ] . From the robust control perspective , one feature of our algorithms is that , during the learning process , a certain level of robustness/risk-sensitivity of the controller is proved to be preserved . This has generalized the results in [ 16 ] with a known model , and has thus enabled the finite-sample convergence guarantees of PG methods for risk-sensitive/robust control design . Our algorithms and sample complexity results also address two-player zero-sum LQ dynamic games in the finite-horizon time-varying setting , which are among the first sample complexity results for the global convergence of policy-based methods for competitive multi-agent RL . Second , in the context of minimax optimization , our results address a class of nonconvex-nonconcave minimax constrained optimization problems , using zeroth-order multi-step gradient descent-ascent methods . Finally , part of our results provide the sample complexity analysis for PG methods that solve the finite-horizon time-varying LQR problem with system noises and a possibly indefinite state-weighting matrix . 2 Background . We first introduce two classic settings in risk-sensitive and robust control , namely LEQG , and LQ disturbance attenuation , and their equivalence to zero-sum LQ dynamic games . 2.1 Linear Exponential Quadratic Gaussian . Consider the finite-horizon LEQG problem [ 22 , 27 , 28 ] , with time-varying systems dynamics described by xt+1 = Atxt +Btut +wt , t 2 { 0 , · · · , N 1 } , where xt 2 Rm represents the system state ; ut 2 Rd is the control input ; wt 2 Rm is an independent ( across time ) Gaussian random noise with wt ⇠ N ( 0 , W ) for some W > 0 ; the initial state x0 ⇠ N ( 0 , X0 ) for some X0 > 0 , which is independent of { wt } ; and At , Bt are time-varying system matrices . The objective function is given by J ( { ut } ) : = 2 logEexp h 2 ⇣PN 1 t=0 ( x > t Qtxt + u > t Rtut ) + x > NQNxN ⌘i , where Qt , QN 0 and Rt > 0 are symmetric weighting matrices ; and > 0 is a parameter capturing the degree of risk-sensitivity , which is upper-bounded by some ⇤ > 0 [ 23 , 28 , 29 ] . The goal in the LEQG problem is to find the J -minimizing optimal control policy µ⇤t : ( Rm ⇥Rd ) t ⇥ Rm ! Rd that maps , at time t , the history of state-control pairs up to time t and the current state xt to the control ut . It has been shown in [ 22 ] that µ⇤t has a linear state-feedback form µ ⇤ t ( xt ) = K ⇤t xt , where K ⇤t 2 Rd⇥m , for all t. Therefore , it suffices to search K ⇤t in the matrix space Rd⇥m for all t 2 { 0 , · · · , N 1 } , without losing any optimality . The resulting PO problem as well as its closed-form objective function and PG are provided in §A.3.1 . 2.2 LQ Disturbance Attenuation . Second , consider the LQ disturbance attenuation problem , with time-varying dynamical systems described by xt+1 = Atxt + Btut +Dtwt and zt = Ctxt + Etut , where xt 2 Rm is the system state ; ut 2 Rd is the control input ; wt 2 Rn is the ( unknown ) disturbance input ; zt 2 Rl is the controlled output ; At , Bt , Ct , Dt , Et are system matrices with appropriate dimensions ; and x0 2 Rm is unknown . In addition , we assume that E > t [ Ct Et ] = [ 0 Rt ] for some Rt > 0 for normalization , with no loss of generality ( see §3.5.1 of [ 25 ] for a simple procedure that can transform the general problem into a form that satisfies this “ normalization ” assumption ) . Subsequently , we denote the ` 2-norms of the vectors ! : = h x > 0 C > 0 , w > 0 , · · · , w > N 1 i > and z : = h z > 0 , · · · , z > N 1 , x > NQ 1/2 N i > as k ! k and kzk = n C ⇣ { ut } , { wt } ⌘o1/2 , where QN 0 . Then , the robustness of a designed controller can be guaranteed by a constraint on the ratio between kzk and k ! k . Specifically , the goal of disturbance attenuation is the following : Given a > ⇤ , where ⇤ = p ( ⇤ ) 1 > 0 is the optimal ( minimax ) level of disturbance attenuation at the output and recall that ⇤ is the upper bound for the risk-sensitivity parameter in LEQG [ 23 , 25 ] , find a control policy µt = Ktxt , t 2 { 0 , · · · , N 1 } , that solves min { µt } J ( { µt ( xt ) } ) subject to supx0 , { wt } n C ( { µt ( xt ) } , { wt } ) o1/2 /k ! k < , where J ( { µt ( xt ) } ) is an upper bound of the LQG cost [ 30 ] . We defer the precise formulations of the LQ disturbance attenuation problem to §A.3.2 . Also , see Remark A.3 for the challenges when addressing LEQG and LQ disturbance attenuation directly using derivative-free PG methods . 2.3 An Equivalent Dynamic Game Formulation . To overcome the challenges spelled out in Remark A.3 , we now introduce an equivalent dynamic game formulation to the LEQG and the LQ disturbance attenuation problems due to that the natural PGs of the objective function of the game can be directly sampled ( as we will show in §4 ) . Under certain conditions to be introduced shortly , the saddle-point gain matrix of the minimizing player in the dynamic game ( if exists ) also addresses these two robust/risk-sensitive control problems . We aim to address these three classes of problems in a unified fashion . Now , consider a two-player zero-sum LQ stochastic dynamic game ( henceforth , game ) with closedloop perfect-state information pattern , characterized by xt+1 = Atxt + Btut + Dtwt + ⇠t , where xt 2 Rm is the system state , and ut : = Ktxt 2 Rd ( resp. , wt : = Ltxt 2 Rn ) is the linear statefeedback controller of the minimizing ( resp. , maximizing ) player1 , which suffices to attain the saddle point whenever exists [ 26 ] . The initial state x0 and the additive process noises ⇠t are drawn independently from a distribution D and At , Bt , Dt are system matrices . Further , we assume that D is zero-mean , has a positive-definite covariance , and satisfies almost surely that kx0k , k⇠tk # , for all t 2 { 0 , · · · , N 1 } and some constant # . The goal of the minimizing ( resp . maximizing ) player is to minimize ( resp . maximize ) a quadratic objective function , namely to solve the game inf { Kt } sup { Lt } Ex0 , ⇠0 , ··· , ⇠N 1 '' N 1X t=0 x > t ( Qt +K > t R u t Kt L > t R w t Lt ) xt + x > NQNxN # , ( 2.1 ) 1Hereafter we will use player and agent interchangeably . Also , restrictions to instantaneous linear statefeedback policies do not lead any loss of generality , as the results of §2.3 hold for general square-integrable policies , allowed to use the entire state history . where Qt , QN 0 , Rut , Rwt > 0 are symmetric weighting matrices . Whenever the solution to ( 2.1 ) exists such that the inf and sup can be interchanged , the value ( 2.1 ) is the value of the game , and the corresponding policies are the saddle-point/Nash equilibrium policies . To characterize the solution to ( 2.1 ) , we first introduce the following time-varying Riccati difference equation ( RDE ) : P⇤t =Qt +A > t P ⇤ t+1⇤ 1 t At , t 2 { 0 , · · · , N 1 } , ( 2.2 ) where ⇤t : = I + ⇣ Bt ( Rut ) 1B > t Dt ( Rwt ) 1D > t ⌘ P⇤t+1 and P ⇤ N = QN . From [ 26 ] , whenever a saddle point exists , the saddle-point control policies are linear state-feedback ( i.e. , µ⇤u , t ( hu , t ) = K ⇤t xt and µ⇤w , t ( hw , t ) = L⇤txt ) , and the gain matrices K ⇤t = ( Rut ) 1B > t P⇤t+1⇤ 1 t At and L ⇤ t = ( Rwt ) 1D > t P⇤t+1⇤ 1 t At are unique , where P ⇤ t+1 0 is generated by ( 2.2 ) . Then , we introduce a standard assumption that ensures the existence of the value of the game [ 25 ] . Assumption 2.1 Rwt D > t P⇤t+1Dt > 0 , for all t 2 { 0 , · · · , N 1 } , where P ⇤ t+1 0 is generated by ( 2.2 ) . Under Assumption 2.1 , the value in ( 2.1 ) is attained by the sequence ⇣ { K ⇤t } , { L⇤t } ⌘ , where K ⇤t 2 Rd⇥m and L⇤t 2 Rn⇥m are gain matrices at the saddle-point/Nash equilibrium . Thus , we can replace the inf and sup in ( 2.1 ) with min and max , respectively . Some further notes on Assumption 2.1 are provided in Remark A.5 , showing that it is not restrictive and is in fact “ quite tight ” . Lastly , we state the equivalences between three classes of problems , with its proof provided in §B.3 . Lemma 2.2 ( Connections ) For any fixed > ⇤ in the LQ disturbance attenuation problem , we can introduce an equivalent LEQG problem and an equivalent zero-sum LQ dynamic game . Specifically , if we set 1I , Rt , C > t Ct , W in LEQG , 2I , Rt , C > t Ct , DtD > t in the LQ disturbance attenuation problem , and Rwt , R u t , Qt , DtD > t in the game to be the same , for all t 2 { 0 , · · · , N 1 } , then the optimal gain matrices in LEQG , the gain matrix in the LQ disturbance attenuation problem , and the Nash equilibrium gain matrix for the minimizing player in the game are the same . By Lemma 2.2 and Remark A.3 , we will hereafter focus on solving the game formulation in §2.3 using PG methods . As a result , the minimizing controller we obtain solves three classes of problems introduced above altogether .

Summary: This article studies the convergence of double-loop policy gradient methods for finite-horizon two-layer zero-sum LQ game arising from the risk-sensitive control problems. It shows that, for suitable choices of the initial guess of the outer iteration, both the natural gradient method and the Gauss-Newton method converge to the unique Nash equilibrium. The authors then extend the algorithm to derivative-free methods based on observed state trajectories. To the best of my knowledge, this is the first theoretical result on convergence of gradient-based methods for competitive multi-agent RL problems.