Ensemble Kalman Filter (EnKF) for Reinforcement Learning (RL)

1 INTRODUCTION . This paper is concerned with the problem of reinforcement learning ( RL ) in continuous-time and continuous ( Euclidean ) state-space settings . A special case is the linear quadratic Gaussian ( LQG ) problem where the dynamic model is a linear system , the cost terms are quadratic , and the distributions – of the random initial condition and the noise – are Gaussian . The LQG problem has a rich and storied history in modern control theory going back to the very origin of the subject ( Kalman , 1960 ) . To obtain the optimal control , the bottleneck is to solve the Riccati equation – the differential Riccati equation ( DRE ) in finite-horizon settings or the algebraic Riccati equation ( ARE ) in the infinite-horizon settings of the problem . There is a body of literature devoted to the study of these equations ( Bittanti et al. , 2012 ; Lancaster & Rodman , 1995 ) with specialized numerical techniques to compute the solution ( Laub , 1991 ; Benner & Bujanović , 2016 ) . There are two issues which makes the LQG and related problems a topic of recent research interest : ( i ) In high-dimensions , the matrix-valued nature of the DRE or ARE means that any algorithm is O ( d2 ) in the dimension d of the state-space ; and ( ii ) the model parameters may not be explicitly available to write down the DRE ( or the ARE ) let alone solve it . The latter is a concern , e.g. , when the model exists only in the form of a black-box numerical simulator . These two issues have motivated the recent research on the infinite-horizon linear quadratic regulator ( LQR ) problem ( Fazel et al. , 2018 ; Mohammadi et al. , 2021b ; Tu & Recht , 2019 ) . In LQR , the bottleneck is to solve an ARE . The algorithms studied in the recent papers seek to bypass solving an ARE by directly searching over the space of stabilizing gain matrices . Global convergence rate estimates are established for both discrete-time ( Fazel et al. , 2018 ; Dean et al. , 2020 ; Malik et al. , 2020 ; Mohammadi et al. , 2021a ) and continuous-time ( Mohammadi et al. , 2020a ; b ; 2019 ; 2021b ) settings of the LQR problem . Extensions to the H∞ regularized LQR ( Zhang et al. , 2020 ) and Markov jump linear systems ( Jansch-Porto et al. , 2020 ) have also been carried out . The motivation and goals of the present work are related to these recent papers , albeit the proposed solution approach is very different . Our inspiration comes from data assimilation ( Reich & Cotter , 2015 ) , where in practical applications , e.g. , in weather prediction , ( i ) only simulation-based models are available , and ( ii ) these are very high-dimensional . The ensemble Kalman filter ( EnKF ) is an efficient simulation-based algorithm to assimilate sensor data in these applications without the need to explicitly solve a DRE ( Evensen , 1994 ; 2006 ; Houtekamer & Mitchell , 2001 ) . Our contribution : A novel extension of the EnKF algorithm is proposed to learn the optimal control law for the general ( stochastic and partially observed ) setting of the LQG optimal control problem . The proposed algorithm is simulation-based ( using particles ) and , in particular , avoids the need to solve the DRE . Assuming full-state feedback ( e.g , setting of the LQR problem ) , the optimal control law is directly obtained . For the partially observed case , the proposed EnKF is combined with a standard EnKF ( for the estimation problem ) to obtain the optimal control input , based on the use of the separation principle . The algorithm is shown to be exact in its mean-field ( when the number of particles N = ∞ ) limit ( Prop . 2 ) . For the finite-N approximation , an error bound is obtained ( Eq . ( 14 ) ) . An extensive discussion is included to provide an intuitive explanation of the algorithm , situate the algorithm in the RL landscape ( see discussion after Eq . ( 12 ) in Sec . 2.2 ) , and to compare and contrast the algorithm with the state-of-the-art ( Sec . 2.5 ) . For this purpose , numerical comparisons with competing algorithms are also included ( Fig . 2 ( c ) ) . The outline of the remainder of this paper is as follows . The LQG optimal control problem and its simulation-based solution is described in Sec . 2 . The algorithms are illustrated with numerical examples in Sec . 3 where comparisons with the state-of-the-art algorithms are also described . 2 LINEAR QUADRATIC GAUSSIAN ( LQG ) PROBLEM . Problem : The partially observed linear Gaussian model is expressed using the Itô-stochastic differential equations ( SDE ) as dXt = AXt dt+BUt dt+ dξt , X0 ∼ N ( m0 , Σ0 ) ( 1a ) dZt = HXt dt+ dζt , Z0 = 0 ( 1b ) where X = { Xt ∈ Rd : t ≥ 0 } is the hidden state process , U = { Ut ∈ Rm : t ≥ 0 } is the control input , and Z = { Zt ∈ Rp : t ≥ 0 } is the observation process . The model parameters A , B , H are matrices of appropriate dimensions , the process noise ξ = { ξt ∈ Rd : t ≥ 0 } and the observation noise ζ = { ζt ∈ Rp : t ≥ 0 } are Wiener processes ( w.p . ) with covariance Q and R , respectively . It is assumed that X0 , ξ and ζ are mutually independent andR 0 . The LQG optimal control objective is to minimize J ( U ) = E ( ∫ T 0 1 2 |CXt|2 + 12U > t RUt dt+ 12X > T PTXT ) ( 2 ) It is assumed that ( A , B ) is controllable , ( A , C ) and ( A , H ) are observable , and the matrices PT 0 and R 0 . In the fully observed settings , Ut is allowed to be a function of Xt . In the partially observed settings , Ut is a function of the past observations { Zs : 0 ≤ s ≤ t } . For this purpose , it is convenient to denote Zt : = σ ( { Zs : 0 ≤ s ≤ t } ) as the sigma-field generated by the observations , and consider control inputs U which are adapted to the filtration Z : = { Zt : t ≥ 0 } . Classical solution : Using the seperation principle1 the LQG controller is obtained in three steps : Step 1 . Filter design : The objective of the filter design step is to compute the causal estimate X̂t : = E ( Xt|Zt ) for t ≥ 0 . The evolution equation for { X̂t : t ≥ 0 } is the Kalman-Bucy filter . Notably , the optimal gain matrix for the filter is obtained by solving a forward ( in time ) DRE . Step 2 . Control design : The objective of the control design step is to compute the optimal feedback control law { ut ( x ) : 0 ≤ t ≤ T , x ∈ Rd } for the fully observed LQG problem . It is well known to be of the linear form ut ( x ) = Ktx , 0 ≤ t ≤ T where the optimal gain matrix Kt is obtained by solving a backward ( in time ) DRE . 1The separation principle hinges on the assumption that the control input Ut does not change the observation sigma-field Zt . This is valid under very mild assumptions on the control policy , e.g. , Lipschitz with respect to estimate X̂t ( Van Handel , 2007 , Sec . 7.3 ) , ( Georgiou & Lindquist , 2013 ) . Step 3 . Certainty equivalence : The optimal control input is obtained by combining the results from steps 1 and 2 : Ut = ut ( X̂t ) = KtX̂t , 0 ≤ t ≤ T The bottleneck is to solve the DREs – the forward DRE for the optimal filter gain and the backward DRE for the optimal control gain2 . In the following three sections , we describe a simulation-based algorithm for these three steps which avoids the need to explicitly solve the DREs . 2.1 STEP 1 . FILTER DESIGN USING ENKF . The filter design objective is to compute the causal estimate X̂t = E ( Xt|Zt ) . In the linear Gaussian settings , the conditional distribution of the Xt is Gaussian whose mean and variance are denoted as mt and Σt , respectively . These evolve according to the Kalman-Bucy filter : dmt = Amt dt+BUt dt+ Lt ( dZt −Hmt dt ) , m0 = E ( X0 ) ( 3a ) d dt Σt = AΣt + ΣtA > +Q− ΣtH > R−1HΣt , Σ0 = var ( X0 ) ( 3b ) where Lt = ΣtH > R−1 is the Kalman gain . Note that ( 3b ) is a forward ( in time ) DRE and its solution Σt is used to compute the optimal Kalman gain Lt . The EnKF is a simulation-based algorithm to approximate the Kalman filter , that does not require an explicit solution of the DRE ( 3b ) . The design of an EnKF proceeds in two steps : 1 . Construct a stochastic process , denoted by X̄ : = { X̄t ∈ Rd : t ≥ 0 } , such that the conditional distribution ( given Zt ) of X̄t is equal to the conditional distribution of Xt ; 2 . Simulate N stochastic processes , denoted by { Xit : t ≥ 0 , 1 ≤ i ≤ N } , to empirically approximate the distribution of X̄t . The process X̄ is referred to as the mean-field process . The N processes in the step 2 are referred to as particles . The construction ensures that the EnKF is exact in the mean-field ( N =∞ ) limit . That is , for any bounded and continuous function f , E [ f ( Xt ) |Zt ] Step 1 = E [ f ( X̄t ) |Zt ] ︸ ︷︷ ︸ exactness condition Step 2≈ 1 N N∑ i=1 f ( Xit ) The details of the two steps are as follows : Mean-field process : The mean-field process is constructed as dX̄t = AX̄t dt+BUt dt+ dξ̄t + L̄t ( dZt − HX̄t +Hm̄t 2 dt ) , X̄0 ∼ N ( m0 , Σ0 ) ( 4 ) where ξ̄ : = { ξ̄t : t ≥ 0 } is an independent copy of the process noise ξ , L̄t : = Σ̄tH > R−1 is the Kalman gain and m̄t : = E [ X̄t|Zt ] , Σ̄t : = E [ ( X̄t − m̄t ) ( X̄t − m̄t ) > |Zt ] are the conditional mean and the conditional covariance , respectively , of X̄t . The right-hand side of ( 4 ) depends upon both the process ( X̄t ) as well as the statistics of the process ( m̄t , Σ̄t ) . Such an SDE is an example of a McKean-Vlasov SDE . The proof of the following proposition is included in the Appendix A ( see also Taghvaei & Mehta ( 2020 , Theorem 1 ) ) . Proposition 1 ( Exactness of EnKF ) . Consider the mean-field EnKF ( 4 ) initialized with a Gaussian initial condition X̄0 ∼ N ( m0 , Σ0 ) . Suppose the control input U is a Z-adapted stochastic process . Then its solution X̄t is a Gaussian random variable whose conditional mean and variance m̄t = mt , Σ̄t = Σt , a.s. , t > 0 evolve the same as the Kalman filter ( 3 ) . 2In the steady-state or infinite horizon settings ( as T → ∞ ) , one may replace the optimal filter gain and the optimal control gain by their steady-state values . These are directly obtained by solving the respective AREs . Finite-N approximation : The mean-field process is simulated as an interacting particle system : dXit = AX i t dt+BUt dt+ dξ i t︸ ︷︷ ︸ i-th copy of model ( 1a ) +L ( N ) t ( dZt − HXit +HX̂ ( N ) t 2 dt ) ︸ ︷︷ ︸ data assimilation step , Xi0 i.i.d∼ N ( m0 , Σ0 ) ( 5 ) where X̂ ( N ) t : = 1 N ∑N i=1X i t is the empirical mean and L ( N ) t = 1 N − 1 N∑ i=1 ( Xit ) ( HX i t −HX̂ ( N ) t ) > R−1 ( 6 ) is the empirical approximation of the optimal Kalman gain matrix . The system ( 5 ) is referred to as the square root form of the EnKF ( Bergemann & Reich , 2012 , Eq ( 3.3 ) ) . Note that the gain is approximated entirely in terms of particles without solving the DRE ( 3b ) . The EnKF ( 5 ) is an example of a simulation-based algorithm in the sense that N copies of the model ( 1a ) are simulated in parallel . The simulations are coupled through a term which is referred to as the data assimilation step . This term has a gain times error feedback control structure .

Summary: The paper is about designing a simulation-based ensemble Kalman filter algorithm for learning the optimal control policy for the Linear Quadratic Gaussian (LQG) control problem. The main contribution is that the paper extends the existing ensemble Kalman filter algorithm for filter (state estimation) design to the optimal control policy learning for the LQG control setting. And it is shown to be exact in its mean-field limit (N=infinity).