Ensemble Kalman Filter (EnKF) for Reinforcement Learning (RL)
Authors: Anant A Joshi
Amirhossein Taghvaei
Prashant G Mehta
Venue: ICLR 2022 Submitted (2022)
Keywords: decision and control
control theory
reinforcement learning

Abstract: This paper is concerned with representing and learning the optimal control law for the linear quadratic Gaussian (LQG) optimal control problem.  In recent years, there is a growing interest in re-visiting this classical problem, in part due to the successes of reinforcement learning (RL).  The main question of this body of research (and also of our paper) is to approximate the optimal control law without explicitly solving the Riccati equation.  For this purpose, a novel simulation-based algorithm, namely an ensemble Kalman filter (EnKF), is introduced in this paper.  The algorithm is used to obtain formulae for optimal control, expressed entirely in terms of the EnKF particles.  For the general partially observed LQG problem, the proposed EnKF is combined with a standard EnKF (for the estimation problem) to obtain the optimal control input based on the use of the separation principle.  The theoretical results and algorithms are illustrated with numerical experiments.    

Under review as a conference paper at ICLR 2022
ENSEMBLE KALMAN FILTER (ENKF) FOR REINFORCEMENT LEARNING (RL)
Anonymous authors Paper under double-blind review
ABSTRACT
This paper is concerned with representing and learning the optimal control law for the linear quadratic Gaussian (LQQ)) optimal control problem. In recent years, there a growing interest in re-visiting this classical problem, in part due to the successes of reinforcement learning (RL). The main question of this body of re- search (and also our paper) is to approximate the optimal control law without explicitly solving the Riccati equation. For this purpose, novel conulation-based algorithm, namely an ensemble Kalman filter (EnKF), is introduced in this paper. The algorithm is used to obtain formulae for optimal control, expressed entirely in terms of the EnKF particles. For the general partially observed I.QG problem, the proposed EnKF is combined with standard EnKF (for the estimation problem) to obtain the optimal control input based on the use of the separation principle. The theoretical results and algorithms illustrated with numerical experiments.
INTRODUCTION
This paper concerned with the problem of reinforcement learning (RL) in continuous-time and continuous (Euclidean) state-space settings. A special case is the linear quadratic Gaussian (LQG) problem where the dynamic model is a linear system, the cost terms are quadratic, and the distribu- tions - of the random initial condition and the noise are Gaussian.
The LQG problem has a rich and storied history in modern control theory going back the very origin of the subject (Kalman 1960). To obtain the optimal control, the bottleneck is to solve the Riccati equation the differential Riccati equation (DRE) in finite-horizon settings or the algebraic Riccati equation (ARE) in the infinite-horizon settings of the problem. There is body of literature devoted to the study of these equations (Bittanti et al.] 2012) Lancaster Rodman 1995) with
specialized numerical techniques to compute the solution (Laub) 1991 Benner & Bujanovie 2016)-
There are two issues which makes the LQG and related problems a topic of recent research interest: (i) In high-dimensions, the matrix matrix-valued nature of the DRE or ARE means that any algorithm is ((2) in the dimension d of the state-space; and (ii) the model parameters may not be explicitly available to write down the DRE (or the ARE) let alone solve it. The latter is concern, e.g., when the model exists only in the form of a black box numerical simulator.
These two issues have motivated the recent research on the infinite-horizon linear quadratic regulator (LQR) problem (Fazel et al. 2018 Mohammadi et al.) 202 Tu & Recht 2019). In LQR, the bottleneck is to solve ARE. The algorithms studied the recent papers seek to bypass solving an ARE by directly searching over the space of stabilizing gain matrices. Global convergence rate estimates are established for both discrete-time (Fazel et al. 2018 Dean et al. 2020 Malik al.
2020 Mohammadi et al. 2021a) and continuous-time (Mohammadi et al. 2020a by 2019 202 lb) settings of the IQQR problem. Extensions to the Hoc regularized LQR (Zhang al. 2020) and
Markov jump linear systems (Jansch-Porto al. 2020) have also been carried out.
The motivation and goals the present work are related to these recent papers, albeit the proposed solution approach is very different. Our inspiration comes from data assimilation (Reich & Cotter 2013), where in practical applications, e.g., in weather prediction, (i) only simulation based models available, and (ii) these are very high-ding nigh-dimensional. The ensemble Kalman filter (EnKF) is an efficient simulation-bast algorithm to assimilate sensor data in these applications without the need to explicitly solve DRE (Evensen 1994 2006 Houtekamer Mitchell 2001).

---

Under review as a conference paper at ICLR 2022
Our contribution: A novel extension of the EnKF algorithm is proposed learn the optimal control law for the general (stochastic and partially observed) setting the LQG optimal control problem. The proposed algorithm is simulation-base (using particles) and, in particular, avoids the need to solve the DRE. Assuming full-state feedhack (e.g, setting of the LQR problem), the optimal control law is directly obtained. For the partially observed case, the proposed EnKF is combined with a standard EnKF (for the estimation problem) to obtain the optimal control input, based on the use of the separation principle.
The algorithm is shown to be exact in its mean-field (when the number of particles N - oo) limit (Prop. 2. For the finite-N approximation, an error bound is obtained (Eq. (14)). An extensive discussion is included to provide an intuitive explanation of the algorithm, situate the algorithm in the RI. landscape (see discussion after Eq. (12) in Sec. 2.2). and to compare and contrast the algo- rithm with the state-of-the-an art (Sec. 2.5 For this purpose, numerical comparisons with competing algorithms are also included (Fig. (c).
The outline of the remainder of this paper is as follows. The LQG optimal control problem and its simulation-based solution is described Sec. 2 The algorithms are illustrated with numerical examples in Sec. where comparisons with the state-of-the-ar algorithms are also described.
2 LINEAR QUADRATIC GAUSSIAN (LQG) PROBLEM
Problem:
The partially observed linear Gaussian model is expressed using the Itô-stochastic dif- ferential equations (SDE) as
dX, = AX, dt + BU, dt + deg, dz. H HX, dt + dhe, Zo 0
Xo ~ N(mo, Eo)
(la)
(Ib)
where X = [X2 E Rdd t 2 0} is the hidden state process, U = {U, E I"' :t 0} the control 0} is the observation process. The model parameters A, H are
input, and Z = (Z, Rº matrices of appropriate dimensions, the process noise { {&. E Rd 2 0} and the observation noise Ç= {5e R 2 0} are Wiener processes (w.p.) with covariance and R, respectively. It is assumed that Xo. & and ( are mutually independent and R > 0.
The LQG optimal control objective is to minimize
J(U)= EE (L, CX, + JU, RU dt 1 PrXx
It is assumed that (A, B) is controllable, (A, C) and (A, H) are observable, and the matrices Pr and R > 0. In the fully observed settings, U, is allowed to be function of X,, In the partially observed settings, U, a function of the past observations {Z. : S S t}. For this purpose, is convenient to denote 2, == a a(Z,. 0 S S f)) as the sigma-field generated by the observations, and consider control inputs which are adapted to the filtration Z == (2. t 23.
Classical solution: Using the seperation principle the LQG controller is obtained in three steps:
The objective of the filter design step is to compute the causal estimate X, == E(X,) Z.) for 2 O. The evolution equation for (&, 2 0} is the Kalman-Bucy filter. Notably, the optimal gain matrix for the filter is obtained by solving a forward (in time) DRE.
Step 1. Filter design:
Step 2. Control design: The objective of the control design step is to compute the optimal feed- back control law {u)()) 0 < t T. E Rd for the fully observed LQG problem. It is well known to be of the linear form
u4 (z) = K,z, DstST
where the optimal gain matrix K, is obtained by solving backward (in time) DRE.
1 'The separation principle hinges on the assumption that the control input U, does change the observation sigma-ficld 2,. This is valid under very mild assumptions on the control policy, e.g., Lipschitz with respect to estimate X, Van Handel 2007 Sec. 7.3). (Georgiou & Lindquist 2013).
2

---

Under review as a conference paper at ICLR 2022
Step 3. Certainty equivalence: The optimal control input is obtained by combining the results from steps and 2:
U, = (x,, = K,X,, 0 S
The bottleneck is to solve the DREs the forward DRE for the optimal filter gain and the backward DRE for the optimal control gain In the following three sections, we describe a simulation-bai algorithm for these three steps which avoids the need to explicitly solve the DREs.
2.1 STEP 1. FILTER DESIGN USING ENKF
The filter design objective is to compute the causal estimate X, = Z,), In the linear Gaussian settings, the conditional distribution of the X, is Gaussian whose and variance denoted as my and y, respectively. These evolve according to the Kalman-Bug, filter:
dm;- = Am, dt + BU, dt + Li( dZz Hm, df),
mo = E(Xo)
(3a)
dt ; = AΣ, + A + Q E,H R HH No = var(
(3b)
where L, 2,11 R-1 1 is the Kalman gain. Note that (3b) is forward (in time) DRE and its solution Σ, is used to compute the optimal Kalman gain L,
The EnKF is a simulation- based algorithm approximate the Kalman filter, that does not require an explicit solution of the DRE (3b). The design of an EnKF proceeds in two steps:
Construct a stochastic process, denoted by X == {X, R. t 0}, such that the conditional distribution (given 2,) of X, is equal to the conditional distribution of X;;
Simulate N stochastic processes, denoted by {Xi 2 0, S i N), to empirically approxi- mate the distribution of X,-
The process X is referred as the mean-fiele process. The N processes in the step are referred to as particles. The construction ensures that the EnKF is exact in the mean-field (N oc) limit. That is, for any bounded and continuous function f.
N
E[)(X,) Z.] Step E(J(X.)ZZ
Step 2
~
1 N
Σ
f(xi)
condition
The details of the two steps follows:
Mcan-field process: The mean-field process is constructed as
dX; = AX, dt + BU, dt + dı, + Le(dZ, HX, + Hm, dt), ~o~N(mo. 2o) (4)
2
where & : L3 t 2 0} is an independent copy of the process noise E, Le == D.Hl T R-1 is the
Kalman gain and
me = E[X,
E( m. m.) [2:
are the conditional mean and the conditional covariance, respectively, of X, The right-hand side of (4) depends upon both the process (X,) as well as the statistics of the process (m,, E,). Such an SDE is an example of McKean- BcKeen-Vlasov Vlasov SDE, The proof of the following proposition is included in the Appendix A see also Taghvaei Mehta (2020) Theorem I)).
Proposition 1 (Exactness of EnKF), Consider the mean-field EnKF initialized with a Gaussian initial condition X, M(mo, Eo). Suppose the control inpul U is 2 Z-udapted stochastic process. Then its solution X, is a Gaussian random variable whose conditional mean and variance
m = mm
Σg = Exg, a.S., 1>
evolve the same as the Kalman,fitle filter
3
In the steady-state infinite horizon settings (as T - no), one may replace the optimal filter gain and the optimal control gain by their steady-state values. These directly obtained by solving the respective AREs.
3

---

Under review as a conference paper at ICLR 2022
Finite-N approximation:
The mean-field process is simulated an interacting particle system:
(2) (dz+ HX + H&(N)
dx; = AX; dt + BU, di dg
d(),
Lid X ~
(mmo Ea) (5)
i-th copy model 1u
assimilation step
where X, X(N)
EE
Xi is the empirical mean and
1 N N
- RR-
(6)
the empirical approximation of the optimal Kalman gain matrix. The system ( is referred to the square root form of the EnKF (Bergemann & Reich 2012] Eg (3.3)). Note that the gain is approximated entirely in terms of particles without solving the DRE (3b).
The EnKF (5) example of a simulation-based algorithm in the sense that N copies of the model (aa are simulated in parallel. The simulations are coupled through a term which is referred as the data assimilation step. This term has gain times error feedback control structure.
2.2 STEP 2. CONTROL DESIGN USING ENKF
This section contains the main contribution of this paper. Assuming full-state feedback, the optimal control law for the LQG problem (Ia) is
Ut (x) K,
where
K, -
the optimal gain matrix and P. is solution of the backward (in time) DRE
P. - ATP, + - di
Pr (given)
known that P. > 0 for < t s T whenever Pr > (Brockett 2013 Sec. 24). Therefore, S, P. is well-defined. is straightforward to verify that S₁ also solves backward DRE
S AS, + S.A BR--BT + SSC di
ST
The bottleneck is to solve the DRE 77. In the following, an EnKF is proposed to obtain simulation- based approximation of the optimal control law ut- As before, the construction proceeds in two steps: (i) definition of an exact mean-field process and (ii) its finite-N approximation.
Mcan-field process: The objective is to construct a stochastic process, denoted Y, E Rd at time whose variance equals Pt the solution of the DRE (7). This is done by constructing y (D)
gd 0 T) as solution of the following backward (in time) McKean-Vlasov SDE:
d}² = ADi dt + Bdd, + 5 S.CT (CJ; Ci,) ~ ST)
and defining
()) m
where 7 {ee E R.
2 0} w.p. with covariance matrix R-1, dij in (9) denotes the backward Itô Itô-integral (Nualart & Pardoux 1988 Sec. 4.2), and
E().). 8, = E n.) me)T
(10)
The proof of the following proposition is included in Appendix B
Proposition 2. Consider the mean-field EnKF D initialized with Gaussian initial condition Pr ~ AO Sr). Then its solution Dr a Gaussian random variable whose mean and variance
n4 0, S, St. 0 T

---

Under review as a conference paper at ICLR 2022
Consequently, Y, is also Gaussian random variable with mean and variance
E(Y,) = O, E(X,Y,)) Pt. 0 OstsT
and therefore (i) the matrix B is explicitly known then the optimal gain matrix
= - B
or else (when is not) then (ii) define the Hamiltoniar (or the O-function) (Mehta Meyn 2009)
(z, a, ||xx|*++ 3/a" Ra + x: E(Y.Y, ]Ax + Ba) from which the optimal control law obtained as ux(x)) arg min (z a,t).
The mean-field process is empirically approximated by simulating system interacting particles (D% E IR t T, i,.,,,
Finite-N approximation:
1,...,NN according to
((N CT cyt + Cr+ (N) S
dy = A) dt B d
dt, 2 (7),
(1I)
i-th copy of model Ta
step
r =
(
where nf' is a copy of 17. and
n (N) = N-1 Σ V
S(N)
N Σor mi (1)) mn )T
(12)
Relationship to RL: The following intuitive explanation is included to situate the proposed dimulation-based algorithm in the RL landscape:
Representation: In designing any RL algorithm, the first issue is representation - how does one rep- resent the unknown value function (P, in the linear case)? Our novel idea the first key innovation of this paper- is to represent P. is in terms of statistics (variance) of the particles. Such a represen- tation is fundamentally distinct from representing the value function, or its proxies, such as the Q function, in terms of set of basis functions (Bradtke et al., 1994 Devraj al., 2020 Maci et al, 2010 Fujimoto et al. 2018 Melo et al. 2008).
The algorithm is entirely simulation based: N copies of the model la are simulated in parallel where the terms on the righthand-side (TT) have the following intuitive explanation:
Dynamics: The first term " "AyY dt" on the right-hand side of [1] simply copy of uncontrolled dynamics in the model laa.
The second term is the control "BU, dt" implemented as "B dry"". That is, the control input U for the i-th particle a white noise process with covariance R-1, One may interpret this as an exploration step whereby the cheaper control directions explored more.
Control:
In summary, for the i-th particle, the dynamics and control are the same as any RL algorithm (white noise is used for exploration). The difference arises due to the third term (which is the second key innovation of this paper).
RL step: The third term indicated as the RL step engenders particle flow that effectively imple- ments the value iteration step of the RL. There are several points to be made:
1. The RL step is function of the state cost term in (2). This is most easily seen by writing the equation for the empirical mean
dni (N) = (A + S.00 C)( di + Bdi
(NN)
(N) 0
nhy
where mi -N Σ. 1/f- Noting that the state cost term is x T T Cr, the RL step imple-
ments gradient with S( ( as the gain matrix.
The Hamiltonian H(,, a) is in the form of an oracle because (Ax + Ba) is the right-hand side of the simulation model Ta
5

---

Under review as a conference paper at ICLR 2022
2. The RL step has linear feedback control structure and serves to couple the particles. Without the RL step, the particles are independent of each other.
3. This form of the RL step possible only if one has access to add additional terms outside the control channel (same the data assimilation step in estimation). For example, this is not possible when the system exists only as an experiment.
simulator and an ability
Arrow of simulation time:
The particles simulated backward from terminal time t T to initial time / O. This is consistent with the dynamic programming (DP) equation which also proceeds backward in time. However, because the focus of the RI. is the infinite-horizon problem, this important property of DP is not considered in algorithm design. It is noted that the infinite-horizon problem is easily handled in our formulation by taking I as suitably large (which how the infinite- horizon limit is defined in DP).
remains use the finite-N system to approximate the optimal control law.
Optimal control:
There are two cases to consider: (i) If the matrix B is explicitly known ther
f
K! (
1 N - 1 (B
(13)
im
or else (when is not) then (ii) approximate the Hamiltonian as
H (x, = |/xx/² jaa" Ra + N N- (a (()) (Ar + Ba)
madel
from which the optimal control law is obtained as
(N) u4 (x)
arg min H (N) (x, a,
acR
There are several zeroth-order approaches to solve the minimization problem, e.g., by constructing 2-point estimators for the gradient. Since the objective function is quadratic and the matrix R is known, m queries of H(N) (z. t) are sufficient to compute un (N) (z).
2.3 ENKF-BASED ALGORITHM FOR THE LQG CONTROLLER
By combining the result of steps and 2, the optimal control input
k2(N) X(N)
if known
U,
U (N) (N) O.W.
The overall algorithm including steps 1, 2 and 3 is tabulated in the Appendix C where additional remarks on numerical approximation of backward SDEs also appear.
2.4 CONVERGENCE AND ERROR ANALYSTS
The mean-field process (9) represents the mean-field limit of the finite-N system 11. as the number of particles N + co. The convergence analysis is delicate matter based on the propagation of chaos (Bishop & Del Moral 2018). In Appendix D under certain additional assumptions on system matrices, the following error bound is derived:
E[|l E[||s[
5,112 <
S1
e -44 -4 X + Ovar
N OstST
(14)
N
where ya and Gvar are positive constants and || is the Frobenius (matrix) norm.
The computational complexity of ENKF is (Nd)). EnKF is workhorse in applications such as weather prediction where models are simulation-based (Evensen, 2006) Reich & Cotter 2015). In these applications, the number of simulations N <<<
tFq. for the optimal control gain matrix k(\) is dual the eq. 6 for optimal filter gain matrix T. (N)

---

Under review as a conference paper at ICLR 2022
2.5 COMPARISON WITH RELATED WORK
Classical RL algorithms for the LQR problem are based on linear function approximation, us- ing quadratic basis functions, of the value function or its surrogate the Q-function (Bradtke et al. 1994). Convergence guarantees typically require (i) persistence of excitation condition, and (ii) use of the on policy methods whereby the parameters are learned for a given fixed policy (which is subsequently improved). For the deterministic LQR problem, the persistence of excitation condi- tion is difficult to justify theoretically, and in practice can lead to poor performance related to slow convergence rates. These limitations have spurred recent research on the L.QR problem.
Our algorithm is compared to the model-free policy optimization based methods of [Mohammadi et al. (2021a) and Fazel et al. (2018). In policy optimization, the value J(U) is minimized over the search space of stabilizing gain matrices, using gradient-descent approach, starting from stabilizing controller (which is not straightforward to obtain in a model-free setting). For each perturbation of the gain matrix, the gradient is estimated by simulating N particles (trajectories) over a time- horizon [0, Tİ. Multiple iterations are needed to converge to the minimizer.
In contrast, using the EnKF, the matrix Pe is approximated by simulating N particles once over the time-horizon [0, 7). The optimal control is obtained from an application of the minimum principle that only requires m evaluations at each & (the minimization of Hamiltonian is online calculation).
The trade-off between the two approaches is as follows: While policy optimization methods require multiple iterations with small number of particles, the EnKF requires only single iteration with relatively larger number of particles. As illustrated with the aid of numerical examples in Sec. this can lead an order of magnitude gain in the computation time.
As final point, our paper provides a simulation-base based algorithm for the most general class of linear quadratic problem: fully or partially observed, finite or harinite-horizon, stochastic or deterministic, and furthermore does not require an initial stabilizing controller.
Duality: In Appendix E is shown that the Gaussian density of the random variable Y, equals a log transformation of the value function (at time t) of the LQG optimal control problem. Additional remarks are also included to expound the duality roots of the proposed algorithm, including survey of the relevant literature on this beautiful subject.
3 NUMERICAL SIMULATIONS
In all the following three numerical examples, we consider the infinite-horizn fully observed deter- ministic LQQ problem. Although our algorithms are more generally applied, such a choice allows us compare and contrast with the recent RL work on the LQR problem. For the LQR problem, the solution of the ARE is denoted Poo and the associated optimal feedback gain is denoted K
In each of the three numerical examples, time-horizon T' is fixed. Over the time-horizon, the finite- N EFKF algorithm (11 is run to obtain an empirical approximation (P(N) E Rdxdd 0 < t S T)- For the sake of comparison, the exact (P. E Rdxd 0 < T) is obtained by numerically solving the backward DRE 7). Typically, we chose Pr = 1, the identity matrix.
3.1 EXPONENTIAL CONVERGENCE
An attractive feature of the proposed EnKF algorithm is that the mean-field limit is exact. This means the EnKF recovers the properties of the DRE in the limit. One such property is the exponential convergence: For any fixed t, P.- + Poo as T' co, starting from any initialization Pr (Ocone
& Pardoux 1996. Remark 2.1). Moreover, because of the emror formula (4). ((N) _ Poo with a small additional bias that decays (
For a = dimensional system, Fig. T depicts this exponential convergence of the four entries of the symmetric P!N*) matrix (using N = 100 particles). As part of Appendix F.1 additional figures are included to depict exponential convergence of the 100 entries of the 10 x 10 matrix p(()) for a d 10 dimensional system (using N = 1000 particles). In these evaluations, the system (A, B) is chosen to be in its controllable canonical form where the appropriate entries (last row) of the A matrix are randomly generated. Because these randomly generated, the matrix A is
7

---

Under review as a conference paper at ICLR 2022
Ol,
X
CL.
FaRF
DRR
ARF.
18
Im
2.00
0.32
1.6
0.6
0.00
1.50
0.4
00.25
1.25
0.2
0.50
P1 1.00
Pa 0.0
Pa
Pie
1.0
-0.5
0.0
(4)
(h)
(0)
(4)
Figure
The d example: (a) Open-loop (OL) and closed-loop (CL) eigenvalues; (b)-(d)
Convergence of P.N: Plot of the entries of the matrix P(N) shows exponential convergence, with a small (random) error, to the ARE limit Poo which also depicted together with the exact DRE solution. (Note the r-axis of the plots (b)-(d) is T t so convergence is easy to see).
often unstable (see also Fig. (a)) which does not pose any problem with our algorithm. Additional details on the numerics appear as part of Appendix E.
3.2 EVALUATION AND COMPARISON WITH LITERATURE: MASS SPRING DAMPER SYSTEM
In order to evaluate and compare the performance of the proposed algorithm, we describe next the results for the coupled mass-spring system studied in Mohammadi et al. (2019). For each mass, there are two states, position and velocity. system with d/2 masses d-dimensional. To evaluate the performance of the finite-N algorithm, we consider the following error metric:
P.
12 P.
P (N
MSE -E T
dt
The expectation is approximated empirically by averaging 100 simulation runs. See Appendix F2 for details on modeling, parameter values, and the numerical discretization. Fig.2 depicts the results of the numerical experiments showing the ( decay of MSE N increases (for fixed).
Part (C) of the figure depicts a comparison with the two state-of-the-art algorithms, namely Moham- madi al. 2021b) and Fazel al. (2018). In the figure, computation time is plotted against the relative error in approximating the LQR gain Koo (plots are qualitatively similar using other error metrics). The computation time is obtained using the Python process. ssmtime
function from the t time library for measuring execution time. The result depicted in the figure is for d = 10 but qualitatively similar results are also obtained for other values of d. See Appendix F33 for details on numerical implementation of the algorithms. We also implemented the classical RL algorithm Bradtke et (1994) but convergence was not reliable because of the persistence of excitation issues.
13P
100
1000
101
Camp. Timer (6)
MS5
MSS
10F
10m4
12/
ExXF
10
[718
102
30 40 60
80
0.05
11
0.15
Number of particles (N)
Dimemssions
(d)
Relative gain
Figure 2: (a)-(b) Plot of MSE function of the number of particles N and system dimension d: (c) Comparison with algorithms in Fazel et al. | (0118) (labeled [F18]) and Mohammadi al. (202Tb) (labeled [M21]). The comparisons depict the computation time (in Python implementation) as function relative error in approximating the LQR gain Koo.
8

---

Under review as a conference paper at ICLR 2022
Based on the preliminary analysis, the main reason for the order of magnitude improvement in computational time is as follows: An EnKF requires only a single iteration over a fixed time-horizon [. Although the number of particles (N) is order of magnitude larger for the EnKF algorithm, certain vectorisation features of numpy yield significant gains in computational time. Because the other algorithms require multiple iterations which must necessarily be carried out serially, these computations are slower. In our comparisons, we used the same time-horizon ' and discretization time-step òt for all the algorithms. optimized to improve the performance of the other algorithms. In particular, one may consider shorter or longer time-horizon T or use parallelization to speed up the gradient calculation.
certainly possible that some of these parameters can be
3.3 EVALUATION OF ENKF FOR NONLINEAR CART-POLE SYSTEM
It is time-honored practice to design the optimal control law for (linearized) LQR model and then implement this control law on a nonlinear system. By the principle of linearization, the control law works well (in theory) for small perturbations and often (in practice) for even large ones. We consider the nonlinear conservative cart pole model (Tedrake] Rawlik et al. 2013). The control acts external foree applied to the cart. The four- diun-dimentional state for the system is (0, 4., v), where 0 E 5 (the circle) is the angle of the pole (pendulum) as measured from the stable equilibrium,
x E R is the displacement of cart along the horizontal, and (w, v) == (6, àt) E R2 is the velocity vector. The control objective is balance the pole stabilize the system the inverted equilibrium (T, 0, 0)... assuming full state feedback. For the purposes of control design, the nonlinear system first linearized the desired equilibrium and an LQR problem formulated (see Appendix [F4 for details). For the purposes of evaluation, the optimal control is applied to the nonlinear model. Figure
33 depict the results of numerical experiments for three different choices of N. is seen that as few as N = 10 particles are sufficient to stabilize the equilibrium. Using N = 1000 particles, the closed-loop trajectories are virtually indistinguishable from the DRE-based solution.
1.26c
Angle 0 (zad)
N 10 particles
N 100 particles
1000 particles
Position (m)
-4
EnKF
DRE
10
time
Figure 3: Trajectories of the closed-loop nonlinear cart pole system.
4 CONCLUSIONS
In recent years, there has been a concerted effort to revisit the LQR problem in the context of RL (Fazel et al., 2018 Mohammadi al. 2021b Tu & Recht 2019 Dean al. 2020 Malik al.) 2020 Zhang et al. 2020 lansch-Porto et al., 2020). The effort is largely in response to the well known issues that arise when the state and action space are continuous (Lu al. 202T).
In this paper, we present new paradigm for RL. There are two key innovations: (i) the representa- tion of the unknown value function in terms of the statistics (variance) of the particles; and (ii) design of interactions between simulations to solve the optimal control problem. For the LQR problem, this the
shown to yield learning rate that closely approximates the exponential rate of convergence solution of the DRE. In numerical examples, this property is shown to lead to order of magnitude better performance than the state-of- the-art art algorithms. Given the enormous success of EnKF in data assimilation (Evensen 2006 Reich & Cotter 2015), the contributions of this paper potentially open up new opportunities for RL. is our hope that the paper will engender new synergies between the data assimilation and the RL communities.

---

Under review as a conference paper at ICLR 2022
5 REPRODUCIBILITY STATEMENT
The implementable pscudo-code of the algorithm is tabulated in Appendix C For the three exam- ples in Sec. the numerical parameters are tabulated in Appendix The codes for all numerical simulations are provided as part of the supplementary material. All the codes have a README file associated with it. Table provides list of the Appendix containing the simulation parameters and location of codes for each numerical example.
Table I: Location of supplementary material
Section
Appendix
Code Directory
3.1
Section 3-1
3.2
R22
F33
Section 3-2
3.3
F.4
Section 3-3
REFERENCES
Benner and Z. Bujanović. On the solution of large-scale algebraic Riccati equations by using low-dimensional invariant subspaces. Linear Algebra and its Applications, 488:430-455 2016.
K. Bergemann and S. Reich. An ensemble Kalman-Buce Bucy filter for continuous data assimilation. Meteorologische Zeitschrijt, 21(3):213-19, 2012. doi: 10.1
A. N. Bishop and P. Del Moral. On the stability of matrix-valued Riccati diffusions. arXiv preprint arXiv:1808.0022 00235, 2018. URL/ttts:: http://aaavvvvvvv wttp///g//baa/aaaaee///b/////a/e 00235
A. N. Bishop. P. Del Moral, K. Kamatani, B. Remillard, et al. On one-dimensional riccati diffusions. Annals of Applied Probability, 29(2): 127-11127711877 2019. 2019.
Sergio Bittanti, Alan J Laub, and Jan C Willems. The Riccati Equation. Springer Science & Business Media, 2012.
S.J. Bradtke, B.E. Ydstie, and A.G. Barto. Adaptive linear quadratic control using policy iteration. In Proceedings of 1994 American Control Conference ACC '94, volume 3, pp. 3475-3479 vol.3, 1994. doi: 10. 10.1109/ACC.11 1994.732244
R. W Brockett. Finite dimensional linear systems. SIAM, 2015.
Dean, H. Mania, N. Matni, B. Recht, and Tu. On the Sample Complexity of the Linear Quadratic Regulator. Found Comput Math, 20(4):633-679, August 2020. ISSN 1615-3383. doi:
(007///1000-009--944266 s1 s1020B-00 01088-019-0 010-0-119-9-------1
URL https:/ //t::////o...// http:///......g..111111 .rr./1..
P. Del Moral and J. Tugaut. On the stability and the uniform propagation of chaos proper- ties of ensemble KalmanBucy filters. Ann. Appl. Probab., 28(2):790 75000, 2018. doi: 10.1214/1-AAP1317
URL.tttss://aa httt:s::/ddi..rg//11
211/17-AAPPP 1317
A. M. Devraj, A. Busic, and S. Meyn. Fundamental design principles for reinforcement learning algorithms. K. Vamvoudakis, Y.Wan, F. L. Lewis, and D. Cansever (eds.), Handbook on Reinforeement Learning and Control. Springer, 2020.
G. Evensen. Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. Journal of Geophysical Research: Oceans, 99(C5): 10162, 1994. doi: 10.1029/941C00572.
G. Evensen. Data Assimilation. The Ensemble Kalman Filter. Soringer-Verlag New York, 2006.
M. Fazel, R. Ge, S. Kakade, and M. Mesbahi. Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator. In International Conference Machine Learning, pp. 1467- 1476. PMLR, July 2018. URL /attp::/proceed/atce mlr. www..rres................... (aness/v80/E5
html ISSN: 3498-
10

---

Under review as a conference paper at ICLR 2022
W. H. Fleming. Exit probabilities and optimal stochastic control. Applied Mathematics and Opti- mization, 4(1):329-344 1977.
W. H. Fleming and S. Mitter. Optimal Control and Nonlinear Filtering for Nondegenerate Diffusion Processes. Stochastics, 8(1):63-77, January 1982. ISSN 0090-9491. doi: 10. 108/0 17442508208833228. URL https:t/t///ooo http:////oooor///..../oo// 1ttp://....0077742222222222220
S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596. PMLR, 10-15 Jul 2018. URL BtLLttt::/// httt:::/pppoceddngs...... www.mm..ppp.. mtt.ppess//v.//
EujimotolBa.htm
T. T. Georgiou and A. Lindquist. The separation principle in stochastic control, redux. IEEE Trans- actions on Automatic Control, 58(10):2481-22 122944, 2013.
C. Hartmann and C. Schütte. Efficient event simulation by optimal nonequilibrium forcing. Journal of Statistical Mechanics: Theory and Experiment, 2012(1 1)::P1 1004, 2012.
C. Hoffmann and P. Rostalski. Linear optimal control on factor graphs message passing per- spective IFAC-PupersOnnL.nne 30(1):6314-66) 4-131,,,949 2017. ISSN 2405-8963. doi: https://doi. 100100000166fffcco22000.....0000 URL https: /tt///wwww www. sciencedirect.. wtt.e/ccc/cccccceeeeeeee Atticiees111/224558911111
20th IFAC World Congress.
P. L. Houtekamer and H. L. Mitchell. A sequential ensemble Kalman filter for atmospheric data assimilation. Monthly Weather Review, 129(1):12--137, 2001.
J. P. Jansch-Porto. B. Hu, and G. E. Dullerud. Convergence guarantees of policy optimization methods for Markovian jump linear systems. In 2020 American Control Conference (ACC), pp- 2882 2887. IEEE, 2020.
R. E. Kalman. Contributions to the theory of optimal control. Boletin de la Sociedad Matematica Mexicana (2), 5:102-102 109, 1960.
H. J. Kappen and H. C. Ruiz. Adaptive importance sampling for control and inference. Journal of Statistical Physics, 162(5)) 2016.
H. J. Kappen, V. Gómez, and M. Opper. Optimal control as graphical model inference problem. Machine Learning, 87(2):159-182 May 2012. ISSN 1573-0565. doi: 10.1007/ 510994-012-5278-7 URL, lo 10.7000.1100///
J. W. Kim and P. G. Mehta. An optimal control derivation of nonlinear smoothing equations. In Proceedings of the Workshop on Dynamics, Optimization and Computation held in honor of the 60th birthday of Michael Dellnitz, pp. 295-311. Springer, 2020.
Peter E Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations.
Springer, Applications of Mathematics, 1999.
Huibert Kwakermakk and Raphael Sivan. Linear optimal control systems. Wiley Interscience, New York, 1972. ISBN 0471511102.
Lancaster and Rodman. Algebraic Riccati Equations. Clarendon press, 1995.
A. J. Laub. Invariant subspace methods for the numerical solution of Riccati equations. In The Riccati Equation, pp. 163-196. Springer, 1991.
Daniel Liberzon. Calculus of Variations and Optimal Control Theory. Princeton University Press, Princeton, NJ, 2012. ISBN 978-0-691--5187-8. A concise introduction.
Lu, P. G. Mchta, S. P. Meyn, and G. Neu. Convex q-learning. In 2021 American Control Confer- (ACC), pp. 4749-4756, 2021. doi: 10.23919/ACC50S1 09241.94

---

Under review as a conference paper at ICLR 2022
H. R. Maei, C. Szepesvári, S. Bhatnagar, and R. S. Sutton. Toward off-policy learnigg control with function approximation. In Proceedings of the 27th International Conference on International Conference on Machine Learning. ICML' 10, pp. 719726, Madison, WI, USA, 2010. Omnipress. ISBN 9781 18558897777
D. Malik, A.n Pananjady, K. Bhatia, K. Khamaru, P. L. Bartlett, and M. J. Wainwright. Derivative- Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems. Journal of Machine Learning Research, 21(21):1-51, 2020. ISSN 1533-7928.
//cttp:/jmm..../ htll:mmlrrro. .rg/
papers/v21/192 198. html
P. G. Mehta and S. P. Meyn. Q-learning and Pontryagin's 's minimum principle. In Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference, pp. 3598-3605 IEEE, 2009.
S. Melo, P. Meyn, and M. Ribeiro. An analysis of reinforcement learning with func- tion approximation. In Proceedings of the 25th International Conference Machine Learn- ing, ICML, "08, pp. 664671, New York, NY, USA, 2008 Association for Computing Machin- doi: 10.1145/1390156.1133
ery. ISBN 9781605582054.
http:://doi..oo
1145/1390156. 1390224
S. K. Mitter and N. J. Newton. A variational approach to nonlinear estimation. SIAM journal on control and optimization, 42(5):1813-1825 2003.
H. Mohammadi, A. Zare, M. Soltanolkotabi, and M. R. Jovanovic. Global exponential convergence of gradient methods over the nonconvex landscape of the linear quadratic regulator. In 2019 IEEE 58th Conference on Decision and Control (CDC), pp. 7474-7479, December 2019. doi: 10. M0111990CDC400222099999999999999 999.99998...
ISSN: 2576-2370.
H. Mohammadi, M. R. Jovanovic, and M. Soltanolkotabi. Learning the model-free linear quadratic regulator via random search. In Learning for Dynamics and Control, pp. 531-539. PMLR, July 2020a. URL/http: .ml.nng................m......mm 2tt..eeeeiiinn.......esss///1110 2640-3498.
htt00//chhmmmaaa00/mm html] ISSN:
H. Mohammadi, M. Soltanolkotabi, and M. R. Jovanovic. Random search for learning the linear quadratic regulator. In 2020 American Control Conference (ACC), pp. 4798 4803, July 2020b. doi:
ISSN: 2378-5861.
H. Mohammadi, M. Soltanolkotabi, and M. R. Jovanovic. On the Linear Convergence of Random Search for Discrete-Time LQR. IEEE Control Systems Letters, 5(3):989-994, July 302la. ISSN 2475-1456.
doi: 10. 10.1109LCSSY.22003002222 Conference Name: IEEE Control Systems
Letters.
Hesameddin Mohammadi, Armin Zare, Mahdi Soltanolkotabi, and Mihailo R. Jovanovic. Conver- gence and sample complexity of gradient methods for the model-free linear quadratic regulator problem. IEEE Transactions Automatic Control, pp. 1-1, 2021b. doi: 10. 10.1109/TAC.2021. 3087455.
Hesameddin Mohammadi, Armin Zare, Mahdi Soltanolkotabi, and Mihailo R. Jovanovi. Conver- gence and sample complexity of gradient methods for the model-free linear quadratic regula- tor problem. arXiv: arXiv: /9/2. 1899 [physics], March 2021c URL. arXiv: 1912.11899.
1912. 11899
D. Nualart and É. Pardoux. Stochastic calculus with anticipating integrands. Probability Theory and Related Fields, 7854)5535-581, 1988.
D. Ocone and E. Pardoux. Asymptotic stability of the optimal filter with respect to its initial condition. SIAM Journal Control and Optimization, 34(1):226-243, 1996. doi: 10.1137/ s0363012993256617,
K. Rawlik, M. Toussaint, and S. Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. In Twenty-third international joint conference on artificial intelligence, 2013.
12

---

Under review as a conference paper at ICLR 2022
S. Reich and C. Cotter. Probabilistic forecasting and Bayesian data assimilation. Cambridge Uni- versity Press, 2015.
C. Schüttte, Winkelmann, and C. Hartmann. Optimal control molecular dynamics using markov state models. Mathematical programming, 134(1):259-288 2012.
Sutter, A. Ganguly, and H. Koeppl. variational approach to path estimation and parameter inference of hidden diffusion processes. Journal of Machine Learning Research, 17:6544-80, 2016.
A. Taghvaci and P. G. Mehta. An optimal transport formulation of the ensemble Kalman filter. IEEE Transactions on Automatic Control, pp. 1-1, 2020. doi: 10. 00.1009TA..2220.011555..0
Amirhossein Taghvaci. Design and analysis of particle-based algorithms for nonlinear filtering and sampling. PhD thesis, University of Illinois at Urbana-Champaign, 2019.
Amirhossein Taghvaci and Prashant G Mehta. Error analysis for the linear feedback particle filter. In 2018 Annual American Control Conference (ACC), pp. 4261-4266. IEEE, 2018. doi: 10.23919/ ACC.2018.88 8.8430867.
R. Tedrake. Underactuated Robotics: Algorithms for Walking, Running, Swimming, Flying, and Manipulation (course Notes for MIT 6.832). URL htto://mndeaatttteeed.. edu/ Last accessed on 16 May 2021.
E. Theodorou, J. Buchli, and S. Schaal. generalized path integral control approach reinforce- ment learming. The Journal of Machine Learning Research, 11:3137-3181, 2010.
Todorov. General duality between optimal control and estimation. 2008 47th IEEE Conference Decision and Control, pp.
Dec 2008
E. Todorov. Efficient computation of optimal actions. Proceedings of the national academy of sciences, 106(28):1 1e11-11488-11955 2009.
M. Toussaint and A. Storkey. Probabilistic inference for solving discrete and continuous state markov decision processes, ICML '06, pp. 945952, New York, NY. USA, 2006. Associa- tion for Computing Machinery. ISBN 1595933832. doi: 10.1145/1143844.1 1143963. URL
hattps://doi.orq// 1111:/....1111444...4414 hLLE:////B...or//....//1144444..
S. Tu and B. Recht. The Gap Between Model-Based and Model-Free Methods on the Linear Quadratic Regulator: An Asymptotic Viewpoint. In Conference on Learning Theory, pp. 3036 3083. PMLR. June 2019. ISSN: 2640-3498.
/ttp:///ss/...prpe///////////// 9a html
R. Van Handel. Filtering, stability, and robustness. PhD thesis, California Institute of Technology, 2006.
R. Van Handel. Stochastic calculus, filtering, and stochastic control. Course notes., URL http:///ww. princeton. cdl//vandccm22177ACM211,, pdf. 14, 2007.
J. Watson and J. Peters. Advancing trajectory optimization with approximate inference: Exploration, covariance control and adaptive risk. In 2021 American Control Conference (ACC), pp. 1231- 1236, 2021. doi:
J. Watson, H. Abdulsamad, and J. Peters. Stochastic optimal control as approximate input inference. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Confer- ence on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 697- 716. PMLR, 30 Oct-01 Nov 2020. URL wat tatton20a... 20t.tmm
Watson, H. Abdulsamad, R. Findeisen, and J. Peters. Stochastic control through approximate bayesian input inference, 2021.
13

---

Under review as a conference paper at ICLR 2022
K. Zhang, B. Hu, and T. Basar. On the Stability and Convergence of Robust Adversarial Reinforce- ment Learning: Case Study on Linear Quadratic Systems. Advances in Neural Information Processing Systems, 22068, 2020. /ttpps///c/cccciaiigg neurips ec/ /capet//000/haahh/bb222233322323 6dee 5934 4eleee444e88977 -bstt .tml
APPENDIX
A PROOF OF PROP. 1
It is first shown that the conditional mean m, and the conditional covariance E, evolve according to the Kalman filter equations (3) The SDE for the mean obtained by taking the conditional expectation of
dm = Am, dt + BU dt + L, dZ Hm, dt)
where we used the fact that Ug assumed be Z, measurable.
To obtain the equation for the covariance, define the error process e, = Xx m which evolves
according
de, = (A L H) + dE.
Then, upon an application of the Itō rule
d(c;e,) = (A (e(e) dt + (ete, (A
[H) dt + Qdt + daef + e dE,
2
and taking the conditional expectation, the equation for the conditional covariance Eg = Elete] Zz] obtained
d E, A5, A + E, HS.
dt
where we used the definition L, Σ.
The equation for the conditional covariance Σ, is identical to the DRE (b).. Therefore, if the initial condition Eg = No then Σ, = M, for all > 0. This also implics Le Lt, which turn implics that the SDE for the conditional mean m, is identical to Kalman filter equation for my If my mo then m = m, for all 0.
By replacing the mean-field terms mh and Σ, by exogenous processes m, and Σ, the McKean- Vlasov SDE (4) simplifies to an Orstein- Uhlenbeck SDE. Because the distribution of the initial condition X, Gaussian, the distribution of X, is also Gaussian and given by N(m,, Ex).
B PROOF PROP. 2
The proof is similar to the proof of Prop. I The equation for the mean ñ, is obtained by taking the expectation SDE 9
dn, (A + S,C S,CTC)), dt
Because är 0, have n; 0 for all [0,
The equation for the covariance 5, is obtained by writing the SDE for the error e, == -b. - n.
de; = (A+ _ (CeTT), dt + B Bdn.
2
Using the Itô rule for T
d(ere,) = (A+ s ()(ee) de (ere, (0 -BR +Bd
The Itō correction term appears with negative sign because the SDE involves backward Wiener
process 77 (Nualart & Pardoux for 5y:
1988 Sec. 4.2). Taking an expectation yields the following equation
5, = (A+ 3,CTC)s 5S(A + 3,C о)T -RR BR-
dt
2
14

---

Under review as a conference paper at ICLR 2022
The SDE is identical to the SDE for St. Because ST Sr, we have S, S, for all t j0. The conclusion that )/ is Gaussian follows from the fact that with A. = ne and S. = St, the SDE for ) is an Ornstein- Uelenbehhhekkkk SDE with Gaussian terminal condition.
The proof for the rest of proposition is straightforward. By definition.
E E[S; (D) m.)] S (1e - R4) =0 0
E| = E| - n.)( - n.)) -
Since E| Pr. the optimal gain matrix K, RR- ElY,Y, For given E R&, the optimal control ue K2 = - B Y, Ja is the unique minimizer of the Hamiltonian. This is referred to the minimum principle of optimal control.
C DETAILS OF THE ALGORITHMS TO SOLVE THE PARTIALLY OBSERVED LQG PROBLEM IN SEC. 2
In this section, the implementation details the EnKF-based algorithms are presented to numeri- cally solve the partially observed LQG problem (1)-(2) introduced in the main body of the paper. For better readability, the overall algorithm is broken down into three separate algorithms:
1. Algorithm is an offline algorithm. is based on the finite-N approximation of the (con- trol) EnKF as described in Sec. 2.2 of the main body of the paper. The algorithm is run offline to obtain an approximation of the {P, : < t < TT.
2. Algorithm 2 an online algorithm. is based on the finite-N approximation of the (filter) EnKF as described in Sec. 2.1 of the main body of the paper. The algorithm is run online in real-time manner. At each time step, processes the sensor measurements from the true system (plant) and computes the optimal control input by calling Algorithm 3.
3. Algorithm computes the optimal control input. It is based upon minimization of the Hamiltonian function. In online implementation, the optimal control input applied to the plant at each time step.
The input structure of each of the three algorithms is clearly delineated. In particular, the algorithms require only the simulator in the form of the function evaluator f(r u) Ar+ Bu for the dynamics, c(x) Ca for the cost function, and h((x) = Hx for the observation function. The algorithms do not require solution of the DRE.
Remark 1. In numerical implementation of step I and step 2. there two sources of error:
(i) error on account of finite-N appreximation; and (ii) error on account of time discretization which depends upon the step size At. The finite-N approximation error has been studied in the
EnKF literature where is shown that the error approximating [| -P( || (in step 1) and the
error in approximating the gain the mean ||X, x (N) (in step 2) converges
with the rate 0 Del Moral Tugaut (2018); Bishop et al. (2019); Bishop Del Moral (2018); Taghvaei & Mehta (2018); Taghvaei (2019). The time discretization error is also expected to be bounded and converges to zero with the rate O (At) when the system is stable Kloeden & Platen 1999..
15

---

Under review as a conference paper at ICLR 2022
Algorithm [offline] EnKF algorithm to approximate { 05
Input: Simulation time T, simulation step-size At, number of particles N, simulator f(z, u)
Ax + Bu, terminal cost Pr, cost function c(z) = Cr, and control cost matrix R.
Output: (P
0, 1, .... Ar
T
Tp T
2r
2: Pir) Pr
3: Initialize (1%- /==
itd (O, PT
4: calculate Trr N-1 E 3/+
(N)
5: for k = Tp to 1 do
6: Calculate (N) N Σ i==1 c(0%)
7: Calculate (N) = (N - -1) -f IT
8: for
N do
Anf. Lid NO0, At 1-
9:
AJ f()% Ark. At + M ( + +
10:
11:
1 V1 A
12:
end for
13: Calculate (N) N-1 -1 D
k-1
14: Calculate 5 (N) (N -1)- (0% -
1N
PM (s
15:
16: end for
Algorithm 2 [online] EnKF algorithm approximate state estimate X, and optimal control
Input: Simulation time T, simulation step-size AI, number of particles N, simulator f(x, u)
Ax Bu, initial distribution N(mo, Eo), process noise covariance R. observation function
M(x) = Hx, {PL =,=0,, 0, 1,,,..., T from the offline algorithm F.I
2r Output: estimate (X(N) k 0,1,2 Ar 0, 2, A7 1}.
T and optimal control input {up (N) :
1: Define TF == T
A7
2: Initialize particles (X fl
Lid N(m,, Ex)
3: for = to T5 do
4: Calculate X( (N)
N ΣN X
5: Calculate (N)
arg min, a H(X (M X aa) from algorithm
dg
6: Apply control u (N)
to the system and obtain the observation AZv Z(x+1)4 -Kxa
7: Calculate i
N ΣN n(x²)
8: Calculate I (N) N-1 Ex - (((()) (
1 IN
9:
for = 1 to N do
Aet. Lid NO QAt)
10:
AXÚ f(X,, f un
(N))
At AGj + L (AZ
A At)
X X X AX
13: end for
end for
16

---

Under review as a conference paper at ICLR 2022
Algorithm 3 Computation of optimal control
Input: state 2, momentum y, control cost matrix
Hamiltonian H a) = y" (a(x) + b(z)a) + 2 dl(x)]² 2/ Ro.
Output: optimal control arg min, M(, y.a)
1: if b(x) is known then
2: u - R- bxx)
3: else
4: for k to m do
5: Ek [0, ...,
...... 0] R
componeot
6: ug H(x, y, R ex) H(x y. 0) - 2)(R 1) kk
7: end for
8:
end if
D ERROR ANALYSIS
The objective of the error analysis is to study the convergence of the empirical variance of the particles s(x² to its mean-field limit Sy N - According to the Proposition the mean-field limit 5t = St hence evolves backward in time according to
d S, AS, + 3,A + S, BT di
ST = Pr
(15)
In order to study the error of S
s St, we obtain stochastic differential equation for the evolution
Lemma 1. The empirical covariance matrix S(M, defined in 12 evolves according to the stochas-
tic differential equation
ds! (AS( +S. A(
CT BR- dt + dMr.
(16)
where M2 is Martingale given by
N (Bdi)) + +
dM N -
e Vi -
(N)
with quadratic variation
d(M), = N MMB )S()) + BR BR + BR-'B" s(M) + BR BBT
Proof. The evolution for the eroor process ef = 3 n2 is obtained by subtracting the evolution
(N)
for the empirical mean na (N)
from III
de; = Aet + B( di
B Bd + S.M dt
N
5=1
The evolution for the empirical covariance s of the Itô rule.
Σ ef(et)" is obtained by application
N-T
Note that the first term in the evolution of (N 16 exactly identical that of S, in 15). while the second term is an additional stochastic term that converges to zero as N the fluctuations scale as O(N- ). the analysis is challenging as has been noted in literature (see the remark after Theorem 3.1 in (Del Moral & Tugaut) 2018)). Error analysis of the (stochastic)
Even though
ensemble Kalman filter has been carried out in the literature (Del Moral & Tugaut 2018 Bishop & Del Moral 2018) under additional assumptions which we also make below:
Assumption 1. The matrix C C is identity. And the control matrix B is full rank.
17

---

Under review as a conference paper at ICLR 2022
The assumption regarding the cost matrix C' can be relaxed to C* C' being non-singular by change of coordinates. The assumption regarding the control matrix is strong. It is an open problem in the literature regarding stability of EnKF to carry out the error analysis under the more natural assumption that the system is controllable and observable.
Under assumption 1 we prove the main result stated in the following proposition.
Proposition 3. Let S, be the mean-field covariance defined in (10) and St be the empirical co- variance of the particles defined in 16). Then, under Assumption the error between s(x) and S. satisfies the apper-bount
3cp Pr
E - S
14p= (TT
0 OstST
(17)
N
N
where Moo: Cus Cvar positive constants that independent of time t.
In the remainder of this section, we prove the proposition 3 To simplify the presentation, we use the time-reversed quantitative 02,0 5T4 and 94 ST which evolve forward in time according
to
dl, = (-As; - Q - 2.0TTTT.S Co + BR-BBBJ dn) (-An, - / AT M CT BR-BBB di dM,
(18)
(19)
where M, is defined in Lemmal
First, we show some basic results regarding the asymptotic properties the Ricatti equation that governs Qz These results straightforward application of the classical stability theory of the Ricatti equation.
Lemma 2. Consider the Ricatti equation
18 that governs Qr. Then,
1. There exists a positive definite matrix solution Doo to the ARE
0 -AA9> - -Dx Co23 + -R-1B"
such that -A Aox CT C is Hurwitz.
2. 9r converges to Dox exponentially fast
3. Let b4 denote the minimum eigenvalue for the matrix F; == A C C. Then, there exists a positive constant P > and cp such that
-i pr Cne Poot V2
(20)
Proof. Under Assumption| 1 the pair (A, BR +) is controllable and the pair (A, C)) is observable.
Part (1) follows from (Kwakernaak & Sivan 1972, Theorem 3.7) and part (2) follows from (Ocone
& Pardoux 1996. Lemma 2,2 and Theorem 2.3). The proof of part (3) follows from the following
two facts (i) (-A - 1
C) -
Hurwitz because
-BR--BT = (-A -x C)sx Poo N.
1
1
is Lyapunov equation, with BR-'BT positive definite since B is assumed to be full rank; and (ii) Q. converges to Qoo exponentially fast. Therefore, there exists a time Ti > such that (-A A
1N,C* is Hurwitz for 2 Ti. As result, the minimum eigenvalue μt of Fr A + 1 C is
lower-bounded 3 2 Ps > for 2 TI. This concludes the bound 200 where C₂ = E 1o p.
Next, we obtain bound for E(Tr(S2, ()) Upon taking the expectation and trace of 19
d dt
}[ (r((EE (M (AT) ur(E(s) (N) oT ( +
L ( (r(E(S) IAT) u<(Es)) (1 (E(O) (BRR-
18

---

Under review as a conference paper at ICLR 2022
where we used E[Tr(X X)] 2 Tr(E(X]E(X]T) for X C
conclude the inequality, Then,
by inspecting the equation for Tr(2+), it follows that
E(Tr(S)( ) S Tr(S,) S Fo
(21)
where Eo is uniformly in time bounded constant, because S2 converges to Ωoo.
Next, we obtain the bound for the error 2 S2, Subtracting (8) from (9) yields
d(n)) () Q4) (-A(s) - 22,
(2 - A4) -12 (M) -22) -2.C (C))CCC( 9te)
-(22 22)
CΩ₁ (C,)- -2, C(() - -(t))dl + dM,
Then, by application of the Itô rule to (52, -2) [(S( -2) (I2 22,2 and taking the trace,
aTr((S2, -2,32) 20r((0, (2,))) - -C CD. cTCC,
24C ())dt
2tr(C C(1, - -22) u((dM ) 24))
+ (1((2()) BR- BT) +
BB)B di
Upon taking the expectation and defining O E(tr((n) -
11)
d
ir(-A A - C. (2) - -2,)3 - -2E [ur(oT (r(o ( -2,)3)
θ. S 2E
dt
θ + 2 tr(s)) + 22 )tr((e))
where we used the bound (11). and Tr(XY) 2 TT(X TT for X -2 Ωt and
BR-1B B..
In order to continue with proving the bound, we use the assumption CTC = / to conclude
d θ < 2E
tr(-A- - A - ( - -
dt
θ, 20((BB-BBT)) (1( tr(()))
because tr( (12 (N) - 22) N Morevoer, due to exponential convergence of 24 to Dosr tr(s))) < E uniformly. Thus,
θ S 4, (e. + 2tr(
2F (BR + E1)
dt where μ is the minimum eigenvalue for 542 Therefore,
θ OnE 4 So ppdt' bt
Cvar N
OocRe 4prot Cuar
where Cuar -2r((BR-1BT)) + tr BBT
F and used (20)
The proof of the
proposition follows by substitution s(N) St = (2 St and the bound Oo
Prlle
E DUALITY ROOTS OF THE PROPOSED ALGORITHM
Fort E (0,T), the LQG value function is defined as follows
(x(x) = min E (J," (HIXx + LUJ RU.) ds + X Pr X, = (U,+S+TT)
(22)
From the DP optimality principle, it is easily shown Liberzon 2012) that
The value function v,(x:) 1x Pex (constant) is quadratic where
The matrix-valued process (P: e Rdxd - < < T) solves the DRE with terminal condition Pr at time / T.
19

---

Under review as a conference paper at ICLR 2022
Log transformation: The log transformation is manifestation of the duality between optimal control and optimal estimation Mitter & Newton (2003): Todorov 2008). Define the probability density pr()) e=((() Fleming & Mitter (1982) : Fleming 1977. For the LQG problem, o quadratic and therefore, Pt Gaussian density with variance P. St. Indeed, the Gaussian density of the random variable Y, equals pt. The following proposition gives the precise connection:
Proposition 4. Suppose Pt x e where is the value function defined in 22 Let Pt be the density of the random variable Vi defined in ( Then, provided pr Pr.
(t(x) = pc(x), VIER" O
Proof. By Proposition 2 De is Gaussian random variable (0, where St P.- Therefore,
p-1by by definition.
1 xx" Pa (constant) (2(2)) + (constant) log((p(x)) (constant)
- log(p((z))
2-
Therefore, since (I) and Pe (z) are both normalized one, the constant is zero and Pc(x) Pa
In the past, duality has been used to:
(i) Obtain linearly solvable sampling algorithms to solve optimal control problems Kappen & Ruiz
2016;; Rawlik et al. (2013): Theodorou et al. (2010): Schütte et al. (2012 Todorov (2009); and
(ii Set up optimal control problems for the purposes of estimation and simulation Sutter et al. 2016); Hartmann & Schütte (2012); Van Handel (2006); Kim & Mchta (2020).
(iii) parallel approach to duality involves posing optimal control problems as problems of prob- abilistic inference, and using methods like expectation maximisation (Toussaint & Storkey 2006),
message passing (Hoffmann & Rostalski] 2017; Watson & Peters 2021: Watson et al. 2021 or
minimisation of KL divergence (Kappen al. 2012 Rawlik et al.] 2013 Watson et al. 2020).
Although novel and distinct from the algorithms described these carlier works which instead rely on the of Feynman-Kan Kac representation the proposed algorithm is example of a sampling algorithm to solve the optimal control problem. In a related work, we have extended the results presented here class of nonlinear optimal control problems. These nonlinear generalization more clearly reveals the duality roots of the proposed algorithm. The EnKF algorithm arises as a special case in the LQG settings. The nonlinear generalization will be a subject of future publication.
F DETAILS OF THE NUMERICAL EXAMPLES IN SEC. 3
Notation: I, E Raxa denotes the identity matrix and 1 Rd denotes a vector with all entries equal to 1.
F.1 EXPONENTIAL CONVERGENCE
The EnKF algorithm is implemented for the linear system ① to approximate the optimal gain ma- trix. The model matrices and B are in the control canonical form
1
0
0
0
0
B
02
0.
where the entries (a,,..., az) E Rd is selected randomly from (1166.1,, Iz). Additional model and simulation parameters are summarized in Table Table 3
The numerical result for single realization of the algorithm, for d is depicted in Figure 1 Part-(a) depicts the spectral properties of the open-loop (1 = Ar) and the closed-loop (i = (A
B BK! (x(N)) dynamics (recall that due the backward in time nature of the EnKF, Ko is the terminal
20

---

Under review as a conference paper at ICLR 2022
gain which the algorithm yields). It is observed that the algorithm leams the gain matrix that make the closed-loop dynamics stable, while the open loop dynamics A is unstable.
The entry-wise convergence of the matrix P(N) as goes from 7 to 0 is depicted in part (b)-(c)-(d), The result is compared with exact solution to the DRE Pe and the asymptomatic limit, as time horizon T oo, which is the solution to the ARE.
1.
Figure Open-loop (o) and closed loop eigenvalues (x) for the dimensional system.
Additional numerical results, for the case where d = 10, are depicted in Fig. for the open and closed-loop eigenvalues and in Fig. for the convergence of the 100 entries of the 10 10 matrix p(
Table 2: Model parameters for the LQR with random dynamics Numerical value
Model parameter
0ax1
En
0.1Za
C' for d =
V5Ia
C for
I,
1.0
Pr
I,
Table 3: Simulation parameters for LQR with random dynamics
Simulation parameter name
Symbol
Numerical value
Simulation time
T
10
Step size
A
0.02
21

---

Under review as a conference paper at ICLR 2022
(N) for d 10 dimensional system. Figure 5: Convergence of the 100 entries of the matrix P
Solution of the ARE is depicted as the red dashed line. As in Fig. I for the 2-dimensional system, these plots are depicted with respect to T t. The initialization is Pr I, the identity matrix.
F.2 COUPLED MASS SPRING DAMPER SYSTEM
This system is taken from Mohammadi et al. (219).
The matrices A and B are as follows:
Od, xd, T
Ia, -T
Oo,, Td,xxd xd, In,
A
B
where d, = is the number of masses and T E xdd. is a Toeplitz matrix with 2 on the main diagonal and -1 on the first sub-diagonal and first super-diagonal. Additional model and simulation parameters are listed in Table and respectively.
Table 4: Model parameters for the coupled mass spring damper system
Model parameter
Numerical value
Ma
0áx1
C' for 0.17a
En
V5Ia
for
Ia
R
Id.
Pr
Ia
Table 5: Simulation parameters for the coupled mass spring damper system
Simulation parameter name
Symbol
Numerical value
Simulation time
T
10
Step size
At
0.02
22

---

where o(4)) is the optimal control cost corresponding to the terminal gainas per EnKF, and by the (N) initial gain as per [M21] and [F18] algorithms; and cx is the optimal control cost corresponding the initial gain per EnKF, and terminal gain as per the [M21] and [F18] algorithms; and coo Under review as a conference paper at ICLR 2022
F.3 COMPARISON BETWEEN ENKF AND POLICY-GRADIEN METHODS
The performance of the EnKF algorithm is compared with policy gradient algorithms presented in Mohaamadi et al. 202 lc) (denoted as [M21]) and Fazel et al. (2018) (denoted as [F18]). These Two algorithms are based gradient free optimization for the gain matrix, with the difference that [M21] is for continuous time systems while [F18] is for discrete- time systems. We used the dis- cretized linear system 2(+2/ (I A At, to implement [F18]. The two algorithms involve similar set of hyper-parameters that are selected optimally for each experiment for fair com- parison.
The three algorithms, EnKF, [M21], and [F18] are implemented for the mass-spring system of Ap- pendix [.2 for 2, 4, 10. The comparison is made by studying the relationship between the computational time and the error in approximating the optimal gain matrix. The error is measured with the formula
[
Koo le. |F
for EnKF
K
error
|
K le
for [M21] and [F18]
|
where K(N) is the terminal gain output by EnKF, K(N) is the terminal gain output by the [M21] and (F18) algorithms, and Kx is the analytical gain obtained through the ARE.
In order to study the error vs computational time relationship for EnKF algorithm, the number of particles N is varied, while for [M21] and [F18] the number of iterations is varied. The simulation time horizon T = 10, and the step-size At = 0.01 is the same for all algorithms. The hyper- parameters of the [M21], and [F18] algorithms are presented in Table 6 The initial guess K 0, initial distribution D = N NO, Ia), and gradient descent step a 0.0001 for both [M21] and [F18]. The values of the other hyper parameters are in Table The numerical results for 10 are depicted in Figure and for d = 2, 4 in Figure 6 Additionally, Figure 7 shows comparison when the error measured as
(N)
Coo
for EnKF
error
for [M21] and [F18)
the optimal control cost achieved by analytical gain obtained through the ARE.
The simulations are implemented in Python 3 on Intel Xeon E3-1220 V2 3.40 Ghz CPU, and the process_te time imme
() function from the t time module is used to evaluate the execution time.
Table 6: Syper-parameters for policy gradient
Symbol
Initial guess
K
Distribution of initial state
Do
Smoothing parameter
Gradient descent step
Simulation horizon
T
Averaging for gradient
N
23

---

Under review as a conference paper at ICLR 2022
EnKF
M21
F188
10a
10
10a
Comp. Time (s)
10a
10
10
10a
d 2
4
109
0.05
0.10
0.02
0.04
0.06
0.08
Relative error gain
Figure 6: Comparison between EnKF, [M21], [F18)
EaKF
[M21]]
10
Comp. Time (8)
107
10p
00201
0U002
0.004
2,000
02000
0,200
0.000
E00%s
0L008
0003)
10302
0.294
11.08
Error it cost
Figure Comparison between EnKF, [M21], (F18)
F.4 CART-POLE SYSTEM
The nonlinear model is taken from (Tedrake Chapter 3.2.1):
W 1(M + mmmmss sin (())
(- F cos(O) mlw? cos(O) sin(O) (m + MMgssin())
=
(F + (ww) + coss()))
M msin?(()
Table 7: Hyper-parameter values for policy gradient
Hyper-param.
[M21]
[F18)
d
2
10
4
10
r
10-1
10-1
10-3
10-1
10-1
10-1
2
10
4
10
24

---

Under review as a conference paper at ICLR 2022
For the LQG control design, linearize the equations about the desired inverted equilibrium (T, 0,0,0.O.. 0, 0). The associated A and B matrices are as follows:
A
B
M7
(Md+
M1
M
The model parameters are listed in Table
and the simulation parameters are Table 9
Table Model parameters for the cart-pole system
Model parameter name
Symbol
Numerical value
Mass of ball
m
0.08
Mass of cart
M
1
Length of rod
1
0.7
Gravity
g
9.81
Unstable equilibrium
(0, 2. W. B)
(x, 0,0,0)
Initial condition
(() v0))
(1.25mr -0.1,
C
diag [10, 10, 1,
LQQ parameters
R
10
Pr
Z4
Table 9: Simulation parameters for the cart-pole system
Simulation parameter name
Symbol
Numerical value
Simulation time
T
10
Step size
At
0.0002
25


Reviews:
[{"review_id": "pU-a98RtTzw", "reviewer": "Reviewer_7yfG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The current version of the manuscript is not ready for publication. \n\n- The problem understudy is not well motivated: \nThe proposed method is not capable of providing a performance better than the classical approach of Riccati equations, and it makes sense computationally, only if the dimension is large. However, in that case the statistical properties enforce requirement of many trajectories, because of hight sample-complexity of covariance estimation, which defeats the purpose. For example, see (14).\n\n- Incomplete literature review: \nMost of the existing references focus on discrete-time settings (for which the literature review is still incomplete, just search adaptive linear systems in google scholar), and none of the modern works on reinforcement learning, estimation, and control in continuous-time linear systems are reviewed in the manuscript (again, google scholar helps). Further, on p7, the rich literatures of single trajectory RL and adaptive stabilization are skipped, and less relevant approaches are emphasized.\n\n- Multiple incorrect or inaccurate statements, including but not limited to:\n\"In LQR, the bottleneck ...\" (a big and inaccurate claim without sufficient reasoning), \nFootnote 1 and related material on p2 (generality and optimality are mixed up in a confusing way, and are inconsistent with (3) and prop 1), \nIn (3), the expected value and the covariance of the initial state are assumed known, but it is not mentioned or justified,\n\"In summary, ... any RL algorithm\": such strong statements need strong justifications,\nIn footnote 3, it is not clear how the oracle is used, since access to an oracle does not imply access to its minimizer,\nThe first para on p7.\n\n- Several unclear statements and/or unsupported claims:\nFirst para on p2, \n\"a forward (in time) DRE\" and \"a backward (in time) DRE\", \nThe control signal in (4),\nComputation of \\bar m_t and \\bar \\Sigma_t in (4),\n\"statistics of the process\",\n\"This term has a gain ...\",\nDiscussions after (9),\nComputation of (10),\nNotation in (11), (12),\nDiscussions in \"Arrow of simulation time\",\nThe quantity m on p6.\n\nMoreover: \n- There are frequent typos and writing errors, and the citation format is set incorrectly.\n-The terms 'particle' and 'mean-field' are commonly used and have rich literature. So, borrowing them needs connection to the existing approaches and semantics.\n-Adaptation of control signal to the filtration Z excludes a class of policies and is not required (re-stating based on information etc might help). The same applies to assuming that control signal cannot depend on previous values.\n- It is unclear why in the context of RL, Markovianity of Step 2 is a good assumption.\n- \"Backward Ito-integral\" is undefined.\n- RL terminology is misused and connections to RL look artificial (see item 3 on p6).\n- An incorrect benchmark is adopted on p6 in Relationship to RL: the reasonable benchmark is to solve DRE/ARE.\n- In the motivating application from weather prediction, what is control input?", "summary_of_the_paper": "The paper proposes a method of calculation for filtering and control in continuous-time linear systems based on using a simulator to generate many trajectories and use their statistical properties to empirically calculate the needed estimation and actuation signals.", "summary_of_the_review": "The current version of the manuscript is not ready for publication. \n- The problem understudy is not well motivated.\n- Incomplete literature review.\n- Multiple incorrect or inaccurate statements.\n- Several unclear statements and/or unsupported claims.\n- and, further items discussed above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "oEmh-nfMfy", "reviewer": "Reviewer_HWCL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper looks at the control as inference duality and proposes the EnKF as a replacement for a Riccati-based solver. Sample based inference, such as the eKF or sequential Monte Carlo, offer greater accuracy especially for complex nonlinear systems.\nThe connection between data assimilation methods and RL is an interesting one and the paper was well written and demonstrated a lot of mathematical rigour. \n\nUnfortunately, this paper has several weaknesses.\n\n1) Scholarship. \nThe duality between Kalman smoothing, LQG and Riccatti equations is well established. This paper barely touched on this literature, and when it did it was only in the appendix. \n\nProbabilistic Inference for Solving Discrete and Continuous State Markov Decision Processes. Toussaint et al. ICML 2006\n\nRobot Trajectory Optimization using Approximate Inference. ICML 2009\n\nOn Stochastic Optimal Control and Reinforcement Learning by Approximate Inference. Rawlik et al R:SS 2013\n\nSampled DIfferential Dynamic Programming, Rajamäki et al. IROS 2016\n\nLinear Optimal Control on Factor Graphs - A Message Passing Perspective, Hoffman et al. 2017 IFAC.\n\nRegularizing Sampled Differential Dynamic Programming Rajamäki et al. ACC 2018\n\nStochastic Optimal Control as Approximate Input Inference, Watson et al. Corl 2019\n\nAdvancing Trajectory Optimization with Approximate Inference: Exploration, Covariance Control and Adaptive Risk, Watson et al. ACC 2021\n\nStochastic Control through Approximate Bayesian Input Inference Watson et al. Arxiv 2021\n\nWhile the Rawlik paper is cited, it is misrepresented by saying it is solving the linearly solveable MDP problem like a path integral method, when the posterior policy iteration algorithm actually uses approximate Gaussian message passing (linearization in the paper). Moreover the R:SS paper is not properly listed in the references.\nThe papers cited above use the duality between Riccati equations and Gaussian smoothing using a range of approximate inference methods, including linearization, quadrature and Monte Carlo.\nNot only is it strange for this large body of work to not be referenced in the paper (when path integral methods were) but it weakes the contribution of applying the eKF to LQG, especially when the eKF isn't well motivated for this setting. Section 2.2 it listed as the main contribution of the paper, but does not seem particularly novel given prior unreferenced work. While the continuous time derivation does seem novel, I did not understand the motivation for this direction since RL is primarily discrete time. \n\n2) Motivation for the eKF. For my understanding, the EnKF is motivated for high-dimensionality linear systems where the number of samples is lower than the dimensionality. While this setting is covered in 3.2 for a synthetic toy task, it was not clear to me why this setting was interesting for actual control problems and did not seem especially relevant to the ICLR community. Perhaps there are more specific problem domains where this is the case.\nA nonlinear extension is teased in the appendix, and in my opinion it is this setting that would be of interest to the ICLR community.\n\n3) Does not seem to perform RL. While RL is mentioned in the title and abstract, it seems the eKF is actually applied in an optimal control setting rather than RL. While it may only use a model implicity and not require linearizing the dynamics, the same could be achieved using quadrature Gaussian message passing (e.g unscented, cubature, Gauss-Hermite). The paper needs to be heavily revised because of this. \n\n4)  Limited experiments. Given the linear setting, the experiments a small scale for a venue like ICLR. In particularly for 3.2, it was not clear why the baselines chosen were used, especially when LQG has a closed-form solution and 80 dimensions are not too large for modern mathematical libraries.\n\n\nMinor comments:\n5) The paper title does not need to include abbreviations\n", "summary_of_the_paper": "The paper proposes how the ensemble Kalman fillter (EnKF) can be used on the LQG problem for both state estimation and control. The algorithm is evaluated on a linear system and a linearized nonlinear system.", "summary_of_the_review": "The paper does not reference a large body of existing work on the Kalman fitler / LQG duality which the paper claims is their contribtion. The EnKF is also not well motivated for this setting and the experiments are very limited and seemingly not reinforcement learning tasks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "N-WQ070ANO", "reviewer": "Reviewer_faGe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n  The motivation for the proposed algorithm is that when the simulation-based models are available and the control problem's state d is high- \n  dimensional, the current differential Riccati equation (DRE) or algebraic Riccati equation (ARE) based algorithms is O(d^2). The proposed \n  ensemble Kalman filter based algorithm computation complexity is O(Nd), where N is the number of particles used for the estimation of the \n  optimal control policy. The proposed algorithm would be more computationally efficient for the above case where N<<d.\n\nWeaknesses:\n  1. Contribution of the extension of ensemble Kalman filter idea to control policy estimation is questionable. Based on the paper's result, it is not that difficult to construct a mean field process in Eq.(9) from backward DRE in Eq.(7) and (8) comparing with the mean field process from existing Kalman filter design. Also, the exactness contribution in Proposition 2 is a straightforward extension of the Proposition 1.\n  2. The claim on the reduced computation complexity from O(d^2) to O(Nd) is debatable. Although N is the number of particles, and we could use N<<d. But will that make the estimated control policy worse than the algorithms with O(d^2)? Should the N depend on the problem size as well? Based on the error bound in Eq.(14), the first term in RHS, ||\\bar{S}_T||_F may also depend on the problem size d as well, which makes the error in Eq.(14) depend on d and thus the particle number N.\n  3. The evaluation metric used in Section 3.2 may not be desirable. The MSE w.r.t. the true matrix is not what the Fazel et al. (2018) tries to minimize. The ideal metric might be using the same loss function to evaluate the performance.\n", "summary_of_the_paper": "The paper is about designing a simulation-based ensemble Kalman filter algorithm for learning the optimal control policy for the Linear Quadratic Gaussian (LQG) control problem.\nThe main contribution is that the paper extends the existing ensemble Kalman filter algorithm for filter (state estimation) design to the optimal control policy learning for the LQG control setting. And it is shown to be exact in its mean-field limit (N=infinity). ", "summary_of_the_review": "Based on my above comments, I think the paper is interesting but not good enough to be accepted at this time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents the use of the Ensemble Kalman Filter (EnKF) to solve the linear quadratic Gaussian (LQG) optimal control problem. After reviewing the paper and taking into consideration of the reviewing process, here are my comments:
- The related work is limited and needs more improvements to contextualize the problem and the solution.
- The reinforcement learning paradigm is not really appreciated in the proposal.
- The results are rather limited, so more experiments are needed to clearly validate the solution.
From the above, the paper does not fulfill the standards of the ICLR. I suggest improving the paper accordingly and submitting it to a control systems venue.
Decision: Reject