Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method of calculation for filtering and control in continuous-time linear systems based on using a simulator to generate many trajectories and use their statistical properties to empirically calculate the needed estimation and actuation signals.
main_review: The current version of the manuscript is not ready for publication. 

- The problem understudy is not well motivated: 
The proposed method is not capable of providing a performance better than the classical approach of Riccati equations, and it makes sense computationally, only if the dimension is large. However, in that case the statistical properties enforce requirement of many trajectories, because of hight sample-complexity of covariance estimation, which defeats the purpose. For example, see (14).

- Incomplete literature review: 
Most of the existing references focus on discrete-time settings (for which the literature review is still incomplete, just search adaptive linear systems in google scholar), and none of the modern works on reinforcement learning, estimation, and control in continuous-time linear systems are reviewed in the manuscript (again, google scholar helps). Further, on p7, the rich literatures of single trajectory RL and adaptive stabilization are skipped, and less relevant approaches are emphasized.

- Multiple incorrect or inaccurate statements, including but not limited to:
"In LQR, the bottleneck ..." (a big and inaccurate claim without sufficient reasoning), 
Footnote 1 and related material on p2 (generality and optimality are mixed up in a confusing way, and are inconsistent with (3) and prop 1), 
In (3), the expected value and the covariance of the initial state are assumed known, but it is not mentioned or justified,
"In summary, ... any RL algorithm": such strong statements need strong justifications,
In footnote 3, it is not clear how the oracle is used, since access to an oracle does not imply access to its minimizer,
The first para on p7.

- Several unclear statements and/or unsupported claims:
First para on p2, 
"a forward (in time) DRE" and "a backward (in time) DRE", 
The control signal in (4),
Computation of \bar m_t and \bar \Sigma_t in (4),
"statistics of the process",
"This term has a gain ...",
Discussions after (9),
Computation of (10),
Notation in (11), (12),
Discussions in "Arrow of simulation time",
The quantity m on p6.

Moreover: 
- There are frequent typos and writing errors, and the citation format is set incorrectly.
-The terms 'particle' and 'mean-field' are commonly used and have rich literature. So, borrowing them needs connection to the existing approaches and semantics.
-Adaptation of control signal to the filtration Z excludes a class of policies and is not required (re-stating based on information etc might help). The same applies to assuming that control signal cannot depend on previous values.
- It is unclear why in the context of RL, Markovianity of Step 2 is a good assumption.
- "Backward Ito-integral" is undefined.
- RL terminology is misused and connections to RL look artificial (see item 3 on p6).
- An incorrect benchmark is adopted on p6 in Relationship to RL: the reasonable benchmark is to solve DRE/ARE.
- In the motivating application from weather prediction, what is control input?
summary_of_the_review: The current version of the manuscript is not ready for publication. 
- The problem understudy is not well motivated.
- Incomplete literature review.
- Multiple incorrect or inaccurate statements.
- Several unclear statements and/or unsupported claims.
- and, further items discussed above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes how the ensemble Kalman fillter (EnKF) can be used on the LQG problem for both state estimation and control. The algorithm is evaluated on a linear system and a linearized nonlinear system.
main_review: The paper looks at the control as inference duality and proposes the EnKF as a replacement for a Riccati-based solver. Sample based inference, such as the eKF or sequential Monte Carlo, offer greater accuracy especially for complex nonlinear systems.
The connection between data assimilation methods and RL is an interesting one and the paper was well written and demonstrated a lot of mathematical rigour. 

Unfortunately, this paper has several weaknesses.

1) Scholarship. 
The duality between Kalman smoothing, LQG and Riccatti equations is well established. This paper barely touched on this literature, and when it did it was only in the appendix. 

Probabilistic Inference for Solving Discrete and Continuous State Markov Decision Processes. Toussaint et al. ICML 2006

Robot Trajectory Optimization using Approximate Inference. ICML 2009

On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference. Rawlik et al R:SS 2013

Sampled DIfferential Dynamic Programming, Rajamäki et al. IROS 2016

Linear Optimal Control on Factor Graphs - A Message Passing Perspective, Hoffman et al. 2017 IFAC.

Regularizing Sampled Differential Dynamic Programming Rajamäki et al. ACC 2018

Stochastic Optimal Control as Approximate Input Inference, Watson et al. Corl 2019

Advancing Trajectory Optimization with Approximate Inference: Exploration, Covariance Control and Adaptive Risk, Watson et al. ACC 2021

Stochastic Control through Approximate Bayesian Input Inference Watson et al. Arxiv 2021

While the Rawlik paper is cited, it is misrepresented by saying it is solving the linearly solveable MDP problem like a path integral method, when the posterior policy iteration algorithm actually uses approximate Gaussian message passing (linearization in the paper). Moreover the R:SS paper is not properly listed in the references.
The papers cited above use the duality between Riccati equations and Gaussian smoothing using a range of approximate inference methods, including linearization, quadrature and Monte Carlo.
Not only is it strange for this large body of work to not be referenced in the paper (when path integral methods were) but it weakes the contribution of applying the eKF to LQG, especially when the eKF isn't well motivated for this setting. Section 2.2 it listed as the main contribution of the paper, but does not seem particularly novel given prior unreferenced work. While the continuous time derivation does seem novel, I did not understand the motivation for this direction since RL is primarily discrete time. 

2) Motivation for the eKF. For my understanding, the EnKF is motivated for high-dimensionality linear systems where the number of samples is lower than the dimensionality. While this setting is covered in 3.2 for a synthetic toy task, it was not clear to me why this setting was interesting for actual control problems and did not seem especially relevant to the ICLR community. Perhaps there are more specific problem domains where this is the case.
A nonlinear extension is teased in the appendix, and in my opinion it is this setting that would be of interest to the ICLR community.

3) Does not seem to perform RL. While RL is mentioned in the title and abstract, it seems the eKF is actually applied in an optimal control setting rather than RL. While it may only use a model implicity and not require linearizing the dynamics, the same could be achieved using quadrature Gaussian message passing (e.g unscented, cubature, Gauss-Hermite). The paper needs to be heavily revised because of this. 

4)  Limited experiments. Given the linear setting, the experiments a small scale for a venue like ICLR. In particularly for 3.2, it was not clear why the baselines chosen were used, especially when LQG has a closed-form solution and 80 dimensions are not too large for modern mathematical libraries.


Minor comments:
5) The paper title does not need to include abbreviations

summary_of_the_review: The paper does not reference a large body of existing work on the Kalman fitler / LQG duality which the paper claims is their contribtion. The EnKF is also not well motivated for this setting and the experiments are very limited and seemingly not reinforcement learning tasks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper is about designing a simulation-based ensemble Kalman filter algorithm for learning the optimal control policy for the Linear Quadratic Gaussian (LQG) control problem.
The main contribution is that the paper extends the existing ensemble Kalman filter algorithm for filter (state estimation) design to the optimal control policy learning for the LQG control setting. And it is shown to be exact in its mean-field limit (N=infinity). 
main_review: Strengths:
  The motivation for the proposed algorithm is that when the simulation-based models are available and the control problem's state d is high- 
  dimensional, the current differential Riccati equation (DRE) or algebraic Riccati equation (ARE) based algorithms is O(d^2). The proposed 
  ensemble Kalman filter based algorithm computation complexity is O(Nd), where N is the number of particles used for the estimation of the 
  optimal control policy. The proposed algorithm would be more computationally efficient for the above case where N<<d.

Weaknesses:
  1. Contribution of the extension of ensemble Kalman filter idea to control policy estimation is questionable. Based on the paper's result, it is not that difficult to construct a mean field process in Eq.(9) from backward DRE in Eq.(7) and (8) comparing with the mean field process from existing Kalman filter design. Also, the exactness contribution in Proposition 2 is a straightforward extension of the Proposition 1.
  2. The claim on the reduced computation complexity from O(d^2) to O(Nd) is debatable. Although N is the number of particles, and we could use N<<d. But will that make the estimated control policy worse than the algorithms with O(d^2)? Should the N depend on the problem size as well? Based on the error bound in Eq.(14), the first term in RHS, ||\bar{S}_T||_F may also depend on the problem size d as well, which makes the error in Eq.(14) depend on d and thus the particle number N.
  3. The evaluation metric used in Section 3.2 may not be desirable. The MSE w.r.t. the true matrix is not what the Fazel et al. (2018) tries to minimize. The ideal metric might be using the same loss function to evaluate the performance.

summary_of_the_review: Based on my above comments, I think the paper is interesting but not good enough to be accepted at this time.
review_title: null

Meta-review: This paper presents the use of the Ensemble Kalman Filter (EnKF) to solve the linear quadratic Gaussian (LQG) optimal control problem. After reviewing the paper and taking into consideration of the reviewing process, here are my comments:
- The related work is limited and needs more improvements to contextualize the problem and the solution.
- The reinforcement learning paradigm is not really appreciated in the proposal.
- The results are rather limited, so more experiments are needed to clearly validate the solution.
From the above, the paper does not fulfill the standards of the ICLR. I suggest improving the paper accordingly and submitting it to a control systems venue.
Decision: Reject