Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The objective of the paper is to construct an estimator for the state of a high-dimensional nonlinear dynamical system given partial observations of the state. The objective is motivated by applications in fluid mechanics or turbulent flows where the state of the system is large obtained by discretizing a PDE. The proposed approach has two main steps:

(1) construction of a reduced order model. In particular, the paper proposes the dynamic mode decomposition method which only requires a single trajectory of the nonlinear dynamics.

(2) formulating the problem of finding the estimator as a MDP problem and application of RL techniques to solve it. In particular, the estimator is modeled as a dynamical system driven by a stochastic control policy that depends on the current value of the estimate and the value of observation. The objective function is modeled with running cost equal to the error in estimating the state and a quadratic penalty on the control. Then, the optimal control policy is learned using a policy gradient method by sampling trajectories from the system.

The proposed approach is evaluated on a benchmark example that involves Burger's equation and compared with Kalman filter applied on the reduced order system.
strength_and_weaknesses: Strength
- Interesting and important problem 
- The general solution approach, using RL for estimation, is definitely exciting

Weakness: 
- No discussion on the meaningfulness of the designed estimator. What does the cost in the optimal control represent? Why does minimizing it lead to a good estimator? 
- The comparison with KF-ROE can me made more in depth. Are the control laws learned through the RL approach nonlinear? What is the tuning process for KF? 
- the inability to consider estimating systems with inputs. 
- the discussion about stationary Markov decision problem and the remark about the problem being actually POMDP and non-Markov is a bit confusing. 


clarity,_quality,_novelty_and_reproducibility: The paper is clear in general. RL formulation of estimator is not novel, but its combination with ROM is novel. 
summary_of_the_review: The paper discusses a very interesting problem and follows a general exciting approach. It should give more motivation for the cost function, and also more in depth comparison to KF, like comparing the learned policy to the best linear policy. Basically, it is interesting to see what would happened if the control action is restricted to be linear and what is the gain from nonlinear parametrization.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to infer the state of a dynamical system based on sparse measurements. More specifically, the authors consider a situation in which there is a dynamical system whose state evolves but we cannot observe it directly. Instead we observe only a low dimensional approximation.. Here, the authors propose a method which combines a reduced order model (ROM) approach with reinforcement learning (RL). Then they provide examples of applications to partial differential equations (PDEs), with numerical results.
strength_and_weaknesses: Strength: the paper is clearly written and the ideas are well presented.

Weaknesses: (1) it seems that the framework and the model purely deal with a discrete time and finite dimensional system. The connection with PDEs (mentioned in the title) is not clear until section 4, and even there, it seems to me more like an application. This is not necessarily a drawback, since it probably shows that the method is quite general. However, I am wondering whether the connection with PDEs should be strengthened. (2) Furthermore, the main contributions are numerical but the only baseline comes from a Kalman filter approach, which is suitable for linear systems but I am not sure if it is suitable for PDEs. I would recommend discussing this choice of baseline and, if possible, to provide other baselines.

clarity,_quality,_novelty_and_reproducibility: As mentioned above, the paper is clearly written. I have two main comments:

(1) To what extent is the proposed method specific to PDEs? It seems that once the PDE is discretized (which is done separately from the proposed method), it is a quite generic method that is developed and applied. In that case, it might be interesting to adjust accordingly the presentation of the paper.

(2) Are there other (better) baselines than a simple Kalman filter? Or could you please provide more details on why this is a good baseline?

summary_of_the_review: The paper is clearly written and the contributions are clearly presented. I think that addressing the two questions mentioned above would help strengthen the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers state space estimation by optimal filtering for latent partial differential equations (PDE). For this, a reinforcement-learning-based estimator is proposed. 

The paper considers a given space-time discretization method of the PDE in question, resulting in a high-dimensional system of time-discretized nonlinear ordinary differential equations (ODEs). The resulting high-dimensional system is reduced in dimensionality using a linear dimensionality reduction method, dynamic mode decomposition (DMD). The state estimation problem is therefore transferred to the low-dimensional linear space. 
For solving the resulting optimal filtering problem an estimator for the posterior mean of the Bayesian filtering problem is proposed. The authors consider for this an estimator which includes the low-dimensional linear dynamics and an additive nonlinear correction term (innovation process) based on the measurements.
The optimal filter is learned using a reinforcement learning (RL) algorithm, proximal policy optimization (PPO). The target of the RL algorithm is a reward parametrized by the negative squared estimation error and a penalty for the innovation gain.
The final algorithm is evaluated for two synthetic PDE filtering problems.
strength_and_weaknesses: The paper discusses in my opinion a very timely topic, state estimation for PDEs. The setup as such is interesting and has probably many application scenarios. The use of a dimensionality reduction method is a clever way to make the state estimation problem tractable. My biggest concern is, however,  that the algorithmic development is very heuristic. The paper misses the principled and "modern viewpoint" of Bayesian inference to model the state estimation problem, see, e.g., [1] for an introduction, and [2,3] for mathematical treatments. 
Hence, the proposed mean estimator in equation (3) is arbitrarily chosen, which resembles the usual structure of the Kushner-Startonovich equation, including the prior dynamics and the innovation process. 
For me, it makes sense that the reinforcement learning-based estimator outperforms the linear Kalman filter, as the estimator is based on a Gaussian error assumption. However, what is missing in such a case is a comparison, to, e.g., a nonlinear particle filter, which probably will easily outperform the proposed RL algorithm.
Another problematic aspect is that the controller only has access to the new observation and the posterior mean estimate. Even when assuming the linear prior dynamics, the innovation process is still dependent on the whole posterior distribution at the previous time step. This is justified in the remark saying that the history $h_k$ represented by $o_k$ is an incomplete summary and therefore sufficient. I would highly object that this is enough for state estimation since even in the linear quadratic case the optimal estimator is dependent on the variance of the posterior distribution (solution to the Ricatti equation).
Otherwise, I think the experiments are fair and convincing for the considered setup. 

As an aside, which is probably out of scope for this work, I would argue that a problematic aspect of the setup is that a time discretization is carried out before the inference. This results in an easier-to-model discrete-time system, however, a continuous-time description is usually much more sensible in these scenarios, see, e.g., [4] for an accessible introduction and [2] for a mathematical treatment. As oftentimes the time-discretization as chosen by the Runge-Kutta integrator is selected arbitrarily to model the prior dynamics, however, the posterior dynamics often require a different time-discretization. Maybe for future work, the authors can consider this setup.

Summary:

Strong points:
- The submission is timely and the setup is very interesting.
- The submission is fairly well-written and technically correct.


Weak points:
- algorithmic development is heuristic
- Bayesian description is missing
- estimator is chosen arbitrarily and not optimal
- a simple comparison to a particle filter algorithm is missing


[1] Särkkä, Simo. Bayesian filtering and smoothing. No. 3. Cambridge university press, 2013.
[2] Elliott, Robert J., Lakhdar Aggoun, and John B. Moore. Hidden Markov models: estimation and control. Vol. 29. Springer Science & Business Media, 2008.
[3] Bain, Alan, and Dan Crisan. Fundamentals of stochastic filtering. Vol. 3. New York: Springer, 2009.
[4] Särkkä, Simo, and Arno Solin. Applied stochastic differential equations. Vol. 10. Cambridge University Press, 2019.
clarity,_quality,_novelty_and_reproducibility: - In my opinion, the submission is well-written and easy to follow. 
- The originality of the work is somehow limited as the principle of learning estimators using reinforcement learning is well known. 
- The experiments are fair and interesting instantiations of the problem. 
- The presented algorithm is easy to implement and, therefore, probably easily reproducible.

summary_of_the_review: The paper is well-written and the setup is interesting. The principle behind the estimation algorithm presented is known. The biggest downside of this paper is, that the algorithm is developed heuristically and consequently a comparison to other nonlinear state space estimators is missing. The experiments considered are interesting and the evaluation of the considered setup is fair.

Altogether I come to the conclusion that this paper is between the ratings of 3 (reject, not good enough) and 5 (marginally below the acceptance threshold).

Some questions I had for the authors:
- What is the motivation behind the considered estimator in equation (3), why is not an estimator considered, which is fully parameterized by a neural network, i.e., $\hat{x}_k=a_k$  
- Is this modeling of one linear dynamical system efficient, did you consider using a switching dynamical system as a model, e.g., [5]?
- Could this setup be extended to nonlinear dimensionality reduction methods, if anyway later a nonlinear estimator is used?
- The estimator in equation (3) assumes a zero mean noise process $w_k$. How is the empirical distribution for $w_k$ looking, is it zero mean?
- In equation (7), the hyper-parameter $\lambda_2$ should be dependent on the observation noise, i.e., if the observation noise is high $\lambda_2$ should be chosen large, as this would diminish the gain on the innovation term ($a_k$) in equation (3). How was the observation noise chosen in the experiments? Can you see an influence of the two hyper-parameters, noise variance, and $\lambda_2$?

[5] Linderman, Scott, et al. "Bayesian learning and inference in recurrent switching linear dynamical systems." Artificial Intelligence and Statistics. PMLR, 2017.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: this paper proposes to estimate ROM from samples by formulating the correction term to be a non-linear policy, which is trained by RL. The problem formulation requires a stationary MDP model, which is constructed by augmenting the state with the original $z_k$ and $\mu$.  
strength_and_weaknesses: Strengths: interesting idea, justified approach, good simulations, well written paper.

Weaknesses/Limitations are reasonably addressed by the authors. 

a few comments:
1. Eq7. lhs depend on t+1, while rhs doesn't not depend on $t+1$

2. in NAVIER-STOKES EQUATION: the dimensionality of state (the discretized solution) is huge. could you elaborate on the sensors in this case? does the sensor matrix, C, include only 1 non zero entries?  I suggest to explicitly state the dimensions of the sensor in the experiments. 

3. could you elaborate on "However, this approach would not work for our purpose since the RL-ROE cannot be restricted to estimating a single or a small finite number of reference trajectories. Instead, by augmenting the state with entire snapshots from the reference trajectory instead of just a phase variable, we ensure that there is no theoretical limit to the number of reference trajectories that the policy πθ can learn"? I don't understand why including the snapshot to the state allows the policy to learn any number of trajectories. The state is augmented with $\mu$, which I thought, is enough for the generalization to unseen trajectories. 
clarity,_quality,_novelty_and_reproducibility: clear, high quality, pending my evaluation of originality on the elaboration of the last paragraph in "Related Work".
summary_of_the_review: well-written paper. the background and limitations are well explained. numerical simulations confirm the claims.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Given a partial state observation in a space-time discretization of the PDE, the paper estimates that PDE.

The paper is interesting and is technically sound.  At the same time it is not very strong, having seemingly ad-hoc derivations.  A theoretical framework based on Bayesian inference would perhaps be more solid.  It was added ex post but more as an add-on, seemingly half-random, than as the theoretical basis of the paper.

All in all, the paper is not strong and convincing enough to merit acceptance at ICLR, but can be rewritten with reasonable effort and submitted to a future venue.
justification_for_why_not_higher_score: See meta-review.
justification_for_why_not_lower_score: n/a
Decision: Reject