Reinforcement Learning State Estimation for High-Dimensional Nonlinear Systems

Abstract: In high-dimensional nonlinear systems such as fluid flows, the design of state estimators such as Kalman filters relies on a reduced-order model (ROM) of the dynamics. However, ROMs are prone to large errors, which negatively affects the performance of the estimator. Here, we introduce the reinforcement learning reduced-order estimator (RL-ROE), a ROM-based estimator in which the data assimilation feedback term is given by a nonlinear stochastic policy trained through reinforcement learning. The flexibility of the nonlinear policy enables the RL-ROE to compensate for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. We show that the trained RL-ROE is able to outperform a Kalman filter designed using the same ROM, and displays robust estimation performance with respect to different reference trajectories and initial state estimates.

Reviews and author responses:

 Dear reviewer,

Thank you for your patience with us. We discussed again among ourselves your concern, and we think that the misunderstanding stems from the following:

1- We perhaps did not emphasize enough that in solving for an optimal policy, we are in fact solving for the whole trajectory from k=0, to k=K (equation 9, and solution of 3a-3b from k=0 to k=K), so during training it is clear that we are not considering only one measurement but the vector of measurements form k=0 to K. Thus, we do have a stationary convergent RL problem, as seen in our numerous numerical tests.

2- And this is the more crucial point; when deploying the policy, we need to underline again that this type of observers (what we call Kalman-type or IIR filters) do not consist of the optimal policy uniquely but consist of the summation of the optimal policy and  the dynamic term A_r \hat_x_(k-1) (equation 3a), and to estimate the state \hat_x at the instant K, one HAS to propagate the estimation starting from the initial guess \hat_x_0, all the way to \hat_x_K. To do so, one has to propagate the solution by making one-step ahead predictions from \hat_x_0 predicting \hat_x_1 by solving equation 3a for one step, which requires the belief that is simply based on the measurement at y_1 (since it is only one step the belief is trivially one measurement), then we have to continue propagating this estimate forward, with \hat_x2 being estimated from \hat_x1 and equation 3a, using the next one-step ahead measurement y_2, etc. This process finally gives us the desired estimate \hat_x_K, which is implicitly (through the one-step ahead propagation of multiple one-step beliefs) function of the vector of measurements from k=0, to k=K. This is to be contrasted with the other type of observers, i.e., the FIR filters, which gives the estimate \hat_x_K using the vector of measurements or beliefs from k=0, to k=K, in one-shot, i.e., not multiple one-step ahead predictions. 

In this sense, we are using the full 'belief' , i.e., all the measurements from k=0 to k=K in the training, since we are solving for the RL problem for the full trajectory (equations 7, 9, 3a-3b), and in the deployment, to predict the state \hat_x_K, implicitly via multiple one-step ahead predictions (propagation of the solution of equations 3a-3b) from k=0 to k=K.  

We will add more explanation in the paper to clarify this difference between Kalman-type dynamical observers and one-shot FIR observers.

Does this help to clarify your concern ?

Thank you again.


 Thanks for the response. I am sorry if my question was not clear. As I said earlier, I understand why the POMDP is stationary and I never disputed that. My point is about the policy. The optimal policy for stationary POMDP is stationary if it is parametrized as a function of "belief". Your parametrization as a function of last measurement does not lead to stationary policy even though you are forcing it to be. In principle, running RL with your parametrization won't converge, since the optimal policy parametrized with last measurement changes with time. This parametrization theoretically works only in linear-Gaussian setting and probably it is practically good for systems with weak nonlinearity such as the one in Ref [2] you mentioned earlier.    



 Thank you again for your insights and engagement. 

We do believe that we have introduced the confusion about non-stationarity of the MDP in our first paragraph in Section 4. We have revised it at this stage to clear any confusion.

 Indeed, the reward function in equation 7, can be seen as time-dependent if we consider an MDP where the state is \hat{x}_k only, because in such case z_k is not part of the state 's MDP and is considered to be a time-varying reference. So we propose to lift the state to a larger state vector s_k which contains both x_k and z_k; now in this new state-space the MPD state transition is defined by equation 6 and is time-independent (stationary). Furthermore, the reward function is now a classical time-independent (stationary) reward function, based on the state s_k, and action a_k only. In these conditions we have a classical stationary MDP in the s_k state space. Since we only have access to partial measurements y_k of s_k, given by y_k=C z_k, then we are dealing with a POMDP in the s_k state-space and y_k measurement space, where y_k=C z_k=(C|O)s_k, where O denotes the zero vector of dimension equal to dimension of \hat_x_k. Thus, this POMDP is also time-independent or stationary.  

After solving this stationary POMDP we obtain a stationary policy given by equation 5, which is only function of the current estimate of the state \hat{x}_{k-1} (which is part of the augmented state vector s_k ), and a partial measurements of s_k, given by  y_k= (C|O)s_k= C z_k= C U x_k (as defined in equation 2b). So eventually, the observer that we obtain is in the form of equations 3a,3b, which is only function of the current estimate and the current partial measurements C U x_k. Thus it is time-independent and has the form of a classical stationary Kalman-type observer (please see our response above). 

We hope that this explanations clear most of the confusion (that we are responsible of, apologies) about the stationarity of the final policy and the associated Kalman-type observer. We have revised the final version of the paper, based on this discussion.

Please let us know if you find these clarifications satisfactory or if you have further questions.

Thank you again for your work and insight which helped us improve the clarity and quality of the presentation.

 I appreciate your response and clarifications. Trying to engage and clarify my miss understandings. 

Can you also  respond to the non-stationary control policy? If the optimal control policy is not stationary, there is no guarantee to learn something meaningful with RL because the optimal policy changes with time.    

 Dear reviewer, 

Firstly, we want to thank you for engaging in a review discussion.

Secondly, I think we understand the confusion. We should have underlined in the Introduction of the paper (we did amend the paper now), that we are seeking a Kalman-type observer, which in 'filters domain' is also known as  an infinite impulse response filter (IIR) (observer). These observers are to be contrasted with the ones that this reviewer is referring to, i.e., finite impulse filters (FIR) (observers). Indeed, the later are well known to be based on a mapping between n previous samples of input/output and the desired observed state at instant n+1 (e.g., see reference [1], equations 2,3, and 12). On the other hand the IIR observer is well known to be based on the last measurement of the output/input only, e.g., (see reference [1], equation 8). Indeed, we agree that the desired performance here, is dictated by our choice of the cost function, i.e., a maximum likelihood estimate in this case. We have added the Remark 3 (in Appendix) clarifying the difference between IIR and FIR observers, as well as, the expected maximum likelihood estimate performance related to our cost function choice.


Reference:
1- CHAUPA et al., Model Predictive Control Using Different State Observers, In Recent Advances in Automatic Control, Information and Communications, 2013 (Link: http://www.wseas.us/e-library/conferences/2013/Valencia/ACIC/ACIC-36.pdf)

 I thank the authors for taking the time to respond to my comments. 

Regarding non-stationary control policy, I am not saying the POMDP problem is not stationary. My point is that the optimal control policy for POMDP, if the control is parametrized as only a function of last observation and current estimate, is not stationary. The optimal control is stationary as a function of the posterior or belief, which is not considered in the paper. Please let me know if I am mistaken. 

And this is related to my first point. To build a nonlinear observer, it is not enough to change the dynamics of the estimate to a nonlinear update law, since this would not be enough to capture the observation history. For example, an alternative approach is to train a RNN with large size so that it can capture the history of observation. [1]  

I would have appreciated if the ideal estimator that the paper aims to find is clearly defined in a probabilistic framework. This would provide a guideline and would motivate the proposed optimal control problem. The form of objective function seems to suggest a maximum likelihood estimate, but there is no discussion about that. I am not asking for error bounds, but some sort of consistency analysis, to know what expect to get if all computations and approximations are carried out exactly.   

[1] Chang Gao, Junkun Yan, Shenghua Zhou, Bo Chen, and Hongwei Liu. Long short-term memorybased recurrent neural networks for nonlinear target tracking. Signal Processing, 164, 05 2019.
doi: 10.1016/j.sigpro.2019.05.027.

 Dear Reviewers,
We appreciate your work and time put into the review of our paper. We also appreciate that you are busy (we are all in the same boat of being authors and reviewers, etc.). We have spent a lot of time in reading and discussing your concerns among ourselves, and then much time in revising the paper and drafting what we believe are thorough responses to each reviewer. 
We would really appreciate if you could read our replies and let us know if your main concerns have been addressed, or if you have anymore questions for us. Alternatively, if you found our responses to be satisfactory, we would appreciate if you could revise your scores accordingly.

Thank you again for your hard work.
Best wishes,
Authors

 We thank all reviewers for taking the time to review our paper and for their interest in our work. Their suggestions led to several improvements in the revised manuscript, which we have posted online with the modifications indicated in blue. To summarize,

1. We clarify the state transition model defining our stationary MDP in equation (6)

2. In a new Appendix F, we study the effect of penalizing the action magnitude in the reward function. In particular, we show that such penalization leads to better estimation performance under the presence of additive observation noise.

3. We add a remark clarifying that knowledge of the high-dimensional state from several reference trajectories is only required in the offline training phase. During online deployment, the RL-ROE only relies on measurements of the state. This paradigm is no different from model-based observer design which uses the information of all the trajectories of the system, in the form of a dynamical model that can be integrated offline forward in time from any selected initial condition.

4. We add a remark regarding the generalization abilities of the RL-ROE, emphasizing the importance of the distribution of the initial reference state and initial estimate during the offline RL training.

5. We implement two alternative data-driven estimator designs which can be trained in a supervised setting, proposed by two reviewers. The performance of the RL-ROE is compared with these alternative methods in a new Appendix G, showing that the RL-ROE is more robust to observation noise and gives smoother predictions.

We are happy to provide more information if the reviewers have additional questions or concerns.

 We warmly thank the reviewer for their very positive comments and for many excellent suggestions that lead to substantial improvements in the revised version of the paper. We provide below detailed answers to the reviewer's questions.

> I'm interested in the interplay between the Q and R terms in Eq. 6. How important is it to have the R to penalize the magnitude of actions? When does it help and hurt? My guess is R might perform some kind of regularization that allows the model to perform better in extrapolation. Can you add a comment describing what you've found.

This is an excellent point raised by the reviewer; we also think that the $R$ term might act as a regularization that allows the estimator to perform better when faced with trajectories it has not previously seen or in the presence of noise. We tried training our estimator for different magnitudes of $R$ and different amounts of additive observation noise. Without noise (similar to the examples shown in the paper), the estimation accuracy decreases as $R$ increases. In the presence of noise, the estimator is indeed more accurate for larger magnitudes of $R$. We have added an appendix describing these findings in the revised paper. The effect of $R$ on the generalization performance of the estimator is a very interesting question that we leave for future work.

> I understand that it's difficult to run too many experiments, but I think one interesting baseline would be to compare against a neural network trained in a supervised learning setting. [...]

We have followed the reviewer suggestion and have implemented an estimator of the form (in the reduced state space) $\hat{x}_k = \hat{x}_k^\mathrm{ROM} + f(\hat{x}_k^\mathrm{ROM},y_k)$, where $\hat{x}_k^\mathrm{ROM}$ is the reduced state generated by the uncorrected DMD ROM, and $f(\hat{x}_k^\mathrm{ROM},y_k)$ is a nonlinear correction parameterized by a feed-forward neural network (DNN). The DNN is trained in a supervised learning setting by minimizing a loss that takes the form $||U \hat{x}_k - z_k||^2$ for each data point, with $z_k$ the true high-dimensional state. We then applied this estimator to the unforced and forced Burgers examples of the paper. The estimation error is similar to the RL-ROE for the time window seen during training ($t$ from 0 to 5), but is 2 to 3 times higher than RL-ROE for the time window unseen during training ($t$ from 5 to 10). Further, the estimation error is worse when we add observation noise. This comparison suggests that our estimator design, in which the RL-trained nonlinear term $a_k \sim \pi$ directly corrects the DMD dynamics, is more robust to data not seen during training and to measurement noise. We have added an appendix  describing these results in the revised manuscript.

> The system studied here is much lower-dimensional than many systems studied in the fluids domain. Do we have any concerns about this approach scaling to higher-dimensional systems? Will learning a policy become more difficult as the dimensionality of the reduced states grows? Have you tried applying this approach to any other problems? etc.

It is true that ROMs for fluid flows oftentimes require as many as 50 dimensions to be reasonably accurate. Learning a policy will then require more computational time, but we expect the approach to remain feasible since the results currently shown in the paper required less than 30 min of training. Furthermore, since the nonlinear term $a_k \sim \pi$ acts as a correction to the reduced-order dynamics (otherwise known as a closure-model, e.g., [1]), one should take into account the fact that a very accurate ROM is not necessary in the first place (and indeed, examples shown in the paper suggest that the RL-ROE works well even when the ROM is imperfect). Therefore, our approach could potentially lead to an accurate estimator for fluid flows having much smaller dimensionality than if a Kalman filter were used. We are currently applying this approach to the Navier-Stokes equations governing the motion of fluid flows.

> (Minor) Using 15 modes for the dimensionality of the ROM seems a little random. Was this chosen based on some kind of criteria with the singular values? A quick explanation for how this was chosen could be helpful.

We chose the number of modes based on an energy criterion, with the squared singular values measuring the amount of energy captured by the corresponding modes. In the forced Burgers example, 15 modes capture 99.99\% of the energy in the snapshots used to construct the ROM. In the unforced case, 15 modes account for an even larger share of the energy, but we kept the dimensionality of the ROM the same for simplicity reasons. We have added this comment to the revised manuscript.

___

References:

[1] M. Benosman, J. Borggaard, Omer San, B. Kramer, Learning-based robust stabilization for reduced-order models of 2D and 3D Boussinesq equations, Applied Mathematical Modelling 49, 162–181, 2017.

 We thank the reviewer for taking the time to review our paper. We provide below clarifications to the concerns raised.

> The reward based on the ground truth high-dimensional state error rather than observation error would limit practical usage. The use of the the ground truth high-dimensional state also as the state variable would further limit its usability.

The reward is based on the full state error, however, this reward function is only used for off-line training of the parameters $\theta$ defining the closure term (or policy) $a \sim \pi_\theta$, which is itself conditioned on input measurements of the full state, as is usually the case in observer design, e.g., [1,2]. Using the full state data in our data-driven off-line design phase is no different from using the closed-form model knowledge in classical, model-based observer design theory, e.g., [1,2]. Indeed, assuming and using such knowledge is equivalent to assuming the use of all possible full state trajectories to design the observer, since these trajectories can be obtained by solving the model from several initial conditions. We believe that this is an important point to clarify, i.e., what is used in the offline design phase, and what is needed in the online deployment phase. We have added a remark clarifying this point at the end of Section 4 in the revised version.

> The problem with reduced dynamics is PDMDP, but treating that as a time-dependent MDP may not be the best way. 

The complication here arises from the fact that we are trying to track time-varying reference full state trajectories $z_k$, by using partial output measurements $y_k$. The state extension proposed here is meant to overcome the time-varying challenge by including $z_k$ into the augmented MDP state $s_k$. This leads to a stationary partially observable MDP (POMDP) formulation, since the action $a_k$ is drawn from a policy conditioned on $y_k$, itself a partial observation of $z_k$. We have amended the draft by explicitly writing the final POMDP problem that we are solving in equations (5), (6), and (7).

> From the demonstrated result, the state estimator has to be learned for a particular forcing inputs, which would also limit its usage.

We agree that we are not treating the case of controlled systems in this paper. In other words, we are not considering a general model of the form $z_{k+1}=f(z_k)+Bu_k$; we simply consider the case $B=0$. The controlled case $B\neq 0$ will be studied in extensions of this work. The confusion comes from the fact that we are presenting two tests, one without the forcing term $f$, and one with this term. This term is not an exogenous control input to the system, it is simply considered here as an extra term that we are adding to or removing from the model (part of the DMD matrix $A$), to test the observer on different flows of the PDE; i.e., we are simply changing the dynamics of the PDE to generate more complex flows. We have clarified this point in the revised version of the paper, and made clear that we are not treating the case of controlled systems.

> Generalization over different forms of initial state and inputs should be demonstrated, analyzed or discussed.

We agree that generalization with respect to initial conditions is an important characteristic of any state observer. In fact, we do demonstrate in our examples that our observer generalizes with respect to initial conditions, since we are testing the observer on a distribution set of initial conditions sampled from (15), rather than on selected values of initial conditions. Of course the generalization w.r.t. the initial conditions observed is of local nature, i.e., within the initial state distribution used for training. This is nevertheless similar to the state of the art in model-based observer theory, which only guarantees convergence of nonlinear observers locally, i.e., near the true initial states; see [2]. We clarified these generalization points in the revised version of the paper.

___

References:

[1]  J. Primbs, Survey of Nonlinear Observer Design Techniques, Mathematics, 1996.

[2] M. Benosman, J. Borggaard,  Robust nonlinear state estimation for a class of infinite-dimensional systems using reduced-order models, International Journal of Control, Volume 94 (5), 2021.

 We thank the reviewer for finding our paper very interesting. The main concern of the reviewer seems to be the absence of comparison with other data-driven approaches, which we address in the revised version by implementing the supervised technique proposed by the reviewer. We provide below detailed answers to the reviewer's questions.

> First, why did you use the reinforcement learning for state estimation ? In the training process, it seems to be assumed that one can access to the high-dimensional state vector $\boldsymbol{z}$. So why don't you use supervised learning method to predict $\boldsymbol{z}_t$ based on the history of past observation with a loss function of prediction error on $\boldsymbol{z}$ ? At least, it is necessary to compare RL-ROE with such straightfoward approach.

It is correct that we have access to the high-dimensional state $z$ during the training phase. The choice of an RL approach was motivated by two reasons:

1. By constructing an estimator based on an imperfect dynamic model (the DMD matrix $A$) and corrected by the RL-trained nonlinear policy $a_k \sim \pi$, we obtain an accurate dynamic model for the system. This will later allow this estimator to be paired with model-based controllers, opening the door to a wide class of control strategies that are commonly used with Kalman filters.

2. The fact that the estimator makes use of partial knowledge of the dynamics (through $A$) makes it more robust to noise or different reference trajectories from what it has seen during training, as opposed to a fully data-driven approach as proposed by the reviewer. Further, because the RL setting optimizes the reward in an average sense, we expect the nonlinear policy $a$ to display a certain amount of robustness as well.

To demonstrate the second point, we have followed the reviewer's suggestion by implementing a fully data-driven estimator consisting of a simple feed-forward neural network (DNN) taking a given number of past observations as input, and returning the state estimate as output. The DNN is trained by minimizing the squared error between the prediction and the ground truth $z_k$. We observed that the resulting estimator performs more poorly than our RL-ROE approach in the presence of observation noise. We include this comparison in the appendix of the revised manuscript.

> Secondly, I think it is unfair to compare the proposed method which is allowed to fully access to the true latent vectors $\boldsymbol{z}_t$ during the training process with Kalman filter which has no training process. In my feeling, it is not surprising that RL-ROE outperforms KF with respect to the prediction error on $\boldsymbol{z}_t$ in the test process, because RL-ROE can use the label information in the training process whereas KF cannot.

We agree with the reviewer that the proposed RL-ROE has access to more information during the training process than the Kalman filter, which never sees the high-dimensional state $z_k$, and this is indeed a major reason why the RL-ROE is able to perform better. In other words, assuming knowledge of the reduced-order model (the DMD matrix A in our case), the RL-ROE can be seen as a hybrid model-based and data-driven approach while the Kalman filter is purely model-based. Comparing model-based and data-driven approaches is very common, and our comparison of RL-ROE and Kalman is relevant since it shows what performance can be obtained in practice from each of the two approaches once a reduced-order model is known. 

Furthermore, following this reviewer feedback and other reviewers' suggestions, we have tested our hybrid observer against purely data-driven observers based on DNNs. The results still show better performance from our hybrid approach, due to the stochasticity of the RL method which has inherent robustness to measurement noises. We have included this comment and these additional numerical results in the updated version of the paper.

 We thank the reviewer for taking the time to review our paper. We are encouraged that the reviewer finds our problem interesting and important, and believes in the potential of RL techniques to learn nonlinear estimators.  We provide below clarifications to the concerns raised.

> There is no analysis on why the proposed estimate, even if the MDP is solved accurately, is a meaningful estimate [...]

We respectfully disagree with the statement `there is no hope that the estimate proposed in the paper be close to the optimal estimate'. Indeed, the idea of building observers that minimize mean-squared error for nonlinear systems, by superposing a copy of the dynamics (the DMD term $A\hat{x}$ in our case) and a nonlinear term based on the current state estimate $\hat{x}$ and the current output measurement $y$, is quite standard in control theory, e.g., [1,2]. We are using a similar paradigm, where the nonlinear part of our observer, i.e., the term $a$ is given by an RL-trained policy conditioned on $\hat{x}$ and $y$. The optimality comes from the reward function formulation, which involves a mean squared norm of the estimation error. 

This being said, we emphasize that the goal of this paper is not to provide theoretical guarantees on the estimation error, but to show via empirical results that the proposed RL-based observer performs well in estimation problems for challenging high-dimensional systems.

> The optimal control policy to the proposed MDP problem is stationary if the control is the function of the state $s = (z,\hat{x})$. However, the control is parametrized as a function of observation and $\hat{x}$. Therefore, it is not clear why such a formulation lead to a stationary policy.

The extended state is defined as $s=(z,\hat{x})$ and the observation is $y = Cz$. Thus, the nonlinear policy $a \sim \pi$ is conditioned on $(y, \hat{x}) = (Cz, \hat{x})$ which is, in fact, part of the extended state $s$. This leads to a partially observable MDP, which does not change the stationarity of the MDP in the extended state $s$; see e.g., [3].

___

References:

[1] J. Primbs, Survey of Nonlinear Observer Design Techniques, Mathematics, 1996.

[2] M. Benosman, J. Borggaard,  Robust nonlinear state estimation for a class of infinite-dimensional systems using reduced-order models, International Journal of Control, Volume 94 (5), 2021.

[3] G. Shani, J. Pineau, R. Kaplow, A Survey of Point-Based POMDPSolvers, Autonomous Agents and Multi-Agent Systems  volume 27, pages1–51, 2013.

This paper presents a method for state estimation of high dimensional dynamics by reinforcement learning. The idea is the same as that of Morimoto and Doya (2007) and the method is specialized for the context of estimating a high dimensional state vector based on a reduced dynamics model. Strength: 
The mathematical derivation looks quite solid.
Weakness: 
The reward based on the ground truth high-dimensional state error rather than observation error would limit practical usage.
The use of the the ground truth high-dimensional state also as the state variable would further limit its usability. The problem with reduced dynamics is PDMDP, but treating that as a time-dependent MDP may not be the best way.
From the demonstrated result, the state estimator has to be learned for a particular forcing inputs, which would also limit its usage. Generalization over different forms of initial state and inputs should be demonstrated, analyzed or discussed.

 From the design and demonstration, the applicability of the method seems to be limited to known, simulated systems.

The objective of the paper is to construct an estimator for the state of a high-dimensional nonlinear dynamical system given partial observations of the state. The objective is motivated by applications in fluid mechanics or turbulent flows where the state of the system is large obtained by discretizing a PDE. The proposed approach has two main steps: 

(1) construction of a reduced order model. In particular, the paper proposes the dynamic mode decomposition method which only requires a single trajectory of the nonlinear dynamics.  

(2) formulating the problem of finding the estimator as a MDP problem and application of RL techniques to solve it. In particular, the estimator is modeled as a dynamical system driven by a stochastic control policy that depends on the current value of the estimate and the value of observation. The objective function is modeled with running cost equal to the error in estimating the state and a quadratic penalty on the control. Then, the optimal control policy is learned using a policy gradient method by sampling trajectories from the system.  

The proposed approach is evaluated on a benchmark example that involves Burger's equation and compared with Kalman filter applied on the reduced order system.  The problem addressed in the paper, finding an estimator for high dimensional nonlinear systems, is interesting and important. Application of RL techniques to learn the nonlinear estimator is definitely interesting and has potential. 

However, I can not recommend acceptance based on two main objections to the basic formulation of the proposed approach. 

1. There is no analysis on why the proposed estimate, even if the MDP is solved accurately, is a meaningful estimate:  

It is not clear what kind of estimator the paper aims to find. If the objective is to find an estimator that is optimal in mean-squared error, like Kalman filter, then such estimator does not evolve with a control that only depends on the current value of the estimate and observation. The optimal estimator is equal to the conditional expectation of the state given history of observation. It is only true in linear Gaussian setting that the estimator has the form of update law proposed in the paper, i.e. a control that only depends on the estimate and current value of observation. In general nonlinear non-Gaussian setting, which is the main motivation of the paper, one has to evolve the posterior distribution in order to compute the optimal estimate. The update law for the estimate will involve a control that depends on the whole history of the observation, not just the current observation as proposed in the paper. In short, there is no hope that the estimate proposed in the paper be close to the optimal estimate.     

Now, if the objective is not the optimal MSE estimate, it is not clear how solving the proposed MDP problem can produce a meaningful estimate. The paper does not provide any analysis on this. 

2. The optimal control policy to the proposed MDP problem is stationary if the control is the function of the state s = (z,\hat{x}). However, the control is parametrized as a function of observation and \hat{x}. Therefore, it is not clear why such a formulation lead to a stationary policy.  I can not recommend acceptance because there is no analysis on why the proposed estimate is meaningful and the argument on why the proposed MDP problem have a stationary policy is wrong. 

This paper proposes a new state estimation method based on reinforcement learning for high-dimensional system obtained by discretizing continuous system originally modeled by partial differential equations (PDEs). The proposed method learns to correct the model errors caused by reduced order model (ROM). In the experiment, the proposed method named RL-ROE performs better than ordinary Kalman filter applied to the ROM. I read the paper with interest. The idea of using reinforcement learning for state estimation instead of Bayesian filtering method such as Kalman filter is very interesting, though it is not completely unique as the authors said in the section of related work.
In the experiment, they applied the proposed method RL-ROE to the data set generated by numerically solving the Burgers equation. 
It outperforms Kalman filter applied to ROM in the forced case where the errors due to ROM is large, while there is not clear advantage in the unforced case where the ROM error is small. This result seems reasonable.

I have mainly two questions. First, why did you use the reinforcement learning for state estimation ? In the training process, it seems to be assumed that one can access to the high-dimensional state vector $\boldsymbol{z}$. So why don't you use supervised learning method to predict $\boldsymbol{z}_t$ based on the history of past observation with a loss function of prediction error on $\boldsymbol{z}$ ? At least, it is necessary to compare RL-ROE with such straightfoward approach. 

Secondly, I think it is unfair to compare the proposed method which is allowed to fully access to the true latent vectors $\{ \boldsymbol{z}_t \}$ during the training process with Kalman filter which has no training process. In my feeling, it is not surprising that RL-ROE outperforms KF with respect to the prediction error on $\boldsymbol{z}_t$ in  the test process, because RL-ROE can use the label information in the training process whereas KF cannot.
 This paper proposes a very interesting state estimation method based on reinforcement learning for high-dimensional system approximated by reduced order model. However the experiment result is not so convincing enough, and it needs comparison with other data-driven approach rather than Kalman filter.

This paper proposes using RL to train a policy to output correction terms in the reduced states of reduced order models in order to perform state estimation. In particular the approach is applied to DMD and Burgers equation, but could presumably be applied to other forms of ROMs and dynamical systems. The method was shown to perform well in experiments, and was found to be more robust than baseline methods in the presence of perturbations and noise. Overall I though this was a strong paper. I think this paper presents a reasonable application for RL in a non-traditional domain, and was able  to show the approach working effectively in experiments. Having said that, my main concern is that this paper is somewhat lacking in experiments. All in all, though, I'm generally in favor of acceptance.

Some questions/clarifications/concerns:
1. I'm interested in the interplay between the Q and R terms in Eq. 6. How important is it to have the R to penalize the magnitude of actions? When does it help and hurt? My guess is R might perform some kind of regularization that allows the model to perform better in extrapolation. Can you add a comment describing what you've found.
2. I understand that it's difficult to run too many experiments, but I think one interesting baseline would be to compare against a neural network trained in a supervised learning setting. The network could have the same inputs and outputs as the policy in this work and the loss could just be the square error between the full state and the full state estimate that you get from adding the network output to the reduced state estimate and mapping back to the state space. The reduced state estimates could be generated by the ROM alone (without any corrections applied). This experiment could provide insight into how much benefit from this method comes from its ability to make nonlinear corrections. The supervised learning approach wouldn't be able to account for the effect of its errors compounding over multiple time steps, but should be easier to train than a policy gradient approach.
3. The system studied here is *much* lower-dimensional than many systems studied in the fluids domain. Do we have any concerns about this approach scaling to higher-dimensional systems? Will learning a policy become more difficult as the dimensionality of the reduced states grows? Have you tried applying this approach to any other problems? etc.
4. (Minor) Using 15 modes for the dimensionality of the ROM seems a little random. Was this chosen based on some kind of criteria with the singular values? A quick explanation for how this was chosen could be helpful. I think this paper presents an interesting approach and demonstrates its usefulness in experiments. While I feel that the experiments could have been more extensive, I think enough has been shown to warrant acceptance.

Meta-review: The paper received splitted scores, 3,3,5,8. While reviewers thought that using reinforcement learning for state estimation is interesting, they are not convinced if the proposed technique makes sense. The paper assumes that the high-dimensional state vectors are known, in which case one may use supervised learning directly. Also there's a debate regarding suboptimality of stationarity policies, which made the reviewers confusing. Although the authors try to argue that this is mostly a clarity issue, the reviewers were not convinced and didn't change their decisions.

Decision: Reject