Reinforcement Learning-Based Estimation for Partial Differential Equations
Authors: Saviz Mowlavi
Mouhacine Benosman
Saleh Nabi
Venue: Submitted to ICLR 2023 (2023)

Abstract: In systems governed by nonlinear partial differential equations such as fluid flows, the design of state estimators such as Kalman filters relies on a reduced-order model (ROM) that projects the original high-dimensional dynamics onto a computationally tractable low-dimensional space. However, ROMs are prone to large errors, which negatively affects the performance of the estimator. Here, we introduce the reinforcement learning reduced-order estimator (RL-ROE), a ROM-based estimator in which the correction term that takes in the measurements is given by a nonlinear policy trained through reinforcement learning. The nonlinearity of the policy enables the RL-ROE to compensate efficiently for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. Using examples involving the Burgers and Navier-Stokes equations, we show that in the limit of very few sensors, the trained RL-ROE outperforms a Kalman filter designed using the same ROM. Moreover, it yields accurate high-dimensional state estimates for reference trajectories corresponding to various physical parameter values, without direct knowledge of the latter.

Under review as a conference paper at ICLR 2023
REINFORCEMENT LEARNING-BASED ESTIMATION FOR PARTIAL DIFFERENTIAL EQUATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
In systems governed by nonlinear partial differential equations such as fluid flows, the design of state estimators such as Kalman filters relies on a reduced-order model (ROM) that projects the original tigh-dimensional dynamics onto com- putationally tractable cow-dimeniional space. However, ROMs are prone to large errors, which negatively affects the performance of the estimator. Here, we in- troduce the reinforcement learning reduced-ordes estimator (RL-ROE), a ROM- based estimator in which the correction term that takes in the measurements is given by a nonlinear policy trained through reinforcement learning. The nonlin- earity of the policy enables the RL-ROE to compensate efficiently for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. Using examples involving the Burgers and Navier- -Sokkrs equations, we show that in the limit of very few sensors, the trained RL-ROE outperforms a Kalman fil- ter designed using the same ROM. Moreover, yields accurate high-dimensional state estimates for reference trajectories corresponding to various physical param- eter values, without direct knowledge of the latter.
INTRODUCTION
Active control of turbulent flows has the potential to cut down emissions across a range of industries through drag reduction in aircrafts and ships or improved efficiency of heating and air-conditioning systems, among many other examples (Brunton Noack (015). But real-time feedback control requires inferring the state of the system from sparse measurements using an algorithm called a state estimator, which typically relies on model for the underlying dynamics (Simon) 2006). Among state estimators, the Kalman filter is by far the most well-known thanks to its optimality for linear systems, which has led its widespread use in numerous applications (Kalman) 1960) [Zarchan 2005). However, continuous systems such as fluid flows are governed by partial differential equa- tions (PDEs) which, when discretized, yield nigh-dimensional and oftentimes nonlinear dynami- cal models with hundreds thousands of state variables. These high- nigh-dimensional models too expensive to integrate with common state estimation techniques, especially in the context of em- bedded systems. Thus, state estimators for control are instead designed based reduced-order model (ROM) of the system, in which the underlying dynamics are projected to a low-dimensional subspace that computationally tractable (Barbagallo et al.) 2009 Rowley & Dawson 2017).
A big challenge is that ROMs provide a simplified and imperfect description of the dynamics, which negatively affects the performance of the state estimator. One potential solution is improve the accuracy of the ROM through the inclusion of additional closure terms (Ahmed et al. 202T]. In this paper, we leave the ROM untouched and instead propose new design paradigm for the estimator itself, which we call a ceiforremeent-earminn
reduced-orded estimator (RL-ROE). The RL-ROE constructed from the ROM an analogous way to a Kalman filter, with the crucial difference that the linear filter gain function, which takes in the current measurement data, is replaced by nonlinear policy trained through reinforcement learning (RL). The flexibility of the nonlinear policy, parameterized by a neural network, enables the RL-ROE to compensate for errors of the ROM while still taking advantage of the imperfect knowledge of the dynamics. Indeed, we show that in the limit of sparse measurements, the trained RL-ROE outperforms Kalman filter designed using the same ROM and displays robust estimation performance across different dynamical regimes. To our knowledge, the RL-ROE is the first application of RI. to state estimation of parametric PDEs.

---

Under review as a conference paper at ICLR 2023
2 GENERAL METHODOLOGY
2.1 PROBLEM FORMULATION
Consider the parametric discrete-time nonlinear system given by
1 f(zx: (4), yk Czz + Tek:
(la)
(1b)
where 2 E I" and yk E R are respectively the state and measurement at time k. f R" + R" is time-invariant nonlinear map from current next state, n& IR is observation noise (assumed zero unless stated otherwise), μ E R is physical parameter, and C E Rex* is a linear map from state to measurement. In this study, we assume that the dynamics given T are obtained from a high- fidelity numerical discretization of a nonlinear partial differential equation (PDE), which typically requires large number n of continuous state variables (on the order of at least a few hundreds). Nonetheless, our work is applicable to any high-dimensional nonlinear system of the form ( We do not account for exogenous control inputs to the system, which is left for future work.
Here, we will focus on the post-transient dynamics of (T); these are the observed dynamics once the transients associated with the initial condition have died down. In particular, we consider systems whose post-transient dynamics limit cycle or a quasi qquas-pprriodic limit cycle, which encompasses the behavior of a large class of phys- ical systems. The nature of the attractor is independent of the initial condition but depends on the value of μu, which we will consider to be in a range [nn M2].
deseribed by an attractor that is either a steady state, periodic
The purpose of the present work is to combine reduced-order modeling (ROM) and reinforcement learning (RL) to construct a state estimator that solves the following problem: given a sequence of measurements (.,,... 3 from a post-transient reference trajectory of (I), estimate the high- dimensional state 2 at current time k without knowledge of itself. The ROM procedure, which follows standard practices, described in Section 2c2 The integration of the ROM with RL solve the estimation problem, which constitutes the main novelty of the paper, is described in Section 2.3
2.2 REDUCED-ORDER MODEL
Since the high dimensionality of (T) renders online estimation impractical, is customary to formu- late reduced-order model (ROM) of the dynamics (Rowley & Dawson 2017). First, one chooses suitable linearly independent set of modes (ui..... u, J. where u, E R defining an r-dimensional subspace of R" in which most of the dynamics is assumed to take place. Stacking these modes as columns of a matrix U E Rexr, one can then express 2₁ = UnK. where the reduced-orde state at E IR' represents the coordinates of 2 in the subspace. Finally, one finds ROM for the dynamics of æR. which is vastly cheaper to evolve than I when r < n.
There exist various ways find appropriate set of modes U and corresponding ROM for the dynamics of æt (Taira et al., 2017). In this work, we employ the Dynamic Mode Decomposition (DMD), purely data-driven algorithm that has found numerous applications fields ranging from fluid dynamics to neuroscience (Schmid, 2010; Kutz et al.) 2016). Importantly, we seek a single ROM to deseribe dynamics corresponding to various parameter values ju E |2] since the state estimator that we will later construct based on this ROM does not have knowledge of J..
In order to apply DMD, we first construct training dataset by solving ① for values of μu belonging to finite set S C [un [22], resulting in concatenated collection of snapshots Ztrain
[ JESS. 2/ a post-transient trajectory of (lab for a specific value μ E S. The
where each Z = [z/ DMD then seeks a best-fit linear model of the dynamics in the form of matrix A E RR such that - ~ Azt for all k and μ. and computes the modes U as the r leading principal component analysis (PCA) modes of Zurain The transformation 2 Ux and the orthogonality of U then yield a linear discrete-time ROM of the form
Dg A, Wk-1,
(2a)
y/ = C, Uk:
(2b)
where A, = AT E R'rr and C, = CU E Rexr are the reduced-ordes state-transin and observation models, respectively. The (unknown) non-Gaussian process noise uy and observation
2

---

term a standard Under review as a conference paper at ICLR 2023
noise Uy account for the neglected PCA modes of Zuraia in U, well the enror incurred by the linear approximation and effective averaging of the dynamics over a range of μ. Additional details regarding the calculation of A, and U are provided in Appendix A of the supplementary materials.
2.3
REINFORCEMENT LEARNING-BASED REDUCED-ORDER ESTIMATOR
Using the ROM 22 defined by A,, C, and U, we now want to solve the estimation problem defined in Section 2.1 To this effect, design a reduced-order estimator (ROE) of the form
ix A, ak: ag ~ Ho( yk 0x-1),
(3a)
(3b)
where 2% an estimate of the reduced-order state æy and ag E R" is an action sampled from a nonlinear and stochastic policy πe, which takes as input the current measurement yı: from the reference trajectory of (Ta) and the previous state estimate 1 The subscript 0 denotes the set of parameters that define the policy, whose goal is to use the sparse measurements yı. to act on the dynamics of Φg in (3a) so that the reconstructed high-dimensional state estimate Eд = Ud converges towards the (hidden) reference state zı. Note that designing state estimators, also called state observers, by correcting the dynamics model with a mecsuremeent-eeeeneee
approach in control theory (Korovin & Fomichev 2009 Besançon 2007).
A Kalman filter a special case of such an estimator, for which the action in 3bb is given by
a₁ K₁ (3/
with Kg E R the optimal Kalman gain. Although the Kalman filter is the optimal linear filter for linear systems (Julier & Uhlmann 2004 Simon [006). its performance suffers in the presence of unmodeled dynamics and parameter uncertainty, both of which are present in our case. Thus, this motivates the adoption of the more general form (3b), which retains the dependence of a on and - but more flexible thanks to the nonlinearity of the policy Te..
We first train the policy Te in an offline phase, using deep RL to solve the optimization problem
0 arg min E
Σ || U + A||ax||
where the expectation is taken over initial estimates Lo. initial true states zo, parameters μ4, estimate trajectories (01, ....... induced by Te through B and true trajectories of states {#1, -2, .., } and measurements f11, ....... } induced by [.. The first squared term in (5) penalizes the error between the high-dimensional estimate &g = Uz and the true zg. The second squared term favors smaller values for the action ag, which acts regularization mechanism. Unless indicated otherwise, will consider À = 0. By considering different values of μu during training, a strategy called domain randomization (Peng et al.) 2018b), we ensure robustness of the policy with respect μ during online deployment of the estimator. Note that the stochasticity of тe lets the RL algorithm explore different actions during the training process, but is turned off during online deployment.
We call the estimator constructed and trained through this process an RL-trained ROE, or RL-ROE for short. Finally, an interpretation of the estimator dynamics 3 and the optimization problem in the context of Bayesian inference is presented in Appendix
2.4 SUMMARY OF THE PROPOSED METHODOLOGY
In summary, the methodology we propose consists of the following three steps. The first two are carried out offline using a training dataset Zurain {2" Jues containing high-dimensional state snapshots from multiple trajectories of (1) for various μ. The third takes place online using sole knowledge of measurements ( } from trajectory of [ corresponding to unknown not necessarily belonging to S.
1. Construction of the ROM (offline). A ROM of the form is obtained by applying the DMD to the training dataset Z.min-
2. Training of the RL-ROE (offline). An RL-ROE of the form 3 is designed based upon the ROM constructed in Step 1, and its policy To is trained using the reference trajectories contained in Ztrinn
3

---

to that stated 5 distributions Under review as a conference paper at ICLR 2023
3. Deployment of the RL-ROE (online). Using measurements {3/...... 3k from refer- ence trajectory of [T) corresponding to unknown μs, the trained RL-ROE returns an estimate 2 U2 of the (unobserved) nigh-dimensiona state Zg.
The RL-ROE, which combines ROM with a nonlinear policy trained in an RL framework to solve the estimation problem, constitutes the main contribution of the present paper. The training procedure for To. which involves non-trivial reformulation of the time-varying estimation problem into a stationary Markov decision process, is described in the next section.
OFFLINE TRAINING METHODOLOGY
In order to train hye with reinforcement learning, we need to formulate the optimization problem as a stationary Markov decision process (MDP). However, this is no trivial task given that the ai of the policy is to minimize the error between the state estimate &n U and time-dependent reference state Zi.- At first sight, such trajectory tracking problem requires time-dependent reward function and, therefore, a con-stationary MDP (Puterman) 2014 Lecarpentier & Rachelson 2019). introduce trick to translate this non-stationary
To be able to use off-the-shell RL algorithms, MDP to an equivalent stationary MDP based on an extended state. Specifically, we show hereafter that the problem removing the time dependence from the reward function.
be framed as stationary MDP whose state has been enhanced with 23 and 14,
Letting & = (# Zk. (1) E B*+n+I denote augmented state at time kt we can define an MDP consisting of the tuple (S,A, PPRRI R). where = -I+*+1 is the augmented state space, A C R' is the action space, P(-|*k: as) is a transition probability, and R( Og 8k+1) is reward function. At each time step k; the agent selects an action 0g E A according to the policy ne defined in (3b which can be expressed as
Te( 1ox),
where O₁ = (yk, 2x-1) = (Czk, 1 is a partial observation of the current state 8g. The state 8k+1 (2), Zk+1, u)) at the next time step is then obtained from equations (Ia and (3a)
Sk+1 = (A, - + ax. f(z:: 1))
which defines the transition model 8k+1 (( ).). Finally, the agent receives the reward
Fk R(sk, ak; 8k+1) - -1 Uw A||ax113,
(8)
which is minus the term to be minimized at each step in 5 Thanks to the incorporation of z into sk. the reward function has no explicit time dependence and the MDP is therefore stationary.
The RL training process then finds the optimal policy parameters
0* arg max E [R(T)], T~-mo
where the expectation over trajectories = (8) 01, 82, a2,...), and R(T) E 2h is the finite-horizon undiscounted return. Thus, the optimization problem ( solved by RL is equivalent At the beginning of every episode, the environment is reset according to the
to ~ 2e-Pa 2n ~ (.) P p~ pp)))
(10)
from which the augmented state 81 (&o, 21, = (wo, f(zu 1); H) follows immediately: 81 thus constitutes the start of the agent-environment interactions. The distribution Pa, (-) bestows robust- ness of the learned policy with respect to the initial state estimate øo. while the distributions Pz, (-) and (-) enable the same policy Tо to be trained on several reference trajectories of (1) correspond- ing to various μ E [Mn /d2]. In practice, we reuse the same reference trajectories corresponding to S from the training dataset Z utilized to construct the DMD in Section 2.2 that we do not have to keep solving (lab during the training process. Thus, at the beginning of every episode, we draw random /a from S and initialize 2u the starting state z/ of the corresponding trajectory Z
To learn 0* employ the Proximal Policy Optimization (PPO) algorithm (Schulman al. 2017 which belongs to the class of policy gradient methods (Sutton et al. 2000). The parameterization of the policy na, implementation details and training hyperparameters are discussed in Appendix
4

---

Under review as a conference paper at ICLR 2023
Remark. Since the policy (6) is conditioned partial observation o of the state 8g, the stationary MDP we have defined in this section is, in fact, a partially observable MDP (POMDP). In this case, is known that the globally optimal policy depends summary of the history of past observations and actions, h2 ; a1...., ). rather than just the current observation Og (Kaelbling et al.) 1998). However, policies formulated based and still achieve good results (Sutton & Barto 2018). We therefore pursue this approach in the present paper, and leave for future work testing the generalization of our policy input to a more complete summary of fag. We also note that policy gradient methods, which PPO belongs to, do not require the Markov property of the state (that is, conditional independence of future states on past states given the present state) and can therefore be readily applied to the POMDP setting. For our problem, this guarantees that the PPO algorithm will converge to locally optimum policy.
incomplete summary of h& are common in practice
4 RESULTS
We evaluate the state estimation performance of the RL-ROE for systems governed by the Burgers equation and Navier-Stokes equations. For each system, we first compute various solution trajecto- ries corresponding to different physical parameter values, which we use to construct the ROM and train the RL-ROE following the procedure outlined in Section 2.4 The trained RL-ROE is finally deployed online and compared against a time-dependent Kalman filter constructed from the same ROM, which we refer to as KF-ROE. The KF-ROE is given by equations (3a) and ( with the calculation of the varying Kalman gain detailed in Appendix of the supplementary materials.
Before proceeding to the results, we discuss choice of baseline. The ensemble Kalman filter and 4D-Var are two estimation techniques for high-dimensional systems such as those governed by PDEs
Lorene 2003 Although they are commonly employed for data assimilation in numerical weather prediction, they require large computational resources since they involve repeated solutions of the high-dimensional dynamics [ Thus, they are not applicable in the context of embedded control systems, whose limited resources call for an inexpensive model such as the ROM 2 Since the ROM that we consider has linear dynamics, extensions of the Kalman filter for nonlinear dynamics such as the extended or unscented Kalman filters (Wan & Van Der Merwe 2000 Julier & Uhlmann 2004 are not relevant, and the vanilla Kalman filter remains the best choice of baseline.
4.1 BURGERS EQUATION
The forced Burgers equation is prototypical nonlinear hyperbolic PDE that takes the form
Ou Ot
Ou
8u
+ u
V
O..2 = f(x t),
(11)
Or
where u(x, t) is the velocity at position x E [0, L] and time f, f fxx, t) is distributed time-dependent forcing, and the scalar acts like viscosity. Here, we choose a forcing of the form
f(x 1) = 2snn(wl km)) + 2 sin(3wl kx)) + 2 sin(t kx),
(12)
where h: = 2 and we let v and w be related through a scalar parameter ya E [0, 1] as follows:
v1 + (v2 v1),,,
w W1 + (w2 w1)H..
(13a)
Thus, a can be regarded as physical parameter that affects the dynamics of the forced Burgers equation through both v and W. We consider periodic boundary conditions and choose L 1, 1/1 = 0.01, = 0.1, w1 = 0.2 W1 = 0.4T.
We solve the forced Burgers equation using a spectral method with n 256 Fourier modes and a fifth-order Runge- Kutta time integration scheme. We define the discrete-time state vector E IR" that contains the values of u at n equally-spaced collocation points and at discrete time steps t t= k&t, where At = 0.05. To generate the training dataset Ztrain [Z") JES used for construct- ing the ROM and training the RL-ROE, we compute solutions of the Burgers equation corresponding to u E S {0, 0.1, 0.2....... 0.13. For each μ., we discard the transient portion of the dynamics and save 201 snapshots Z = {=%..... 2 } in the post-transient regime. We retain r = 10 modes reduction in the dimensional- ity of the system. We train the RL-ROE using episodes of length K = 200 steps to make full use of the trajectories stored in Zirain and we end the training process when the return no longer increases
when constructing the ROM, corresponding to
5

---

Under review as a conference paper at ICLR 2023
J 0.05
u140.45
J 0.95
2.0
normalized error
BLROE
normalized L, errar
1.5
1.0
lower bound
normalzzd
1.0
0.5
0.5
2.5
5.0
10.0
7.5
time
time
time
Figure I: Burgers equation with = 4 sensors. Normalized Le error of the RL-ROE and KF-ROE for the estimation of trajectories corresponding to values of not seen during training.
y 0.
μl 0.45,
H- 0.95, reference
0.75
0.6
1.8
12
0.0
0.00
0.5
0.0
0.0
0.3
0.9
-0.5
0.00
-1.8
-1.0
1.5
-0.9
1.2
H 0.05,
μ- 0.45, RL-ROE
H-9.95, RL-ROE
WW
0.75
1.6
-10
0.8
0.55
0.4
0.50
0.0
0.0
0.5
-0.4
-1.6
-1.0
-0.8
0.00
-2.4
11.5
#- 0.05,
H4 0.45,
# 0.95,
RE 2e
0.75
0.5
0.50
0.0
0.0
-066
-0.5
-1.0
0.00
-1.
10
2
10
6 10
Figure 2: Burgers equation with p == sensors. Reference trajectories for values of μ not seen during training and corresponding RI-ROE and KF-ROE estimates. The dashed lines on the reference trajectory plots indicate the sensor data seen by the RL-ROE and KF-ROE.
on average. The RL hyperparameters and learning curve displaying the performance improvement of the RL-ROE during the training process are reported in Appendix D.
The trained RL-ROE and the KF-ROE are now compared based on their ability to track reference trajectories corresponding to various using sparse measurements of from a limited number p of cqually-spaced sensors (Appendix describes the corresponding C). The evaluation carried out using 20 initial state estimates sampled from Gaussian distribution with unit standard deviation. Beginning with p = 4 sensors, Figure reports the mean (lines) and standard deviation (shaded areas) of the nor- malized L2 error for 3 values of not present in the training dataset Zurain used to construct the ROM and train the RL-ROE. The normalized Ly error is defined as || = 1//1| Il, where 2₁ is the figh-dimensional ref-
0.8
error (time avgl
AL-ROE (train)
0.6
RL-RDE (test)
KF-ROE (train)
lower bound
erence solution and & = UZ is the high-dimensional reconstruction of the RL-ROE The error of the RL-ROE is very close to the lower bound, which the error incurred by projecting the reference solution to the modes U, i.e. the lowest possible er- ror achievable by an ROE based on U. Seatio-temporal contours of the reference solutions for the same 3 values of μ and the corresponding RL-ROE and KF-ROE high- dimensional estimates shown in Figure 2 The RI-
KF-ROE estimate á
0.0
0.0
0.2
0.4
0.6
0.8
1.0
parameter
Figure 3: Burgers equation with p = sensors. Time average of the normal- ized L2 error versus μ.. Values of μ present Zerann shown by large circles.

---

Under review as a conference paper at ICLR 2023
ROE vastly outperforms the KF-ROE, which demonstrates the superiority of nonlinear correction to the estimator dynamics Bap. We emphasize that the Kalman filter was tuned to obtain the best possible results for the KF-ROE (see Appendix for details the tuning process).
Figure reports the time average of the normalized L2 error as a function of μ for p = 4, with the values present in train indicated by large circles. The RL-ROE exhibits robust performance across the entire parameter range μ E [0, 1]. including when estimating previously unseen trajectories.
Finally, Figure 4 displays the average over time and over of the normalized Lz2 error for varying number of sen- sors. Note that each value of p corresponds to a sep- arately trained RL-ROE. As the number of sensors in- creases, the KF-ROE performs better and better until its accuracy overtakes that of the RL-ROE, We hypothesize that the accuracy of the RL-ROE is limited by the inabil- ity of the RL training process to find an optimal policy, due to both the non-convexity of the optimization land- scape well as shortcomings inherent to current deep RL algorithms. This being said, the strength of the non- linear policy of the RL-ROE becomes very clear in the very sparse sensing regime; its performance remains re- markably robust as the number of sensors reduces to 2 or even 1. In Appendix[F] spatio-temporal contours (similar as in Figure of the reference solution and corresponding estimates for p 2 and 12 illustrate that the slight advantage held by the KF-ROE for 12 is reversed into clear superiority of the RL-ROE for p =4. 4.
RLROE KF-ROE
1.00
0.75
0.50
0.25
10
12
14
number of p
Figure 4: Burgers equation. Average over time and over μ of the normalized L2 error versus number p of sensors.
4.2 NAVIER- STOKES EQUATIONS
The Navier-Stoke equations are a set of nonlinear PDEs that describe the motion of fluids flows. For incompressible fluids, the Navier-Stokes equations take the form
Du Øi + (u V)u = - Vp + Au,
(14a)
Re
v.u=0,
(14b)
where u(x, t) and p(x,t) are the velocity vector and pressure at position and time t, and the scalar Re is the Reynolds number. We consider the classical problem of flow past cylinder in a 2D domain, which well known to exhibit Hopf bifurcation from a steady wake to periodic vortex shedding at a critical Reynolds number Re, ~ 40 (Jackson) 1987). For our study, we focus on the range Re E [10, 110), which makes the estimation problem very challenging since this range includes the bifurcation and therefore comprises solution trajectories with very different dynamics- steady for Re+ < Re periodic limit cycle for Re > Rec- Furthermore, the shedding frequency and spacing between consecutive vortices in the limit cycle regime both vary with Re.
We solve the Navier-Stokes equations with the open source finite volume code OpenFOAM using mesh consisting of 18840 nodes and second-order implicit scheme with time step 0.05. The discrete-time state vector 2₁ E 3736680 contains the two velocity components of u at discrete time steps = k&t, where we choose At 0.25. To generate the training dataset Zirain = { Z fte ReEs for constructing the ROM and training the RL-ROE, we run simulations of the Navier-Stokes equa- tions for Re E S = {10, 20, 30,... .., 110). We discard the transient portion of the dynamics (for the cases Re > Re,) and save 201 snapshots Z Re = [=
200 Rc ..... z Re in the post- trasstranieet regime.
We retain r 20 modes when constructing the ROM, corresponding to a three-orders-ol -f-agnituue reduction in the dimensionality of the system. These modes are shown in Appendix G We train the RL-ROF using episodes of length K 200 steps to make full use of the trajectories stored in Zirain and end the training process when the return no longer increases average. The RL hyperparame- ters and learning curve displaying the performance improvement of the RL-ROE during the training process are reported in Appendix D.
The trained RL-ROE and the KF-ROE are now compared based on their ability to track ref- erence trajectories corresponding to various Re using sparse measurements of u from lim- ited number of sensors randomly distributed in the wake region behind the cylinder (see Ap- pendix B for the construction of the corresponding C). The evaluation is carried out using
7

---

Under review as a conference paper at ICLR 2023
Re
Re
Ro - 105
noralizzd errar
1.00
RLROE
1.00
1.00
lower bound
0.50
pormaizzo
0.50
0.50
0.25
0.25
0.25
10
40
10
40
10
40
time
time
time
Figure 5: Navier-Stokes equations with p = sensors. Normalized L2 error of the RL-ROE and KF-ROE for the estimation of trajectories corresponding values of Re not seen during training.
Re 35, reference
Re 65, reference
Re 105, reference
1.2
122
1.0
1.0
0.8
0.48
0.6
0.32
0:16
0.00
0.0
0.0
Re 35, RL-ROE
Re 65, RL-DDE
Re 105, RL-ROE
1.28
0.64
0.80
0.64
0.8
0.48
0.48
0.32
0.32
0.16
0.16
0.00
0.00
0.0
Re 35, KF-ROE
65, KF-ROE
105, KF-ROE
1.12
0.80
0.96
0.48
0.60
0.64
0.80
0.64 3
0.36
0.48
0.48
0.24
0.32
0.12
0.16
0.00
0.00
Figure 6: Navier-Stokes equations with p 3 sensors. Velocity magnitude at f 50 of the reference trajectories for values of He seen during training and corresponding RL-ROE and KF-ROE estimates. The black crosses in the contours of the reference solutions indicate the sensor locations.
initial state estimates sampled from a Gaussian distribution with unit standard deviation. Be- ginning with p 3 sensors, Figure reports the mean (lines) and standard deviation (shaded areas) of the normalized Le error for 3 values of Re not present in the training dataset Zurain used to construct the ROM and train the RL-ROE. The RL-ROE almost accurate the lower bound, while the KF-ROE has normalized error around 1, which indicates complete failure. The velocity magnitude of the reference solutions for the same 3 values of Re and the corresponding RL-ROE and KF-ROE reconstructed high-dimensional estimates shown at t 50 in Figure 6 Remarkably, the RL- ROE manages to estimate very precisely the entire flow field across different dynamical regimes, with the steady wake at Re = 35 being reproduced equally well as the wake of vortices at Re 65 and 105. The KF-ROE, the other hand, struggles to estimate the flow fields for all 3 Reynolds numbers, and instead predicts velocity field that is almost everywhere zero except in the wake. Again, the superiority of the RL-ROE granted by the nonlin- earity of its policy in fact, note that bifurcations such as the one exhibited by this flow are inherently nonlinear phenomena. Appendix HotixxxHShhooo shows corresponding results in the presence non-zero observation noise.
1.0
error (time avg)
RL-ROE (train) (test)
KF-ROE (train)
0.4
KF-ROE (test)
lower bound
0.2
0.0
20
40
80
100
Reynolds number
Figure 7: Navier-Stokes equations with
p = 3 sensors. Time average of the nor- malized Le error versus Re. Values of Re present in Zurain shown by large cir- cles.
8

---

Under review as a conference paper at ICLR 2023
Figure 7 reports the time average of the normalized L2 error as a function of μ for p = 3, with the values present in Zrrin indicated by large circles. The RL-ROE exhibits robust performance across the entire range of Reynolds numbers, including in the vicinity of the bifurcation at Re,- 40 and for values of Re not seen during training.
1.0
error (time and j2daavv avg)
RLRDE KF-ROE
0.8
0.6
0.4
0.2
FigureBddispays the average over time and over Re of the normalized L2 error for varying number p of sensors. Al- though the KF-ROE eventually becomes very accurate in the presence of a large number p of sensors, the accuracy of the RI.ROR remains remarkably stable as p decreases, allowing it to vastly outperforms the KF-ROE as soon as p < 8. Once again, this showcases the benefits of using a nonlinear policy to correct the estimator dynamics (3a)
0.0
12
number of sensors p
Figure 8: Navier-Stokes equations. Av- erage over time and over Re of the nor- malized L2 error versus number of sensors.
5 RELATED WORK
Previous studies have already proposed designing state estimators using policies trained through reinforcement learning. Morimoto & Doyal (2007) introduced an estimator of the form Tt = f(x-11) + L( the state-dependent filter gain matrix -1) is defined using Gaussian basis functions whose parameters are learned through a variant of vanilla policy gradient. HHu et al. (2020) proposed an estimator of the form & = f(ix-1)+ + L(x; - Z&k (yk - Cf (&)= where L(x kx) is ap- proximated by neural networks trained with a modified Soft-Actor Critic algorithm (Haarnoja al.) 2018). Although they derived convergence properties for the estimate error, the dependence of the filter gain L(x;- - āxx) on the reference state æ limits its practical application. Furthermore, a ma- jor difference between these past studies and our work is that they only consider low-dimensional systems with four state variables at most. Our RL-ROE, on the other hand, handles parametric PDEs described by tens of thousands of state variables as shown in the previous section.
Cxx-1), where f(-) is the state-transition model of the system, and
In another line of works, reinforcement learning has been applied learn control policies for joint torques that enable simulated characters to imitate given reference motions consisting of a sequence of target poses; see e.g. |Png et al.] (2018a) or Lee et al. (2019). These are essentially trajectory tracking problems, as the policy learns to drive the character's 's pose towards that defined by the ref- erence motion. Similar to us, these studies also propose to transform the problem into a stationary MDP by augmenting the state; but they do by appending a scalar phase variable ø E [0, 1] that represents the normalized time elapsed in the reference motion. The reward can then be formulated in terms of q(ő), where q-)) describes the reference motion, and no explicit time dependence ap- pears. However, this approach would not work for our purpose since we do not wish to restrict the RL-ROE to estimating single reference trajectory for each value of p. Many dynamical systems indeed admit multiple post-transient trajectories for given parameter values (Cross & Hohenberg. 1993 Augmenting the MDP's state with the entire snapshot from the reference trajectory instead of just time ensures that the policy Te can learn any number of reference trajectories for each p.
CONCLUSIONS
In this paper, we have introduced the reinforcement learning reduced-order estimator (RL-ROE), a new state estimation methodology for parametric PDEs. Our approach relies on the construc- tion of computationally inexpensive reduced-order model (ROM) to approximate the dynamics of the system. The novelty of our contribution lies in the design, based on this ROM, of reduced- order estimator (ROE) in which the filter correction term is given by a nonlinear stochastic policy trained offline through reinforcement learning. We demonstrate using simulations of the Burgers and Navier-Stoker equations that in the limit of very few sensors, the trained RL-ROE vastly out- performs a Kalman filter designed using the same ROM, which is attributable to the nonlinearity of its policy (see Appendix | for quantification of this nonlinearity). Finally, the RL-ROE also yields accurate high-dimensional state estimates for reference trajectories corresponding to various parameter values without direct knowledge of the latter.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Shady E Ahmed, Suraj Pawar, Omer San, Adil Rasheed, Traian Iliescu, and Bemd R Noack. On closures for reduced order models-4 spectrum of first-principle to machine-learned avenues. Physics of Fluids, 33(9):99331,, 2021.
Alexandre Barbagallo, Denis Sipp, and Peter J Schmid. Closed-loop control of open cavity flow using reduced-order models. Journal of Fluid Mechanics, 641:1-50, 2009.
Gildas Besançon. Nonlinear observers and applications, volume 363. Springer, 2007.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.0/ 2016.
Steven L Brunton and Bernd R Noack. Closed-loop turbulence control: Progress and challenges. Applied Mechanics Reviews, 67(5), 2015.
Mark C Cross and Pierre C Hohenberg. Pattern formation outside of equilibrium. Reviews of modern physics, 65(3):851, 1993.
Fred Daum and Jim Huang. Curse of dimensionality and particle filters. In 2003 IEEE Aerospace Conference Proceedings (Cat. No. 03TH8652), volume pp. 19119994449999 1993. IEEE, 2003.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Liang Hu, Chengwei Wu, and Wei Pan. Lyapunov-based reinforcement learning state estimator. arXiv preprint arXiv:2010.13529, 2020.
CP Jackson. A finite-llement study of the onset of vortex shedding in flow past variously shaped bodies. Journal of fluid Mechanics, 182:23-45 1987.
Simon Julier and Jeffrey Uhlmann. Unscented filtering and nonlinear estimation. Proceedings of the IEEE, 92(3):401-422 2004.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Plamning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):9-134, 1998.
R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82(1):35-45, 1960.
Steven M Kay. Fundamentals of statistical signal processing: estimation theory. Prentice-Hall, Inc., 1993.
Sergey Korovin and Vasily Fomichev. State observers for linear systems with uncertainty. In State Observers for Linear Systems with Uncertainty. de Gruyter, 2009.
J Nathan Kutz, Steven Brunton, Bingni Brunton, and Joshua I. Proctor. Dynamic mode decom- position: data-driven modeling of complex systems. SIAM, 2016.
Erwan Lecarpentier and Emmanuel Rachelson. Non-stationary markov decision processes, worst- case approach using model-based reinforcement learning. Advances in neural information pro- cessing systems, 32, 2019.
Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. Scalable muscle-actuated human simulation and control. ACM Transactions On Graphics (TOG), 38(4):1-13, 2019.
Andrew C Lorenc. The potential of the ensemble kalman filter for nwp comparison with 4d-var. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography, (11555)) 2003.
Jun Morimoto and Kenji Doya. Reinforcement learning state estimator. Neural computation, 19(3): 730-756, 2007.
10

---

Under review as a conference paper at ICLR 2023
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1-14, 2018a.
Xue Bin Peng, Marcin Andrychowicz. Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3803-3810 IEEE, 2018b.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dor- 2019.
mann. Stable baselines3, http://///g.auuu...o/o//////////
Clarence W Rowley and Scott TM Dawson. Model reduction for flow analysis and control. Annual Review of Fluid Mechanics, 49:387-417, 2017.
Simo Särkkä. Bayesian filtering and smoothing. Cambridge university press, 2013.
Peter Schmid. Dynamic mode decomposition of numerical and experimental data. Journal fluid mechanics, 656:5-25 28, 2010.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1aoo:::777004444 2017.
Dan Simon. Optimal state estimation: Kalman, H infinity, and nonlinear approaches. John Wiley & Sons, 2006.
Chris Snyder, Thomas Bengtsson, Peter Bickel, and Jeff Anderson. Obstacles to high-dimensional particle filtering. Monthly Weather Review, 136(1 (1611)::469-44640 2008.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advaccee in neural informa- fion processing systems, pp. 1057-1063, 2000.
Kunihiko Taira, Steven L Brunton, Scott TM Dawson, Clarence W Rowley, Tim Colonius, Bev- erley J McKeon, Oliver Schmidt, Stanislav Gordeyev, Vassilios Theofilis, and Lawrence Ukeiley. Modal analysis of fluid flows: An overview. Aiaa Journal, 55(12):4013-404) 2017.
Jonathan H Tu, Clarence W Rowley, Dirk M Luchtenburg. Steven L Brunton, and Nathan Kutz. On dynamic mode decomposition: theory and applications. Journal of Computational Dynamics, 1(2):391 421, 2014.
Eric A Wan and Rudolph Van Der Merwe. The unscented kalman filter for nonlinear estimation. In Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373), pp. 153-155 leee, 2000.
Paul Zarchan. Progress in astronautics and aeronautics: fundamentals of Kalman filtering: prac- tical approach, volume 208. Aiaa, 2005.
A DYNAMIC MODE DECOMPOSITION
In this appendix, we describe the DMD algorithm (Schmid, 2010 Tu et al. 2014), which is a pop- ular data-driven method to extract spatial modes and lowedimensional dynamics from a dataset of high-dimensional snapshots. Here, we use the DMD to construct a ROM of the form Q given an observation model C and concatenated collection of snapshots Zerain z = - 2/ Fundamentally, the DMD seeks best-fit linear model of the dynamics in the form of a matrix A E Rax* such that Z ~ Azg. First, arrange the snapshots into two time-shifted matrices
Zz Jess, where each contains snapshots from a trajectory of (la) for specific value
X
(=/1
20
(15a)

---

Under review as a conference paper at ICLR 2023
Y {2]
H2g mm J
(15b)
where & denotes the number of elements in S. The best-fit linear model is then given by A = Yxt, where X is the pseudoinverse of X. The ROM is then obtained by projecting the matrices A and C' onto basis U consisting of the leading left singular vectors of X, which approximate the leading PCA modes of Z. Using the truncated singular value decomposition
X = =
(16)
where U, V E R=xr and E Rexr, the resulting reduced-ort orddr -tatetrransiiinn and observation models are given by
A, = U A -UA
(17a)
C, CU.
(17b)
Conveniently, the ROM matrices A, and C, can be calculated directly from the truncated SVD of X, which avoids forming the large n x n matrix A.
B BAYESIAN INTERPRETATION
Here, we evaluate the meaningfulness of the RL-ROE design by framing the estimator dynamics ( and the optimization problem (5) in the context of Bayesian inference. Unless stated otherwise, we
consider the estimation problem in terms of the reduced state Øs governed by the ROM 2
B. BAYESIAN OPTIMAL FILTER
From Bayesian perspective, the goal at each time h 1,..., K is calculate p(æx [V*ak), the posterior probability density function (pdf) measuring our belief the true state Ti given the mea- surement data fı.k (.-,..., yv ). It is assumed that we know the transition model p[ag//r>> describing the dynamics of the system, the observation model p(yx (aa)) relating state and measure- ment data, and the initial pdf p(20))- Then, the posterior pdf p(æ; y1.k) can formally be obtained recursively by alternating between prediction and update steps (Särkää 2013
Prediction step. Starting from the posterior pdf p(æx-i|b)ua of the system to compute the prior pdf at ls via the Chapman-Kolmogory equation
first uses the dynamics
p(ar 1onklynk-11 = / p(ax ()(()))((()))) 1d)
(18)
where the Markovian property of the dynamics has been used.
Update step. The prior is then updated using the new measurement yk via Bayes' rule, yielding the posterior pdf
p( (( p(yn (x (11
(19)
pl 8k 911k-1)
where the normalizing constant is p(y 11 p(yn ax (p(#x/yyik-iddaa
These equations generally do not admit analytical solutions, except in the linear and Gaussian setting which results in the Kalman filter (Särkkä 2013). Particle filters provide approximate solution
in the general setting but they are computationally expensive, their required ensemble size scaling
exponentially with the state dimension (Daum & Huang 2003 Snyder al. 2008
B.2 RL-ROE FROM BAYESIAN PERSPECTIVE
To frame the RL-ROE in the context of Bayesian inference, we first need to relate the state estimate 2 to the posterior p(æ: 31.4). Specifically, we define T; as the mean of the posterior,
/ x2p(( (dakk
E. [ y1:k]
(20)
which, assuming that p(z, y1.k) is known, is commonly referred to as the minimum mean square enror (MMSE) estimator (Kay 1993). Using this definition, can interpret the estimator
12

---

Under review as a conference paper at ICLR 2023
dynamics and the optimization problem from the perspective of the Bayesian filter equations,
with the transition model p(æ* (Tk-1) and the observation model p(3 (x)) given by (2ah and 2b
Prediction step. Starting from the estimate Øx-1 E[ y1 at &s - 1, we take the expecta- tion of (8) to obtain the 'prior' estimate at kt
K = E[ V1.--1] A, 2
(21)
where we have made use of the linearity of the transition model 2a and we have assumed that the process noise wy zero-mean (which is empirically observed in our examples; see Appendix )
Update step. Formally, the estimate Ly is given by 20 using the posterior pdf p(#x/v/1:)) from the Bayes update [9]. However, 19 cannot be solved explicitly since the RL-ROE does not evolve the full prior ((=x\/11k-1) Instead, we make use of the well-known fact that the MMSE estimator equivalent to minimizing the mean square error, or variance, of the posterior estimate (see, e.g.
chapters 10 and 11 in Kay (993) or chapter 2 in Särkkä 2013)). This can be seen follows:
El y1:k]
map(ar yr.k)dar
arg min ||00 tx p<
arg min E [||2 Edll [1
(22)
2b
We relax the above minimization problem by restricting ag specific class of functions, as is done when deriving linear MMSE estimators. In the latter case, the optimal solution is
2x
(31 -
(23)
where the prior estimate 2x is given by 21 and K₂ is obtained from the closed-form solution of Kay 1993). We generalize this approach by considering the nonlinear form
22
# के + Me, (yk: +T)):
(24)
where noo is a nonlinear function parameterized by Og. whose role is to update the prior t5. using yk in such a way to minimize the posterior variance. Then, the minimization problem (2
becomes
0% = arg min E a - =+
(25)
02
where it is implicitly assumed that the expectation is conditioned Y1-k-
RL-ROE, To obtain the estimator dynamics 3 and the optimization problem we further con-
sider that the function μee is stochastic and independent of time; it is therefore expressed by
stationary policy Te parameterized by 0. Then, combining the posterior estimate 24 with the prior estimate gives the recursion
a 0, + Ok::
(26a)
0g. TT( 1y
#x=1,,
(26b)
which is the estimator dynamics ing the minimization problem 25 to all time steps, yielding
The parameters θ of the stationary policy are found by extend-
K
0t arg min E
Σ
I - 1
(27)
In the RL setting, this optimization problem solved in a data-driven manner by sampling system trajectories. Thus, the expectation is taken over initial estimates &n. initial true states 2p, parame- ters μ4, estimate trajectories (#1, ....... induced by Te through (3). and true trajectories of states {æ1, ....... } and measurements {y1, y2, J. Finally, we express the posterior variance through the reconstructed digh-dimensional estimate &µ = U Ub and the high-dimensional true state #k, and add regularization term that penalizes the magnitude of the action ag. This gives
0 = arg min
Σ
|
U + Allaa11
(28)
0
13

---

Under review as a conference paper at ICLR 2023
which is the optimization problem
We conclude this analysis with a couple remarks. First, it demonstrates that the cost being minimized in (28) derives from the variance minimization principle underlying the definition of the MMSE estimator. Second, an important difference between the RL-ROE and the Bayesian optimal filter is that the RL-ROE does not require knowledge of the distribution of the process noise wt and observation noise Ug. Rather, it leverages knowledge of the true trajectories and corresponding measurements in the offline training phase find the policy He that yields the minimum mean
square error. Third, the RL solves the optimization problem (28) in a batch approach during the offline training phase, sampling entire trajectories of the estimate between each update of the policy parameters. The trained RL-ROE is then applied online in a recursive manner.
C KALMAN FILTER
The time- -eme-depeddent Kalman filter that we use as a benchmark in this paper, KF-ROE, is based on the same ROM (2) as the RL-ROE, with identical matrices A,, C, and U. Similarly to the RL-ROF, the reduced-order estimate & is given by equation (3a), from which the high-dimensional estimate is reconstructed as bい U x. However, the KF-ROE differs from the RL-ROE in its definition of
the action ag in 3a). which is instead given by the linear feedback term (4). The calculation of the
optimal Kalman gain K2 in requires the following operations each time step:
PE = A + Qs
(29)
S C,P, C. + R
(30)
K₂ PV C.S.
(31)
PA = (I -I-KK.C.)P..
(32)
where PK and Pa respectively the a priori and a posteriori estimate covariance matrices, Sy is the innovation covariance, and Qs and Rs are respectively the covariance matrices of the process noise wt and observation noise Uy in the ROM ( Following standard procedure, we tune these noise covariance matrices to yield the best possible results (Simon) 2006). We assume that Qx = I6I and R₂ = IgI and perform a line search find the values BQ and 3R that yield the best performance. This resulted in BQ 10 and 3R 1 for the Burgers example, and SQ 10 and RR = 1 for the Navier-Stokes example. At time step k = O, the posteriori estimate covariance initialized as Po Cov(U 20- zo), which can be calculated from the distributions (10).
D RL ALGORITHM, HYPERPARAMETERS AND LEARNING CURVES
We employ the Proximal Policy Optimization (PPO) algorithm (chhlman al. 2017 to find the optimal policy parameters 0* PPO alternates between sampling data by computing a set of trajecto- ries (T1, 72, T..... using the most recent version of the policy, and updating the policy parameters in a way that increases the probability of actions that led to higher rewards during the sampling phase. The policy no encodes diagonal Gaussian distribution described by neural network that maps from observation to mean action, Ho (ox), together with a vector of standard deviations or, so that 0 O0', ).. We utilize the Stable Baselines3 (SB3) implementation of PPO RRalin al. 2019) and define our MDP as a custom environment in OpenAl Gym (Brockman et al. 2016).
For both the Burgers and Navier-Stokes examples, the stochastic policy Te is trained with PPO using the default hyperparameters from Stable Baselines3, except for the discount factor which we choose as 0.75. The mean output of the stochastic policy and the value function are approximated by two neural networks, each containing two hidden layers with 64 neurons and tanh activation functions. The input to the policy is normalized using a running average and standard deviation during the training process, which alternates between sampling data for 10 trajectories (of length 200 timesteps each) and updating the policy. Each policy update consists of multiple gradients steps through the most recent data using 10 epochs, minibatch size of 64 and learning rate of 0.0003 The policy is trained for a total of one to three million timesteps, corresponding to 5000 to 15000 trajectories, which depending on the dimensionality of the ROM takes between 15 min and an hour on a Core 17-12700K CPU. Figure Segure/repore reports the learning curves corresponding to the unforced and forced cases. During training, the policy is tested (with stochasticity switched off) after each update
14

---

Under review as a conference paper at ICLR 2023
p-1
20000
total reward
-40000
-60000
80000
-100000
10
total reward
-40000
-60000
-8000
100000
5000
10000
5000 10000 episode
5000 10000 episode
5000 episode
10000
episode
(a) Burgers
1e6
0.0
-0.2
tocal reward
-0.4
-0.6
-0.8
1e6
p=9
-0.2
tocal reward
-0.4
0.6
-0.8
1.0
2000 4000
6080
2000 4000 6000
2000 4000 6000 episode
opisade
opisode
(b) Navier-Stokes
Figure 9: Learning curves for the stochastic policy of the RL-ROE for the (a) Burgers and (b) Navier- Stokes cases. Each plot corresponds to different number of sensor measurements. The line and shaded area show the mean and standard deviation of the results over 10 runs, each smoothed with moving average of size 100 episodes.
using 20 separate test trajectories, and is saved if it outperforms the previous best policy, Finally, the RL-ROE is assigned the latest saved policy upon ending of the training process, and the stochasticity of the policy is switched off during subsequent evaluation of the RL-ROE.
E CONSTRUCTION OF THE OBSERVATION MATRIX
We describe how construct the observation matrix C in the Burgers and Navier-Stokes examples, once the number, type and locations of the sensors have been chosen.
In the Burgers example, the state vector 2 RR" contains the values of at collocation points, and the measurements yı E R consist of the values of at p equally-spaced sensors. Let us introduce the indices (f1,-.-3āp) of the entries in žμ corresponding to the measurements yk- Then, yr and
15

---

Under review as a conference paper at ICLR 2023
24 can be related by yı: Czy. where the matrix C E IR**" contains ones at the entries indexed ((1.))...... (p:jp),, (P)p))) and zeros everywhere else.
In the Navier-Stokes example, the state vector 2 E 2222 contains the horizontal and vertical com- ponents of velocity u at collocation points, and the measurements ponents of u at equally-spaced sensors. Let us introduce the indices ..,,..., J2p of the entries in corresponding to the measurements yıs. Then, Yı and can be related by y Czz. where the matrix C' E R2px" contains ones at the entries indexed ((1,1)..... (2p, }2p) and zeros ev- erywhere else.
2 consist of the com-
F ADDITIONAL RESULTS FOR THE BURGERS EQUATION
Figure T0 shows spatio-temporal contours of the reference solution and corresponding estimates, for p and 12. The RI-ROOE vastly outperforms the KF-ROE for p = 2, while for p = 12 the KF-ROE is slightly more accurate.
G MODES FOR FLOW PAST CYLINDER
The first 18 modes (i.e. columns of U) of the 20-dimensional ROM for the flow past cylinder are displayed in Figure
H EFFECT OF OBSERVATION NOISE FOR THE NAVIER-STOKES EQUATIONS
In this appendix, we evaluate the estimation accuracy of the RL-ROE in the presence of non-zero observation noise n& polluting the sensor measurements yt in ( Specifically, we consider that ng is Gaussian white noise standard deviation 0.1. For the case of p 3 sensors, Figure 12 shows the time series of the noise-free measurements contained in yı: for various values of Re, together with their polluted counterpart (recall each sensor measures two components of velocity, as detailed in Appendix E Using the noisy measurements, we then repeat the same experiments carried out the main text; the results are shown in Figures [3 and 14 which are the counterparts of Figures 5 and 6 We observe excellent robustness of the RL-ROE in the presence of noise, with the estimate maintaining high accuracy.
I NONLINEARITY OF THE TRAINED POLICY
In this appendix, we evaluate the degree of nonlinearity of the RL-trained policies wo obtained in our Burgers and Navier-Stokes examples. Since the stochasticity of me is switched off during online deployment, He can be expressed by its mean function per (see Appendix so that the action in 3b becomes
a2 = #4 (yk+ #b-1).
(33)
We therefore quantify the nonlinearity of the policy by evaluating the Jacobian of the function Ho with respect to its two arguments and - The Jacobian is a matrix whose components are the first-order derivatives 0p:/0y; and 0u,,/Oit where (i, refers to the indices of the vectors entries in μ and y/ or - respectively. Instead of looking at individual components, we consider the Frobenius norm of the Jacobian, defined
ipl Oly x)
Op Og:
2
Öjt ae
+
(34)
The Jacobian (and its norm) of linear policy will be independent of the input values, while the Jacobian (and its norm) of a nonlinear policy will change with the input values. distribution of the norm of the Jacobian of the trained policies obtained in the Burgers and Navier- Stokes examples for various values of p. The distributions are obtained by calculating the Jacobian along solution trajectory of the RL-ROE corresponding the results shown in Sections 4. and 4.2 The wide spread of the distributions demonstrates that the mean function p4 trained by the RL process is highly nonlinear, opposed to the linear correction term in the KF-ROF.
the
16

---

Under review as a conference paper at ICLR 2023
J
reference
μ
24
WMM
2.0
WMWM WWWM
0.75
18 12
MMM
0.25
-1.2 -1.6
-0.5
1.0
0.00
-1.5
2.4
1.8
0.75
12
0.8
0.6
0.0
0.0
0.6
-0.4
0.25
1.2
0.8
0.00
-2.4
-1.8
-1.2
μ
0.95,
0.75
5pace
0.0
0.5
0.3
1.0
-0.6
0.00
-1.
-0.9
10
2
10
6
10
time
time
(a) 2
J 0.05, reference
11 0.45,
1-00455rcferenee reference
Jr 0.95, reference
2.0
1.5
0.50
0.5
122
0.0
-0.6
0.0
1.0
0.5
-05
0.0
0.3
0.3
-1.0
μ- 0.05, RL-RDE
μ. 0.45, RL-RDE
# 0.95, RL-ROE
WW
2.4
M
2.0
W
1.6
1.0
0.50
0.5
0.0
0.25
0.8
-0.5
-1.6
# 0.05, KF-ROE
0.45, KF-ROE
0.95, KF-ROE
3.2
2.0
1.5
1.0
0.50
0.5
0.0
0
0.25
time
time
time
(b) p 12
Figure 10: Burgers equation with (a) p= 2 and (b) 12 sensors. Reference trajectories for values of μ not seen during training and corresponding RL-ROE and KF-ROE estimates. The dashed lines on the reference trajectory plots indicate the sensor data seen by the RL-ROE and KF-ROE.
17

---

Under review as a conference paper at ICLR 2023
Mode 0
Mode 1
Mode 2
0.0096
0.035
0.035
0.0084
0.030
0.030
0.0072
0.025
0.025
0.0060
0.0008
0.0036
0.015
0.015
0.0024
0.010
0.010
0.0012
0.0000
0.000
0.000
Mode 3
Mode 4
Mode 5
0.028
0.035
0.024
0.030
0.036
0.025
0.030
0.020
0.011
0.015
0.018
0.008
0.012
0.005
0.000
0.000
0,000
Mode
Mode 7
Mode 8
0.036
0.036
0.042
0.036
0.030
-0.030
0.030
0.0024
0.018
0.018
0.018
0.012
0.012
0.006
0.000
0.000
0.000
Mode 9
Mode 10
Mode 11
0.040
0.064
0.064
0.056
0.025
0.040
0.040
0.020
0.032
0.010
0.016
0.016
0.005
0.008
0.008
0.000
0.000
0.000
Mode 12
Mode 13
Mode 14
0.036
0.030
0.030
0.025
0.020
0.018
0.015
0.010
0.012
0.010
0.006
Mode 15
Mode 16
Mode 17
0.030
0.036
0.025
0.030
0.020
Oc
0.024
0.015
0.015
0.018
0.010
0.010
0.005
0.005
0.000
0.000
0,000
Figure 11: First 18 modes of the 20-dimensional ROM for the flow past a cylinder.
18

---

Under review as a conference paper at ICLR 2023
Ro =
35, a=0
Re = 65, d
Re 105, o
1.0
messirementt
0.4
messinemennt
measurements
0.5
0.2
VL
V2
0.0
0.0
-0.5
-0.5
measurement
0.5
mensurements
measurements
1.0
0.5
0.0
0.0
0.0
-0.5
10
20
30
40
50
10
20
30
50
10
30
40
50
time
time
time
Figure 12: Navier-Stokes equations with sensors and observation noise of std Ø. Time series of measurements for different values of Re for a (top row) and o 0.1 (bottom row).
RL-ROE KF-ROE
normalized eтror
1.00
1.00
lower bound
L,
normalizdd
0.50
0.50
0.50
0.25
10
20
40
50
10
20
40
50
10
20
30
40
50
Figure 13: Navier-Stokes equations with p 3 sensors and observation noise of std 0.1. Normalized Le eгror of the RL-ROE and KF-ROE for the estimation of trajectories corresponding values of Re not seen during training.
J DISTRIBUTION OF PROCESS NOISE
We evaluate empirically the distribution the process noise wy in the ROM dynamics 2a First, note that the vector wy can be evaluated wt - 0k where 2k-1 U Zk-1 and a UT z are the reduced-order projections of two consecutive states zk-1 and 2 solving the high- dimensional dynamies [ For given ROM, we can then sample the process noise by evaluating values of along trajectories of the high-dimensiona dynamics ( We show in Figure [6 the distributions of the components of wt for the Burgers and Navier-Stokes examples, using the ROM constructed in Sections 4.1 and 4.2 The sampling is done along trajectories of ( corresponding different parameter values (the parameter being μ for Burgers, Re for Navier-Stokes), and the corresponding distributions of process noise are shown separately for each parameter value. (Note that the same ROM shared across all parameter values.) The distributions reveal that the process noise is non- Gaus-aa but approximately zero-mean.
19

---

Under review as a conference paper at ICLR 2023
Ro 35, reference
65, rafarance
Re 105, reference
1.2
1.0
1.0
0.48
0.64
0.6
0.8
0.32
0.4
0.16
0.00
0.0
0.0
Re 35, RL-ROE
65, ALADE
Re 105, RL-ROE
1.12
0.96
0.96
0.80
0.80
0.80
0.64
0.64
0.64 5
0.48
0.48
0.44
0.32
0:32
0.32
0.16
0.16
0.16
0.00
0.00
0.00
35, KF-ROE
105,
0.80
0.64
0.8
0.6
0.32
0.16
Figure 14: Navier-Stokes equations with p 3 sensors and observation noise of std o = 0.1. Ve- locity magnitude at 50 of the reference trajectories for values of Re seen during training and corresponding RL-ROE and KF-ROE estimates. The black crosses in the contours of the reference solutions indicate the sensor locations.
Distribution |laplllly, R||l-
Distribution [|ap/aty.Ri)
15
10
10
14
11
13
Burgers equation
(b) Navier-Stokes equations
Figure 15: Nonlinearity of the trained policies obtained in the (a) Burgers and (b) Navier-Stokes examples for various p. Distribution of the Frobenius norm of the Jacobian of the trained mean policy por, sampled along solution trajectories of the RL-ROF.
20

---

Under review as a conference paper at ICLR 2023
5 O, distribution of
02, distribution w
values
values
1.0
0.5
values
0.0
0.0
-0.5
-0.5
-1.0
distribution of
1. distribution of
1.0
1.
0.5
values
0.0
-0.
0.5
1.0
-1.0
7
10
7
10
(a) Burgers equation
Re 15, distribution of
Re 35, distribution of
0.02
values
0.00
-0.02
0.05
-0.04
Re 55, distribution
Re 75, distribution
0.02
values
values
0.01
0.00
0.00
0.01
85,
105,
0.04
0.02
values
valuees
4
0.00
-0.02
-0.02
1011 1313145555 161551117555
181920
indexi
indexi
(b) Navier-Stokes equations
Distribution of process noise in the (a) Burgers and (b) Navier-Stokes examples, sam- pled along solution trajectories of the high-dimensional dynamics ( The distributions of each component of wg obtained for each parameter value are shown individually.
Figure 16:
21


Reviews:
[{"review_id": "zMFfeOdmEai", "reviewer": "Reviewer_jaEk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The objective of the paper is to construct an estimator for the state of a high-dimensional nonlinear dynamical system given partial observations of the state. The objective is motivated by applications in fluid mechanics or turbulent flows where the state of the system is large obtained by discretizing a PDE. The proposed approach has two main steps:\n\n(1) construction of a reduced order model. In particular, the paper proposes the dynamic mode decomposition method which only requires a single trajectory of the nonlinear dynamics.\n\n(2) formulating the problem of finding the estimator as a MDP problem and application of RL techniques to solve it. In particular, the estimator is modeled as a dynamical system driven by a stochastic control policy that depends on the current value of the estimate and the value of observation. The objective function is modeled with running cost equal to the error in estimating the state and a quadratic penalty on the control. Then, the optimal control policy is learned using a policy gradient method by sampling trajectories from the system.\n\nThe proposed approach is evaluated on a benchmark example that involves Burger's equation and compared with Kalman filter applied on the reduced order system.", "summary_of_the_review": "The paper discusses a very interesting problem and follows a general exciting approach. It should give more motivation for the cost function, and also more in depth comparison to KF, like comparing the learned policy to the best linear policy. Basically, it is interesting to see what would happened if the control action is restricted to be linear and what is the gain from nonlinear parametrization.    ", "strength_and_weaknesses": "Strength\n- Interesting and important problem \n- The general solution approach, using RL for estimation, is definitely exciting\n\nWeakness: \n- No discussion on the meaningfulness of the designed estimator. What does the cost in the optimal control represent? Why does minimizing it lead to a good estimator? \n- The comparison with KF-ROE can me made more in depth. Are the control laws learned through the RL approach nonlinear? What is the tuning process for KF? \n- the inability to consider estimating systems with inputs. \n- the discussion about stationary Markov decision problem and the remark about the problem being actually POMDP and non-Markov is a bit confusing. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WFemYf53P2", "reviewer": "Reviewer_AsBn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers state space estimation by optimal filtering for latent partial differential equations (PDE). For this, a reinforcement-learning-based estimator is proposed. \n\nThe paper considers a given space-time discretization method of the PDE in question, resulting in a high-dimensional system of time-discretized nonlinear ordinary differential equations (ODEs). The resulting high-dimensional system is reduced in dimensionality using a linear dimensionality reduction method, dynamic mode decomposition (DMD). The state estimation problem is therefore transferred to the low-dimensional linear space. \nFor solving the resulting optimal filtering problem an estimator for the posterior mean of the Bayesian filtering problem is proposed. The authors consider for this an estimator which includes the low-dimensional linear dynamics and an additive nonlinear correction term (innovation process) based on the measurements.\nThe optimal filter is learned using a reinforcement learning (RL) algorithm, proximal policy optimization (PPO). The target of the RL algorithm is a reward parametrized by the negative squared estimation error and a penalty for the innovation gain.\nThe final algorithm is evaluated for two synthetic PDE filtering problems.", "summary_of_the_review": "The paper is well-written and the setup is interesting. The principle behind the estimation algorithm presented is known. The biggest downside of this paper is, that the algorithm is developed heuristically and consequently a comparison to other nonlinear state space estimators is missing. The experiments considered are interesting and the evaluation of the considered setup is fair.\n\nAltogether I come to the conclusion that this paper is between the ratings of 3 (reject, not good enough) and 5 (marginally below the acceptance threshold).\n\nSome questions I had for the authors:\n- What is the motivation behind the considered estimator in equation (3), why is not an estimator considered, which is fully parameterized by a neural network, i.e., $\\hat{x}_k=a_k$  \n- Is this modeling of one linear dynamical system efficient, did you consider using a switching dynamical system as a model, e.g., [5]?\n- Could this setup be extended to nonlinear dimensionality reduction methods, if anyway later a nonlinear estimator is used?\n- The estimator in equation (3) assumes a zero mean noise process $w_k$. How is the empirical distribution for $w_k$ looking, is it zero mean?\n- In equation (7), the hyper-parameter $\\lambda_2$ should be dependent on the observation noise, i.e., if the observation noise is high $\\lambda_2$ should be chosen large, as this would diminish the gain on the innovation term ($a_k$) in equation (3). How was the observation noise chosen in the experiments? Can you see an influence of the two hyper-parameters, noise variance, and $\\lambda_2$?\n\n[5] Linderman, Scott, et al. \"Bayesian learning and inference in recurrent switching linear dynamical systems.\" Artificial Intelligence and Statistics. PMLR, 2017.", "strength_and_weaknesses": "The paper discusses in my opinion a very timely topic, state estimation for PDEs. The setup as such is interesting and has probably many application scenarios. The use of a dimensionality reduction method is a clever way to make the state estimation problem tractable. My biggest concern is, however,  that the algorithmic development is very heuristic. The paper misses the principled and \"modern viewpoint\" of Bayesian inference to model the state estimation problem, see, e.g., [1] for an introduction, and [2,3] for mathematical treatments. \nHence, the proposed mean estimator in equation (3) is arbitrarily chosen, which resembles the usual structure of the Kushner-Startonovich equation, including the prior dynamics and the innovation process. \nFor me, it makes sense that the reinforcement learning-based estimator outperforms the linear Kalman filter, as the estimator is based on a Gaussian error assumption. However, what is missing in such a case is a comparison, to, e.g., a nonlinear particle filter, which probably will easily outperform the proposed RL algorithm.\nAnother problematic aspect is that the controller only has access to the new observation and the posterior mean estimate. Even when assuming the linear prior dynamics, the innovation process is still dependent on the whole posterior distribution at the previous time step. This is justified in the remark saying that the history $h_k$ represented by $o_k$ is an incomplete summary and therefore sufficient. I would highly object that this is enough for state estimation since even in the linear quadratic case the optimal estimator is dependent on the variance of the posterior distribution (solution to the Ricatti equation).\nOtherwise, I think the experiments are fair and convincing for the considered setup. \n\nAs an aside, which is probably out of scope for this work, I would argue that a problematic aspect of the setup is that a time discretization is carried out before the inference. This results in an easier-to-model discrete-time system, however, a continuous-time description is usually much more sensible in these scenarios, see, e.g., [4] for an accessible introduction and [2] for a mathematical treatment. As oftentimes the time-discretization as chosen by the Runge-Kutta integrator is selected arbitrarily to model the prior dynamics, however, the posterior dynamics often require a different time-discretization. Maybe for future work, the authors can consider this setup.\n\nSummary:\n\nStrong points:\n- The submission is timely and the setup is very interesting.\n- The submission is fairly well-written and technically correct.\n\n\nWeak points:\n- algorithmic development is heuristic\n- Bayesian description is missing\n- estimator is chosen arbitrarily and not optimal\n- a simple comparison to a particle filter algorithm is missing\n\n\n[1] Särkkä, Simo. Bayesian filtering and smoothing. No. 3. Cambridge university press, 2013.\n[2] Elliott, Robert J., Lakhdar Aggoun, and John B. Moore. Hidden Markov models: estimation and control. Vol. 29. Springer Science & Business Media, 2008.\n[3] Bain, Alan, and Dan Crisan. Fundamentals of stochastic filtering. Vol. 3. New York: Springer, 2009.\n[4] Särkkä, Simo, and Arno Solin. Applied stochastic differential equations. Vol. 10. Cambridge University Press, 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qdbxau8qcC", "reviewer": "Reviewer_TCZP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to infer the state of a dynamical system based on sparse measurements. More specifically, the authors consider a situation in which there is a dynamical system whose state evolves but we cannot observe it directly. Instead we observe only a low dimensional approximation.. Here, the authors propose a method which combines a reduced order model (ROM) approach with reinforcement learning (RL). Then they provide examples of applications to partial differential equations (PDEs), with numerical results.", "summary_of_the_review": "The paper is clearly written and the contributions are clearly presented. I think that addressing the two questions mentioned above would help strengthen the paper.", "strength_and_weaknesses": "Strength: the paper is clearly written and the ideas are well presented.\n\nWeaknesses: (1) it seems that the framework and the model purely deal with a discrete time and finite dimensional system. The connection with PDEs (mentioned in the title) is not clear until section 4, and even there, it seems to me more like an application. This is not necessarily a drawback, since it probably shows that the method is quite general. However, I am wondering whether the connection with PDEs should be strengthened. (2) Furthermore, the main contributions are numerical but the only baseline comes from a Kalman filter approach, which is suitable for linear systems but I am not sure if it is suitable for PDEs. I would recommend discussing this choice of baseline and, if possible, to provide other baselines.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GJhf3Eu6U8", "reviewer": "Reviewer_rh8q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "this paper proposes to estimate ROM from samples by formulating the correction term to be a non-linear policy, which is trained by RL. The problem formulation requires a stationary MDP model, which is constructed by augmenting the state with the original $z_k$ and $\\mu$.  ", "summary_of_the_review": "well-written paper. the background and limitations are well explained. numerical simulations confirm the claims. ", "strength_and_weaknesses": "Strengths: interesting idea, justified approach, good simulations, well written paper.\n\nWeaknesses/Limitations are reasonably addressed by the authors. \n\na few comments:\n1. Eq7. lhs depend on t+1, while rhs doesn't not depend on $t+1$\n\n2. in NAVIER-STOKES EQUATION: the dimensionality of state (the discretized solution) is huge. could you elaborate on the sensors in this case? does the sensor matrix, C, include only 1 non zero entries?  I suggest to explicitly state the dimensions of the sensor in the experiments. \n\n3. could you elaborate on \"However, this approach would not work for our purpose since the RL-ROE cannot be restricted to estimating a single or a small finite number of reference trajectories. Instead, by augmenting the state with entire snapshots from the reference trajectory instead of just a phase variable, we ensure that there is no theoretical limit to the number of reference trajectories that the policy πθ can learn\"? I don't understand why including the snapshot to the state allows the policy to learn any number of trajectories. The state is augmented with $\\mu$, which I thought, is enough for the generalization to unseen trajectories. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject