Year: 2021

Abstract:
In high-dimensional nonlinear systems such as fluid flows, the design of state estimators such as Kalman filters relies on a reduced-order model (ROM) of the dynamics. However, ROMs are prone to large errors, which negatively affects the performance of the estimator. Here, we introduce the reinforcement learning reduced-order estimator (RL-ROE), a ROM-based estimator in which the data assimilation feedback term is given by a nonlinear stochastic policy trained through reinforcement learning. The flexibility of the nonlinear policy enables the RLROE to compensate for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. We show that the trained RL-ROE is able to outperform a Kalman filter designed using the same ROM, and displays robust estimation performance with respect to different reference trajectories and initial state estimates.

text: In high-dimensional nonlinear systems such as fluid flows, the design of state estimators such as Kalman filters relies on a reduced-order model (ROM) of the dynamics. However, ROMs are prone to large errors, which negatively affects the performance of the estimator. Here, we introduce the reinforcement learning reduced-order estimator (RL-ROE), a ROM-based estimator in which the data assimilation feedback term is given by a nonlinear stochastic policy trained through reinforcement learning. The flexibility of the nonlinear policy enables the RLROE to compensate for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. We show that the trained RL-ROE is able to outperform a Kalman filter designed using the same ROM, and displays robust estimation performance with respect to different reference trajectories and initial state estimates.

heading: 1 INTRODUCTION
text: Active control of turbulent flows has the potential to cut down emissions across a range of industries through drag reduction in aircrafts and ships or improved efficiency of heating and air-conditioning systems, among many other examples (Brunton & Noack, 2015). But real-time feedback control requires inferring the state of the system from sparse measurements using an algorithm called a state estimator, which typically relies on a model for the underlying dynamics (Simon, 2006). Among state estimators, the Kalman filter is by far the most well-known thanks to its optimality for linear systems, which has led to its widespread use in numerous applications (Kalman, 1960; Zarchan & Musoff, 2015). However, continuous systems such as fluid flows are governed by partial differential equations (PDEs) which, when discretized, yield high-dimensional and oftentimes nonlinear dynamical models with hundreds or thousands of state variables. These high-dimensional models are too expensive to integrate with common state estimation techniques, including the Kalman filter or its numerous extensions. Thus, state estimators are instead designed based on a reduced-order model (ROM) of the underlying dynamics (Barbagallo et al., 2009; Rowley & Dawson, 2017).
A big challenge is that ROMs provide a simplified and imperfect description of the dynamics, which negatively affects the performance of the state estimator. One potential solution is to improve the accuracy of the ROM itself through the inclusion of additional closure terms (Ahmed et al., 2021). In this paper, we leave the ROM untouched and instead propose a new design paradigm for the estimator itself, which we call a reinforcement-learning reduced-order estimator (RL-ROE). The RL-ROE is constructed from the ROM in an analogous way to a Kalman filter, with the crucial difference that the linear filter gain function is replaced by a nonlinear stochastic policy trained through reinforcement learning (RL). The flexibility of the nonlinear policy enables the RL-ROE to compensate for errors of the ROM, while still taking advantage of the imperfect knowledge of the dynamics. We describe how we frame the problem as a stationary Markov decision process in order to enable RL training, which is non-trivial since the RL-ROE must be able to estimate time-varying states. Finally, we show that the trained RL-ROE is able to outperform a Kalman filter designed using the same ROM, and displays robust estimation performance with respect to different reference trajectories and initial state estimates. The RL-ROE is the first application of reinforcement learning to state estimation for high-dimensional systems.
Under review as a conference paper at ICLR 2022

heading: 2 PROBLEM FORMULATION

heading: 2.1 SETUP
text: Consider the discrete-time nonlinear system given by
zk+1 = f(zk), (1a) yk = Czk, (1b)
where zk ∈ Rn and yk ∈ Rp are respectively the state and measurement at time k, f : Rn → Rn is a time-invariant nonlinear map from current to next state, and C ∈ Rp×n is a linear map from state to measurement. In this study, we assume that the dynamics given in (1) are obtained from the numerical discretization of a nonlinear partial differential equation (PDE), which typically requires a large number n of state dimensions. Note that we do not account for exogenous control inputs to the system, which will be studied in future extensions of the present work.

heading: 2.2 REDUCED-ORDER MODEL
text: Because the high dimensionality of (1) makes online prediction and control impractical, it is instead customary to formulate a reduced-order model (ROM) of the dynamics (Rowley & Dawson, 2017). First, one chooses a suitable linearly independent set of modes {u1, . . . ,ur}, where ui ∈ Rn, defining an r-dimensional subspace of Rn in which most of the dynamics is assumed to take place. Stacking these modes as columns of a matrix U ∈ Rn×r, one can then express zk ' Uxk, where the reduced-order state xk ∈ Rr represents the coordinates of zk in the subspace. Finally, one finds a ROM for the dynamics of xk, which is vastly cheaper to evolve than (1) when r n. There exist various ways to find an appropriate set of modes U and corresponding ROM for the dynamics of xk (Taira et al., 2017). In this work, we employ the Dynamic Mode Decomposition (DMD), a purely data-driven algorithm that has found wide applications in fields ranging from fluid dynamics to neuroscience (Schmid, 2010; Kutz et al., 2016). Starting with a collection of snapshots Z = {z0, . . . ,zm} collected along a trajectory of (1a), the DMD seeks a best-fit linear model of the dynamics in the form of a matrixA ∈ Rn×n such that zk+1 ' Azk, and computes the modes U as the r leading principal component analysis (PCA) modes of Z. The transformation zk ' Uxk and the orthogonality of U then yield a linear discrete-time ROM of the form
xk+1 = Arxk +wk, (2a) yk = Crxk + vk, (2b)
whereAr = UTAU ∈ Rr×r andCr = CU ∈ Rp×r are the reduced-order state-transition and observation models, respectively. In order to account for the neglected PCA modes of Z as well as the unmodeled dynamics incurred by the linear approximation zk+1 ' Azk, we add (unknown) nonGaussian process noise wk and observation noise vk. Additional details regarding the calculation ofAr and U are provided in Appendix A.

heading: 2.3 REDUCED-ORDER ESTIMATOR
text: This paper uses reinforcement learning (RL) to solve the following estimation problem: given a sequence of measurements {y0, · · · ,yk} from a reference trajectory {z0, · · · , zk} of (1) and knowing the ROM (2) defined byAr,Cr andU , we want to estimate the high-dimensional state zk at current time k. To this effect, we design a reduced-order estimator (ROE) of the form
x̂k = Arx̂k−1 + ak, (3a) ak ∼ πθ( · |yk, x̂k−1), (3b)
where x̂k is an estimate of the reduced-order state xk, and ak ∈ Rr is an action sampled from a stochastic policy πθ which depends on the current measurement yk and the previous state estimate x̂k−1. The subscript θ denotes the set of parameters that defines the stochastic policy, whose goal is to minimize the mean square error E[zk − ẑk] over a range of reference trajectories and initial reduced-order state estimates. Here, ẑk = Ux̂k denotes the high-dimensional state estimate reconstructed from x̂k. A Kalman filter is a special case of such an estimator, for which the action in (3b) is given by
ak = Kk(yk −CrArx̂k−1), (4)
Under review as a conference paper at ICLR 2022
with Kk ∈ Rr×p the optimal Kalman gain. Although the Kalman filter is optimal when the statetransition and observation models are known exactly, its performance suffers in the presence of unmodeled dynamics. In our case, such model errors are unavoidable due to the ROM (2) being an inherent approximation of the high-dimensional dynamics (1), which motivates our adoption of the more general form (3b). This form retains the dependence of ak on yk and x̂k−1 but is more flexible thanks to the nonlinearity of the stochastic policy πθ, which we train with deep RL in an offline stage. The stochasticity of πθ forces the RL algorithm to explore different actions during the training process, in order to find eventually an optimal θ∗ such that E[zk − ẑk] is minimized for various reference trajectories and initial estimates. We call the estimator constructed and trained through this process an RL-trained ROE, or RL-ROE for short.
Thus, the methodology we propose consists of two steps. In a first offline stage, a ROM of the form (2) is obtained using high-dimensional snapshots zk from a single trajectory of (1). The RL-ROE (3) is then constructed based on this ROM, and its policyπθ is trained using high-dimensional snapshots zk from multiple reference trajectories of (1). Finally, the trained RL-ROE is deployed online to track a reference trajectory of (1). In the online stage, the RL-ROE only requires measurements yk from the reference trajectory, and gives an estimate ẑk = Ux̂k for the high-dimensional state.
In summary, our contributions in this paper are two-fold:
1. We propose RL-ROE, a reduced-order state estimator for high-dimensional nonlinear systems. The RL-ROE takes the form (3), which combines two unique features: first, the state transition modelAr is a ROM of the high-dimensional dynamics; second, the term ak that assimilates measurements is sampled from a stochastic policy πθ trained with RL. The training procedure for πθ, which involves a non-trivial reformulation of the time-varying tracking problem as a stationary Markov decision process, is described in Section 4.
2. The performance of the RL-ROE is compared in Section 5 with that of KF-ROE, a Kalman filter constructed from the same ROM. The comparison is performed in the context of the Burgers equation using a range of reference trajectories and initial state estimates.

heading: 3 RELATED WORK
text: Previous studies have already proposed designing state estimators using policies trained through reinforcement learning. Morimoto & Doya (2007) introduced an estimator of the form x̂k = f(x̂k−1) + L(x̂k−1)(yk−1 −Cx̂k−1), where f(·) is the state-transition model of the system, and the state-dependent filter gain matrix L(x̂k−1) is defined using Gaussian basis functions whose parameters are learned through a variant of vanilla policy gradient. Their reward function, however, was calculated using the measurement error instead of the state estimate error, potentially limiting the performance of the trained estimator. Hu et al. (2020) proposed an estimator of the form x̂k = f(x̂k−1) + L(xk − x̂k)(yk − Cf(x̂k−1)), where L(xk − x̂k) is approximated by neural networks trained with a modified Soft-Actor Critic algorithm (Haarnoja et al., 2018). Although they derived convergence properties for the estimate error, the dependence of the filter gain L(xk − x̂k) on the reference state xk limits its practical application.
A major difference between these past studies and our work is that they do not construct a ROM of the dynamics and only consider low-dimensional systems with four state variables at most, in comparison with the hundred or more state dimensions that our RL-ROE can handle. Therefore, RL-ROE represents the first application of reinforcement learning to state estimation for high-dimensional systems, which makes it applicable to systems governed by PDEs such as fluid flows.

heading: 4 TRAINING METHODOLOGY
text: In this section, we describe the offline training process for the policy πθ in the RL-ROE (3). In order to train πθ with reinforcement learning, we need to formulate the problem as a stationary Markov decision process (MDP). However, this is no trivial task given that the aim of the policy is to minimize the error between the state estimate ẑk = Ux̂k and a time-dependent reference state zk. At first sight, such trajectory tracking problem requires a time-dependent reward function and, therefore, a time-varying MDP.
Under review as a conference paper at ICLR 2022
To be able to use off-the-shelf RL algorithms, we introduce a trick to translate this time-varying MDP to an equivalent, extended stationary MDP. Indeed, we show hereafter that the problem can be framed as a stationary MDP by including zk into our definition of the MDP’s state. Letting sk = (zk, x̂k−1) ∈ Rn+r denote an augmented state at time k, we can define an MDP consisting of the tuple (S,A,P,R), where S = Rn+r is the augmented state space, A ⊂ Rr is the action space, P(·|sk,ak) is a transition probability, and R(sk,ak, sk+1) is a reward function. At each time step k, the agent selects an action ak ∈ A according to the policy πθ defined in (3b), which can be expressed as ak ∼ πθ( · |ok), (5) where ok = (yk, x̂k−1) = (Czk, x̂k−1) is a partial observation of the current state sk. The state sk+1 = (zk+1, x̂k) at the next time step is then obtained from equations (1a) and (3a) as
sk+1 = (f(zk),Arx̂k−1 + ak), (6)
which defines the transition model sk+1 ∼ P(·|sk,ak). Finally, the agent receives the reward
rk = R(sk,ak, sk+1) = −(zk −Ux̂k)TQ(zk −Ux̂k)− aTkRak, (7) where Q ∈ Rn×n and R ∈ Rr×r are positive semidefinite and positive definite matrices, respectively. The first term in the reward functionR penalizes the difference between the high-dimensional state estimate ẑk = Ux̂k and the reference zk, which is only partially observed by the agent. The second term favors smaller values for the action ak; such regularization leads to more robust estimation performance in the presence of noise during online deployment of the RL-ROE, as we will see later. Unless indicated otherwise, we will consider Q = I and R = I . Thanks to the incorporation of zk into sk, the reward function (7) has no explicit time dependence and the MDP is therefore stationary.
The goal of the RL training process is then to find the optimal policy parameters
θ∗ = arg max θ E τ∼πθ [R(τ)], (8)
where the expectation is over trajectories τ = (s1,a1, s2,a2, . . . ), and R(τ) is the finite-horizon undiscounted return
R(τ) = K∑ k=1 rk, (9)
with the integer K denoting the length of each training trajectory. Contrary to conventional RL notation, trajectories here start at time k = 1. Indeed, the environment is initialized at time k = 0 according to the distributions
z0 ∼ pz0(·), (10a) x̂0 ∼ px̂0(·), (10b)
from which the augmented state s1 = (z1, x̂0) = (f(z0), x̂0) follows immediately. Thus, s1 constitutes the start of the trajectory of agent-environment interactions. This sequence of operations mirrors that of a Kalman filter: the calculation of the current state estimate uses the previous state estimate as well as the current measurement, and therefore begins from the second time step, which is k = 1 in our case.
To find the optimal policy parameters θ∗, we employ the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017), which belongs to the class of policy gradient methods (Sutton et al., 2000). PPO alternates between sampling data by computing a set of trajectories {τ1, τ2, τ3, . . . } using the most recent version of the policy, and updating the policy parameters θ in a way that increases the probability of actions that led to higher rewards during the sampling phase. The policy πθ encodes a diagonal Gaussian distribution described by a neural network that maps from observation to mean action, µθ′(ok), together with a vector of standard deviations σ, so that θ = {θ′,σ}. We utilize the Stable Baselines3 (SB3) implementation of PPO (Raffin et al., 2019) and define our MDP as a custom environment in OpenAI Gym (Brockman et al., 2016). Besides the discount factor γ, all results to follow are obtained with the default PPO hyperparameters in SB3, which demonstrates the robustness of our approach with respect to the RL hyperparameters.
Remark 1. The offline RL training phase described in this section (to find the optimal parameters θ∗) requires knowledge of the high-dimensional state zk from several reference trajectories of (1).
Under review as a conference paper at ICLR 2022
During online deployment, however, the RL-ROE (3) only relies on measurements yk, since the trained policy πθ∗ is conditioned on yk and the previous reduced state estimate x̂k−1.
Remark 2. Similar to any learning method, the trained RL-ROE is expected to perform well for reference trajectories and initial estimates sampled from the same distributions used during the offline training phase. Thus, the choice of the distributions for z0 and x̂0 in (10) is critical since it defines the range of initial reference states and initial estimates that the trained RL-ROE will be able to handle during online deployment. This is similar to the state of the art in model-based observer theory, which only guarantees convergence of nonlinear observers locally, i.e., near the true initial states (Benosman & Borggaard, 2021).

heading: 5 RESULTS
text: We evaluate our proposed RL-ROE using simulations of the Burgers equation, a prototypical nonlinear hyperbolic PDE which takes the form
∂u ∂t + u ∂u ∂x − ν ∂
2u
∂x2 = f(x, t), (11)
where u(x, t) is the velocity at position x ∈ [0, L] and time t ∈ [0, T ], f(x, t) is a distributed time-dependent forcing, and the scalar parameter ν acts like a viscosity. The boundary conditions are periodic and the initial condition u(x, 0) = u0(x) will be specified later. We choose L = 1, T = 10, and ν = 0.01.
We discretize the Burgers equation (11) with a spectral method using n = 256 Fourier modes, and integrate forward in time using a fifth-order Runge-Kutta method. Solution snapshots are saved every ∆t = 0.05 time unit, yielding the discrete high-dimensional state
zk = [u(x0, k∆t), u(x1, k∆t), . . . , u(xn−1, k∆t)] T ∈ Rn, (12)
where xi = iL/n are the collocation points, and k = 0, . . . , T/∆t is the time index. Further, we place p = 8 sensors at equidistant locations, providing the sparse measurement vector
yk = [u(x̄0, k∆t), u(x̄1, k∆t), . . . , u(x̄p−1, k∆t)] T ∈ Rp, (13)
where x̄i = iL/p are the sensor locations. Thus, the state zk and measurement yk are governed by a high-dimensional discrete-time nonlinear system of the form given in equation (1).
We then follow the procedure outlined in Section 2 to construct offline a ROM and corresponding ROE, which we train offline using PPO following the methodology presented in Section 4. Finally, the trained RL-ROE is deployed online and compared against a Kalman filter under various initial conditions for the reference trajectory as well as the state estimate. Specifically, we adopt the following steps:
1. Construction of the ROM. Starting from the pulse-shaped initial condition
u0(x) = 1
cosh(20(x− L/2)) , (14)
we calculate one solution trajectory of (1) for t ∈ [0, T/2] = [0, 5], and we denote as ZDMD = {zDMD0 , . . . ,zDMDm } the resulting solution snapshots at times k = 0, . . . ,m = T/2∆t. The DMD is then applied to these snapshots, yielding a ROM of the form (2) defined by matrices Ar, Cr and U . The ROM governs the evolution of a reduced-order state xk ' UTzk ∈ Rr. We pick r = 15 for the dimensionality of the reduced-order subspace (which corresponds in this case to 99.99% of the energy of the snapshots ZDMD being included in the modes U ), giving the ROM a significant computational advantage compared with the high-dimensional system (1) of size n = 256.
2. Training of the RL-ROE. We train the stochastic policy πθ of the RL-ROE (3) using PPO, as described in Section 4. In order for the resulting estimator to perform well under various reference trajectories and initial estimates, we initialize each trajectory of the MDP during the offline training process with
z0 = αz DMD 0 , (15a) x̂0 = U TzDMD0 + β, (15b)
Under review as a conference paper at ICLR 2022
(a) (b) (c)
(d) (e) (f)
where α ∼ U(0.5, 2) ∈ R and β ∼ N (0, 0.1I) ∈ Rr, which defines the distributions given in (10). In other words, the reference trajectories are initialized as a randomly scaled up or scaled down version of the pulse defined in (14), and the reduced-order state estimate is initialized as a reduced-order projection of that pulse, polluted with additive Gaussian noise. During training, we limit each trajectory to the same time window t ∈ [0, T/2] that was used in constructing the ROM – that is, we pick K = T/2∆t in the finite-horizon return (9). We end the training when the return no longer increases on average.
3. Evaluation of the RL-ROE. We evaluate the trained RL-ROE against a time-dependent Kalman filter constructed from the same ROM, which we refer to as KF-ROE. The KFROE is given by equations (3a) and (4), with the calculation of the time-varying Kalman gain detailed in Appendix B. The RL-ROE and KF-ROE are compared online based on three specific reference trajectories initialized from (15a) using α = 0.5, 1, and 2. For each reference trajectory, we consider 20 different initial state estimates sampled from (15b), and feed measurements yk to the RL-ROE and KF-ROE. Their tracking performance of the reference state zk is then evaluated and compared over the full time window t ∈ [0, T ].
We carry out the above procedure for two different choices of the forcing f(x, t), each leading to qualitatively different dynamics: first, the unforced case f(x, t) = 0, and second, a sinusoidal forcing of the form f(x, t) = sin(ωt − kx) with ω = π and k = 2π/L. We emphasize that the RL-ROE is not (yet) designed to account for different exogenous control inputs to the system. Here, the forcing f(x, t) is simply considered as a way to generate more complex solutions by altering the dynamics of the PDE. Thus, the two choices for f(x, t) will be treated separately in the results below, and we will construct and train a different RL-ROE in each case.

heading: 5.1 UNFORCED CASE
text: Beginning with the unforced case f(x, t) = 0, we first construct the ROM on which the RL-ROE and KF-ROE will be based. Figure 1 depicts the solution of the discretized Burgers equation to the initial condition (14). The solution snapshots belonging to the time window t ∈ [0, T/2] = [0, 5] constitute the training data ZDMD fed to the DMD algorithm, which yields the ROM (2). Further, this figure illustrates the type of dynamics exhibited by the Burgers equation in the unforced regime: a quickly decaying and widening pulse for t ≤ 2, followed by a slow convergence to a uniform steady state for t > 2.
Before training and evaluating the RL-ROE, we quantify the accuracy of the ROM itself – that is, given knowledge of the true initial condition. For this purpose, we consider three initial conditions defined by (15a) with α = 0.5, 1, 2, and we compute the corresponding reference solution (using the discretized Burgers equation) as well as the ROM solution. The results, reported in Appendix C, show that the model error is very low for the initial condition α = 1 and time window t ∈ [0, 5],
Under review as a conference paper at ICLR 2022
ROM training data
(a) (b)
since this case corresponds to the same solution trajectory used for constructing the ROM. On the other hand, the model error increases for larger times or other values of α.
The RL-ROE is then trained using PPO following the methodology outlined in Section 4, together with the distributions (15) for trajectory initialization. The RL hyperparameters and learning curve displaying the performance improvement of the RL-ROE during the training process are reported in Appendix D. The trained RL-ROE is now compared with the KF-ROE, a Kalman filter constructed from the same ROM. Figures 2(a,b,c) show the L2 error of the RL-ROE and KF-ROE with respect to specific reference trajectories of the Burgers equation, initialized from (15a) using α = 0.5, 1, 2. For each reference trajectory, we consider 20 different initial state estimates sampled from (15b). The curves and shaded area reported in Figures 2(a,b,c) indicate the mean and standard deviation of the error, defined at time step k by |Ux̂k−zk|, where x̂k is the reduced-order estimate given by the RL-ROE or KF-ROE, and zk is the high-dimensional reference solution. (Appendix E shows the same data for t ∈ [0, 2].) Figures 2(d,e,f) show the trajectories of the reference, RL-ROE and KFROE solutions in a two-dimensional slice of phase space spanned by the second and third columns1 of U – in other words, the time history of the second and third components of UTzk and x̂k.
A few important observations emerge from Figure 2. First, when the ROM suffers from large model errors due to initial conditions deviations, as is the case for α = 0.5 and α = 2, the RL-ROE is able to outperform the KF-ROE. In the time window t ≤ 2 during which most of the transient dynamics take place, the RL-ROE displays up to an order of magnitude lower error than the KF-ROE. Second, when the ROM is very accurate, as is the case for α = 1, the KF-ROE gives lower error for most of the time duration. Even then, however, Figure 2(e) shows that the RL-ROE converges faster to the reference trajectory. Last, the RL-ROE manages to keep the error at a low level in the time window t ∈ [5, 10], despite the fact that it was trained using trajectories that end at t = 5.
1Since the columns of U approximate the PCA modes of the training snapshots ZDMD without centering, the first column will be dominated by the mean of the data. Thus, we display the trajectory coordinates associated with the second and third columns, which capture the largest amount of variance within the data.
Under review as a conference paper at ICLR 2022
(a) (b) (c)
(d) (e) (f)
The Burgers equation converges to a uniform steady state in this unforced case, which can explain why both the RL-ROE and KF-ROE display low errors at larger times. We now investigate a fundamentally different dynamical regime in the next section by adding nonzero forcing to the Burgers equation, which prevents the solution from settling on a steady state.

heading: 5.2 FORCED CASE
text: We now consider the forced case f(x, t) = sin(ωt − kx), with ω = π and k = 2π/L. Repeating the steps of the previous section, we first construct the ROM on which the RL-ROE and KF-ROE will be based. Figure 3 depicts the solution of the discretized Burgers equation with forcing to the initial condition (14). The solution snapshots belonging to the time window t ∈ [0, T/2] = [0, 5] constitute the training data ZDMD fed to the DMD algorithm for constructing the ROM (2). This figure illustrates the type of dynamics exhibited by the Burgers equation in the forced regime: a transient phase with for t ≤ 2 during which the pulse changes shape into a skewed wave, followed by a limit cycle regime with the wave traveling at constant velocity for t > 2.
As in the unforced case, we first quantify the accuracy of the ROM itself. For this purpose, we consider three initial conditions defined by (15a) with α = 0.5, 1, 2, and we compute the corresponding reference solution (using the discretized Burgers equation with forcing) as well as the ROM solution. This time, the results reported in Appendix C show that the model error is higher for all cases, which reflects the more complicated nature of the dynamics in this forced regime.
The RL-ROE is then trained using PPO following the methodology outlined in Section 4, together with the distributions (15) for trajectory initialization. The RL hyperparameters and learning curve displaying the performance improvement of the RL-ROE during the training process are reported in Appendix D. The trained RL-ROE is now compared with the KF-ROE, a Kalman filter constructed from the same ROM. Figures 4(a,b,c) show the L2 error of the RL-ROE and KF-ROE with respect to specific reference trajectories of the Burgers equation, initialized from (15a) using α = 0.5, 1, 2. As before, we consider 20 different initial state estimates sampled from (15b) for each reference trajectory. The curves and the surrounding shade reported in Figures 4(a,b,c) indicate the mean and standard deviation of the resulting error, respectively. (Appendix E shows the same data for t ∈ [0, 2].) Figures 4(d,e,f) show the trajectories of the reference, RL-ROE and KF-ROE solutions in a two-dimensional slice of phase space spanned by the second and third columns of U – in other words, the time history of the second and third components of UTzk and x̂k.
The RL-ROE outperforms the KF-ROE in both the initial transient phase as well as the later limit cycle regime, for all three reference trajectories. This is consistent with our previous observation that the RL-ROE has an advantage over the KF-ROE when the ROM suffers from large model errors. Moreover, the estimation performance of the RL-ROE remains stable in the time window t ∈ [5, 10], even though trajectories stop at t = 5 during the training process. All together, these
Under review as a conference paper at ICLR 2022
ROM training data
(a) (b)
results demonstrate the estimation performance of the RL-ROE, and its robustness with respect to model errors.
We include a number of additional results as appendices. In Appendix F, we show that the regularization term in the reward function (7) leads to better estimation performance in the presence of observation noise. In Appendix G, we compare our RL-ROE approach with two alternative datadriven estimators formulated as standard supervised learning problems, and we observe that the RL-ROE is more robust to observation noise and gives smoother predictions.

heading: 6 CONCLUSIONS
text: In this paper, we have introduced the reinforcement learning reduced-order estimator (RL-ROE), a new methodology for estimating the state of a high-dimensional nonlinear dynamical system. Our approach follows the standard practice of constructing a computationally inexpensive reduced-order model (ROM) to approximate the dynamics of the system. The novelty of our contribution lies in the design, based on this ROM, of a reduced-order estimator (ROE) in which the feedback correction term is given by a nonlinear stochastic policy trained through reinforcement learning. To be able to use off-the-shelf RL algorithms, we introduce a trick to translate this trajectory tracking problem, i.e., time-varying MDP, to an equivalent stationary MDP based on an augmented state. We show using simulations of the Burgers equation in two very different dynamical regimes that the trained RL-ROE is able to outperform a Kalman filter designed using the same ROM and displays robust estimation performance with respect to different reference trajectories and initial state estimates.
This work opens the door to a number of potential future directions. A logical next step is to evaluate the performance of the RL-ROE on other physical systems, for instance the Navier-Stokes equations governing the motion of fluid flows. Different types of ROM could also be considered, potentially leading to improved performance of the RL-ROE.
Under review as a conference paper at ICLR 2022

heading: A DYNAMIC MODE DECOMPOSITION
text: In this appendix, we describe the DMD algorithm (Schmid, 2010; Tu et al., 2014), which is a popular data-driven method to extract spatial modes and low-dimensional dynamics from a dataset of highdimensional snapshots. Here, we use the DMD to construct a ROM of the form (2) given a snapshot sequence Z = {z0, . . . ,zm} collected along a trajectory of (1a) and an observation model C. Fundamentally, the DMD seeks a best-fit linear model of the dynamics in the form of a matrix A ∈ Rn×n such that zk+1 ' Azk. Arranging the snapshots into two time-shifted matrices
X = {z0, . . . ,zm−1}, Y = {z1, . . . ,zm}, (16)
the best-fit linear model is given by A = Y X†, where X† is the pseudoinverse of X . The ROM is then obtained by projecting the matrices A and C onto a basis U consisting of the r leading left singular vectors of X , which approximate the r leading PCA modes of Z. Using the truncated singular value decomposition
X = UΣV T (17)
where U ,V ∈ Rn×r and Σ ∈ Rr×r, the resulting reduced-order state-transition and observation models are given by
Ar = U TAU = UTY V Σ−1, (18a)
Cr = CU . (18b)
Conveniently, the ROM matrices Ar and Cr can be calculated directly from the truncated SVD of X , which avoids forming the large n× n matrixA.

heading: B KALMAN FILTER
text: The time-dependent Kalman filter that we use as a benchmark in this paper, KF-ROE, is based on the same ROM (2) as the RL-ROE, with identical matrices Ar, Cr and U . Similarly to the RL-ROE, the reduced-order estimate x̂k is given by equation (3a), from which the high-dimensional estimate is reconstructed as ẑk = Ux̂k. However, the KF-ROE differs from the RL-ROE in its definition of the action ak in (3a), which is instead given by the linear feedback term (4). The calculation of the optimal Kalman gainKk in (4) requires the following operations at each time step:
P−k = ArPk−1A T r +Qk, (19)
Sk = CrP − k C T r +Rk, (20)
Kk = P − k C T r S −1 k , (21)
Pk = (I −KkCr)P−k , (22)
where P−k and Pk are respectively the a priori and a posteriori estimate covariance matrices, Sk is the innovation covariance, and Qk and Rk are respectively the covariance matrices of the process noise wk and observation noise vk in the ROM (2). Since these noise covariance matrices are unknown, we choose Qk = Rk = I for all k after verifying empirically that these values yield the best possible results. At time step k = 0, the a posteriori estimate covariance is initialized as P0 = cov(UTz0 − x̂0), which can be calculated from the initial reference and estimated state distributions (15).
Remark 3. We are seeking a Kalman-type observer, which in ‘filters domain’ is also known as an infinite impulse response filter (IIR). These observers are to be contrasted with the finite impulse filters (FIR). Indeed, the later are well known to be based on a mapping between n previous samples of input/output and the desired observed state at the current instant (e.g., Chaupa et al., equations 2,3, and 12), and lead to exact convergence in finite-time, in the noiseless setting. On the other hand the IIR observer is well known to be based on the last measurement of the output/input only, e.g., (Chaupa et al., equation 8), and lead to an average asymptotic performance, e.g.,maximum likelihood estimate in the our case.
Under review as a conference paper at ICLR 2022
ROM training data
(a) (b)

heading: C MODEL ERROR
text: In this appendix, we quantify the accuracy of the two ROMs utilized in Sections 5.1 and 5.2 for the unforced and forced Burgers equations, respectively. For this purpose, we consider three initial conditions defined by (15a) with α = 0.5, 1, 2, and we compute the corresponding reference solution (using the discretized Burgers equation) as well as the ROM solution. Results pertaining to the unforced and forced cases are shown in Figures 5 and 6, respectively.
Figures 5(a,b,c) and 6(a,b,c) show the resulting L2 error of the ROM solution, defined at each time step k by |Uxk − zk|, where xk is the reduced-order state given by the ROM and zk is the highdimensional reference solution. Figures 5(d,e,f) and 6(d,e,f) show the trajectories of the reference and ROM solutions in a two-dimensional slice of phase space spanned by the second and third columns of U – in other words, the time history of the second and third components of UTzk and xk. The L2 error curves and ROM trajectory curves are colored according to the value of time using the same color scheme, which confirms that most of the dynamics take place during the first two time units, as already observed in Figure 1 for α = 1.

heading: D TRAINING HYPERPARAMETERS AND LEARNING CURVES
text: The stochastic policy πθ is trained with PPO using the default hyperparameters from Stable Baselines3, except for the discount factor γ which we choose as 0.75. The mean output of the stochastic policy and the value function are approximated by two neural networks, each containing two hidden layers with 64 neurons and tanh activation functions. The training process alternates between sampling data for 20 trajectories (of length 100 timesteps each) and updating the policy. Each policy update consists of multiple gradients steps through the most recent data using 10 epochs, a minibatch size of 64 and a learning rate of 0.0003. The policy is trained for a total of one million timesteps, corresponding to 10000 trajectories. Figure 7 reports the learning curves corresponding to the unforced and forced cases. During training, the policy is tested (with stochasticity switched off) after each update using 10 separate test trajectories, and is saved if it outperforms the previous best policy.
Under review as a conference paper at ICLR 2022
ROM training data
(a) (b)
Finally, the RL-ROE is defined using the latest saved policy upon ending of the training process, and the stochasticity of the policy is switched off during subsequent evaluation of the RL-ROE.

heading: E ESTIMATION ERROR FOR SHORT TIMES
text: Figures 8 and 9 show the same data as the first row in Figures 2 and 4, but for t ∈ [0, 2]. Focusing on this initial transient phase, it becomes clear that the variance of the RL-ROE estimate decays much faster than the KF-ROE estimate.
Under review as a conference paper at ICLR 2022
ROM training data
(a) (b)

heading: F EFFECT OF PENALIZING THE ACTION MAGNITUDE IN THE REWARD
text: We investigate the effect of penalizing the magnitude of the action ak in the reward function (7). To this effect, we repeat the experiments of Sections 5.1 and 5.2, this time training the RL-ROE using different magnitudes R for the matrix R = RI . When evaluating the performance of the trained RL-ROE, we consider different amounts of Gaussian observation noise added to the measurements yk. The results for the unforced case are shown in Figures 10, 11, and 12 for observation noise of standard deviation σ = 0, 0.1, and 0.3, respectively. The results for the forced case are shown in Figures 13, 14, and 15 for observation noise of standard deviation σ = 0, 0.1, and 0.3, respectively. In absence of noise, the highest estimation accuracy is obtained for R = 0, and decreases as R increases. However, in the presence of noise, Figures 11, 12, and 14, 15, the estimation accuracy is generally highest for R = 10. This confirms that penalizing the magnitude of the action ak in the reward function acts as a regularization that allows the RL-ROE to perform better on noisy measurement data.

heading: G COMPARISON WITH TWO SUPERVISED LEARNING APPROACHES
text: In this appendix, we compare our proposed RL-ROE with two alternative estimation techniques trained offline in a supervised learning setting.
The first approach is purely data-driven. Here, the estimator takes the form
ẑk = UfDNN(yk, . . . ,yk−q), (23)
where the nonlinear function fDNN : Rp×q → Rr is a feed-forward deep neural network (DNN) taking a given number q of past observations as input. The DNN is then trained by minimizing the loss given for each data point by the squared estimation error ||ẑk − zk||2, where zk is the groundtruth high-dimensional state. The entire data set consists of the same high-dimensional reference trajectories used for training the RL-ROE – that is, they are initialized from the distribution (15a) and are limited to the time window t ∈ [0, T/2] = [0, 5]. This results in a total of 14646 data points for all trajectories and time indices. The DNN consists of 4 hidden layers of 50 neurons each, and
Under review as a conference paper at ICLR 2022
Under review as a conference paper at ICLR 2022
Under review as a conference paper at ICLR 2022
Under review as a conference paper at ICLR 2022
is trained using the Adam optimizer over 1000 epochs with minibatch size of 256. The resulting estimator is denoted DNN in the following results.
The second approach is a hybrid ROM-based and data-driven approach, similarly to our proposed RL-ROE. Here, the estimator takes the form
ẑk = U(x̂ ROM k + fDNN(x̂ ROM k ,yk)), (24)
where x̂ROMk is the reduced state generated by the raw ROM (in our case, the DMD model), and the nonlinear function fDNN : Rr+p → Rr is a DNN acting as a nonlinear correction to the ROM prediction. The DNN is then trained by minimizing the loss given for each data point by the squared estimation error ||ẑk − zk||2, where zk is the ground-truth high-dimensional state. The entire data set consists of pairwise combinations of high-dimensional reference trajectories and reduced ROM trajectories. Similar to the training process for the RL-ROE, the reference and ROM trajectories are initialized from the distributions (15a) and (15b), and are limited to the time window t ∈ [0, T/2] = [0, 5]. This results in a total of 305020 data points for all pairs of trajectories and time indices. The DNN consists of 4 hidden layers of 80 neurons each, and is trained using the Adam optimizer over 500 epochs with minibatch size of 256. The resulting estimator is denoted ROM+DNN in the following results.
We compare in Figures 16 and 17 the estimation performance of the RL-ROE with that of the DNN estimator (23) using q = 3. Since the DNN estimator essentially acts as an interpolation function, it is more interesting to compare them in the presence of observation noise, which we model as Gaussian noise of standard deviation σ = 0.1 added to the measurements yk. The RLROE produces more accurate estimates with smoother trajectories; such robustness to measurement noise is a consequence of the stochasticity inherent to the RL training process. Furthermore, a clear benefit of the RL-ROE design philosophy is that it yields an accurate dynamic model, since the imperfect ROM dynamics (in our case given by DMD) are corrected by the RL-trained nonlinear policy ak ∼ π, unlike the DNN estimator design which has no inherent dynamics. This will later allow the RL-ROE to be paired with model-based controllers, opening the door to a wide class of control strategies that are commonly used with Kalman filters.
We compare in Figures 18 and 19 the estimation performance of the RL-ROE with that of the ROM+DNN estimator (24). The comparison is also performed in the presence of Gaussian observation noise of standard deviation σ = 0.1 added to the measurements yk. Once more, the RL-ROE
Under review as a conference paper at ICLR 2022
produces more accurate estimates with smoother trajectories, especially in the forced case. Finally, we show in Figure 20 that in the absence of observation noise, the RL-ROE has better generalization abilities once the time goes beyond the window t ∈ [0, 5] seen during training. These comparisons suggest that the RL-ROE design, in which the policy directly corrects the ROM dynamics and is trained in an RL setting that accounts for errors compounding over time, is more robust to data not seen during training and to measurement noise.
Under review as a conference paper at ICLR 2022
Under review as a conference paper at ICLR 2022

Review summaries:
The objective of the paper is to construct an estimator for the state of a high-dimensional nonlinear dynamical system given partial observations of the state. The objective is motivated by applications in fluid mechanics or turbulent flows where the state of the system is large obtained by discretizing a PDE. The proposed approach has two main steps:   (1) construction of a reduced order model. In particular, the paper proposes the dynamic mode decomposition method which only requires a single trajectory of the nonlinear dynamics.    (2) formulating the problem of finding the estimator as a MDP problem and application of RL techniques to solve it. In particular, the estimator is modeled as a dynamical system driven by a stochastic control policy that depends on the current value of the estimate and the value of observation. The objective function is modeled with running cost equal to the error in estimating the state and a quadratic penalty on the control. Then, the optimal control policy is learned using a policy gradient method by sampling trajectories from the system.    The proposed approach is evaluated on a benchmark example that involves Burger's equation and compared with Kalman filter applied on the reduced order system. 

This paper proposes a new state estimation method based on reinforcement learning for high-dimensional system obtained by discretizing continuous system originally modeled by partial differential equations (PDEs). The proposed method learns to correct the model errors caused by reduced order model (ROM). In the experiment, the proposed method named RL-ROE performs better than ordinary Kalman filter applied to the ROM.

This paper proposes using RL to train a policy to output correction terms in the reduced states of reduced order models in order to perform state estimation. In particular the approach is applied to DMD and Burgers equation, but could presumably be applied to other forms of ROMs and dynamical systems. The method was shown to perform well in experiments, and was found to be more robust than baseline methods in the presence of perturbations and noise.