Controlling Nonlinear Dynamical Systems with Linear Quadratic
Regulator-based Policy Networks in Koopman space
Tomoharu Iwata and Yoshinobu Kawahara
Abstract— We propose a data-driven control method for
nonlinear dynamical systems based on the Koopman operator
theory. Existing Koopman-based control methods apply linear
optimal control methods after system identification by approximating the original cost function in the Koopman space. Therefore, errors in system identification and cost approximation
deteriorate the control performance. On the other hand, theproposed method directly maximizes the control performancewith reinforcement learning, where a controller is modeled bya neural network that consists of a linear quadratic regulator
and an encoder that embeds data into the Koopman space.
We experimentally demonstrate the effectiveness of the proposed method over existing Koopman-based and reinforcement
learning-based methods with two nonlinear dynamical systems.

I. INTRODUCTION
Controlling nonlinear dynamical systems is an important task in a wide variety of fields, such as engineering,
physics, epidemiology, and sociology [6], [18]. Recently, theKoopman operator theory [13], [23] has attracted attention
for data-driven control. Based on the theory, a nonlinear
dynamical system is lifted to the corresponding linear one
in a possibly infinite-dimensional space by embedding statesusing a nonlinear function. Therefore, various methods developed for controlling linear dynamical systems can beextended to nonlinear [24], [17].
For the lifting, we need an appropriate encoder to embed
data in a finite-dimensional subspace, where the functions
span a function space invariant to the Koopman operator.
We call this subspace the Koopman space. To automaticallylearn encoders from data, several methods based on neuralnetworks have been proposed [29], [20], [31], [16], [1], [9].
These methods have shown to achieve high performanceon predicting future values due to the high representation
power of neural networks. With existing control methods
based on the Koopman space [7], [17], [14], [10], [32], [21],[8], [22], system identification is firstly performed, and then,control methods for linear dynamics are applied while fixing
the identified system and approximating the original costfunction in the Koopman space. Since encoders and systems
are trained to improve the prediction performance without
considering control, errors in system identification and cost
approximation cannot be corrected in learning controllers.
In this paper, we propose a method for learning encoders
and dynamics in the Koopman space that directly maximizes
T. Iwata is with NTT Communication Science Laboratories, Japan,

tomoharu.iwata.gy@hco.ntt.co.jp
Y. Kawahara is with Institute of Mathematics for Industry, Kyushu
University, Japan, and with Center for Advanced Intelligence Project,
RIKEN, Japan, kawahara@imi.kyushu-u.ac.jp

the control performance based on reinforcement learning. We
consider a task of making measurement vectors close to apredefined target value for a black-box nonlinear dynamical system. By representing the task as a linear quadratic
problem in the Koopman space, we can obtain a controller
based on a linear quadratic regulator (LQR) [15] that takes
a Koopman embedding as input. We combine an encoderwith the LQR for transforming it to a controller that takes ameasurement vector as input. The controller can be seen as
a neural network, in which an encoder and LQR are used as
its layers. We use the controller as a policy network within
the reinforcement learning framework, and train it by thepolicy gradient method [28] to maximize the rewards defined
based on the control performance. Since the LQR layers,which include procedures for solving a Riccati equation, aredifferentiable, we can backpropagate the expected rewards
to update the controller. Figure 1 illustrates our proposed
method.
The remainder of this paper is organized as follows. In
Section II, we briefly describe related works. In Section III,
we explain the Koopman operator theory, on which theproposed method is based. In Section IV, we formulate ourproblem, and present our method. In Section V, we experimentally demonstrate that the proposed method achieves
better control performance than the existing method using
FitzHugh-Nagumo and Lorenz equations. Finally, we present
concluding remarks and discuss future work in Section VI.

II. R ELATED WORK
Optimal control theory has been studied for a longtime [12]. LQRs are developed in the optimal control theory, and have been successfully used in a wide variety
of applications. With LQRs, we can obtain the optimal
control when dynamics is linear and the cost is quadratic.
The proposed method incorporates LQRs for controlling
nonlinear dynamical systems based on the Koopman operator
theory.
Reinforcement learning methods are categorized into two
types: model-free, and model-based. With neural networks,
model-free reinforcement learning gives a flexible way tolearn controllers, or policies [27]. The proposed method ismodel-free in a sense that a policy network is trained tomaximize the expected reward. It can also be seen as modelbased in a sense that it models linear dynamics and finds an
LQR in the Koopman space.

where xt ∈ X is the state, yt ∈ RD is the measurement
vector, and ut ∈ RJ is the exogenous control vector attimestep t. Functions f and h are black-boxes, i.e., we do notknow them, but we can select control vector ut and observemeasurement vector yt+1 for each timestep. Our aim is tofind appropriate control sequence {ut }Tt=1 that makes themeasurement vectors close to a target y∗ . It is evaluated by
the following cost function,
E=

T 
X


k yt − y∗ k2 +u>
Ru
,
t
t

(5)

t=1

where the first term is the distance between the observed
and target measurements, and the second term is the cost ofcontrols defined by R ∈ RJ×J .
Fig. 1. Our proposed method. At each timestep, the controller receivesa measurement vector from a black-box nonlinear dynamical system. The
controller then chooses a control vector, which is subsequently sent to thesystem. The controller assumes linear dynamics and quadratic cost in the
Koopman space. The cost is approximated from the original cost in themeasurement space. The optimal gain is calculated from the dynamics andcost based on linear quadratic regulators. In the controller, a measurement
vector is embedded in the Koopman space by the encoder. The control
vector is determined using the optimal gain and Koopman embedding. The
parameters of the controller, which consist of the dynamics in the Koopman
space and the parameters of the encoder, are trained by minimizing theoriginal cost using reinforcement learning.

III. P RELIMINARIES : KOOPMAN OPERATOR THEORY
We consider nonlinear discrete-time dynamical system,
xt+1 = f (xt ),(1)

where xt ∈ X is the state at timestep t. Koopman operator
A is defined as an infinite-dimensional linear operator thatacts on observables g : X → R (or C) [13],g(xt+1 ) = Ag(xt ),(2)with which the analysis of nonlinear dynamics can be liftedto a linear (but infinite-dimensional) regime.
Although the existence of the Koopman operator is theoretically guaranteed in various situations, its practical use
is limited by its infinite dimensionality. We can assume therestriction of A to a finite-dimensional subspace G [29]. If
G is spanned by a finite number of functions {g1 , . . . , gK },then the restriction of A to G, which we denote A ∈ RK×K ,becomes a finite-dimensional operator,
gt+1 = Agt ,(3)

where gt = [g1 (xt ), . . . , gK (xt )] ∈ RK is a vector ofobservables at timestep t.

A. Problem formulation
We assume the following nonlinear dynamical system,
yt = h(xt ),According to the Koopman operator theory, we consider
the following linear dynamics with control in a finitedimensional Koopman space [26], [2],gt+1 = Agt + But ,(6)

where gt ∈ RK is the observables at timestep t, A ∈ RK×K
is a finite-dimensional approximation of the Koopman operator, and B ∈ RK×J represents the effect of a control vector
on the observables at the next timestep.
To realize a linear dynamics in the Koopman space, we
obtain the observables as a function of measurement vector,
φ : RD → R K ,gt = φ(yt ),(7)

where φ is a feed-forward neural network, and we call it anencoder.
Suppose that we are given a set of N sequences of measurement and control vectors, D = {{(ynt , unt )}Tt=1 }N
n=1using the dynamical system in Eq. (4), where ynt and unt
are the measurement and control vectors at timestep t in thenth sequence. Let ψ : RK → RD be is a decoder modeledby a feed-forward neural network that maps the observables
into the measurement space. We can estimate A, B, andparameters of φ and ψ by minimizing the following objectivefunction,
H=

N
T
X
X
n=1 t=1
T−1
X

k ynt − ψ(gnt ) k2!
2

k yn,t+1 − ψ(Agnt + Bunt ) k

+η

,

(8)

t=1

IV. P ROPOSED METHOD

xt+1 = f (xt , ut ),B. System identification with control in Koopman space(4)

where the first term is the reconstruction loss, the second
term is the prediction loss when the next measurement vector
is predicted through the Koopman space, and η > 0 is thehyperparameter.

C. Control via Koopman space
We approximate the cost in the measurement space in
Eq. (5) by the cost in the Koopman space,
Ẽ =

T 

X
k gt − g∗ k2 +u>
Ru
t ,
t

(9)

t=1

where g∗ = φ(y∗ ) is the observables of target measurementy∗ . Here, we assume that if observables are closely locatedin the Koopman space, their measurement vectors are also
closely located in the measurement space.
The approximated cost (9) can be rewritten in a quadratic
form,
Ẽ =

T
X


∗ 2
gt0> Qgt0 + u>t Rut + T k g k ,

(10)

t=1

where
gt
1



I −g∗
∗>
−g
0



gt0 =


Q=



∈ RK+1 ,(11)

D. Improving control with reinforcement learning∈ R(K+1)×(K+1) .

(12)

Using Eq.(6), the dynamics of gt0 is given by0
gt+1= A0 gt0 + B0 ut ,

(13)

where
A0 =



A 0
0 1







B0 =

B
0

Fig. 2. Our policy network π. The parameters in π to be estimated are
Ψ = {A, B, θ}, where A and B are the dynamics in the Koopman spacein Eq. (6), and θ is the parameters of encoder φ. First, solution P of the
Riccati equation in Eq. (18) is solved by iterating Eq. (18) using A, B,
Q, and R with initialization P = Q. Second, optimal gain matrix F iscalculated using P, A, B, and R by Eq. (17). Third, measurement vector
y is transformed to observable g in the Koopman space by encoder φ withparameter θ. Finally, control vector u is obtained using F and g by Eq. (16).
Our policy network can be seen as a neural network with parameters Ψ thattakes measurement vector y as input and outputs control vector u.

∈ R(K+1)×(K+1) ,(14)∈ R(K+1)×J .

(15)

Since the dynamics is linear in Eq. (6) and the cost isquadratic in Eq. (10), linear quadratic regulators (LQRs) areapplicable in the Koopman space. With LQRs, the optimal
control vector is given by
ut = −Fgt0 ,(16)

where F ∈ RJ×(K+1) is the optimal gain matrix calculatedbyF = (R + B0> PB0 )−1 (B0> PA0 ).

(17)

P ∈ R(K+1)×(K+1) is the solution of the following discretetime Riccati equation,
P = A0> PA0− (A0> PB0 )(R + B0> PB0 )−1 (B0> PA0 ) + Q.

(18)

The Riccati equation can be solved by iterating Eq. (18) untilconvergence with initialization P = Q.

If we perfectly identify the dynamical system A, B, φ via
the Koopman space, and the cost in the Koopman spacein Eq. (9) is exactly equivalent to that in the measurement
space in Eq. (5), the control based on LQRs in Eq. (16) is
optimal. However, the system identification is difficult due
to its nonlinearity and measurement noise. In addition, thecost in the Koopman space might not approximate that in
the measurement space well when these two spaces have far
different metrics.
Let θ be parameters of encoder φ. To improving control
via the Koopman space, we propose to tune parameters Ψ ={A, B, θ} using reinforcement learning, such that the costin the measurement is minimized when an LQR is adopted
in the Koopman space. Our problem in Section IV-A canbe defined on a Markov decision process (MDP), where thecurrent measurement vector yt is a state of the MDP, thecontrol vector ut is an action, and the cost at a timestep
E(y, u) =k y − y∗ k2 +u> Ru,(19)

is a negative reward. Let π : RD → RJ be a controller, or
policy in the MDP, that takes a measurement vector as input,
and outputs an appropriate control vector. We model policy
π using A, B and φ considering an LQR in the Koopman
space as in Eq. (16),π(y; Ψ) = −F(A, B)[φ(y)> 1]> ,(20)

where F(A, B) is the optimal gain matrix obtained by
Eqs. (17,18), which is a function of A and B. Figure 2
illustrates our policy network π. Since Eq. (17) and theiteration of Eq. (18) are differentiable, we can backpropagate
the rewards through our policy network in Eq. (20) to updateparameters Ψ.

We estimate the parameters of the policy network Ψ using
the policy gradient method [28]. Letrt (yt , ut ) = −TXγ τ −t E(yτ , uτ )

(21)

τ =t

be the cumulative discounted reward at timestep t withdiscount factor γ. With the reinforcement learning framework, the objective function to be maximized is the expectedcumulative discounted reward,
J(Ψ) =TX

Eyt ,ut [rt (yt , ut )],

(22)

t=1

where E is the expectation when measurement yt is generated from the dynamical system in Eq. (4), and control ut
is generated from the following policy distribution,p(ut |yt ; Ψ) = N (π(yt ; Ψ), σ 2 I).

(23)

Here, N (µ, Σ) is the normal distribution with mean µ andcovariance Σ. With the policy gradient theorem, the gradientof objective function J with respect to parameters Ψ is givenbyT∂ log p(ut |yt ; Ψ)∂J(Ψ) X
Eyt ,ut rt (yt , ut )=. (24)∂Ψ∂Ψ
t=1
The training procedure is shown in Algorithm 1. The
expectation in Eqs. (22,24) is approximated by the Monte
Carlo method using sampled measurement vectors, control
vectors, and cumulative discount rewards. We use the averageof the discounted sum of the future rewards as a baseline to
reduce the variance [30], [4].
V. E XPERIMENTS
A. Dynamical systems
We evaluated the proposed method using two nonlinear
dynamical systems: FitzHugh-Nagumo and Lorenz equations. The FitzHugh-Nagumo equation [3], [25] is a modelof an excitable system such as a neuron. We added scalar
control variable u1 to the FitzHugh-Nagumo equation as
follows,
dx1
x3= x1 − 1 − x2 + I,
dt
3

dx2
= c(x1 − a − bx2 ) + u1 ,dt
(25)

where we used a = 0.7, b = 0.8, c = 0.08, and I = 0.8. The
control variable was bounded a continuous value [−2.5, 2.5].
The Lorenz equation [19] is a chaotic nonlinear system,
which was derived as as a model for atmospheric convection.
We added scalar control variable u1 to the Lorenz equation
as follows,
dx1
dx2= σ(x2 − x1 ),= x1 (ρ − x3 ) − x2 + u1 ,
dt
dt
dx3
= x1 x2 − βx3 ,(26)dtwhere we used σ = 10, ρ = 28, and β = 8/3. The control
variable was bounded a continuous value [−50, 50].

Algorithm 1 Training procedure of our policy network.
Input: Black-box dynamical system, target y∗ , control costR, number of iterations to solve the Riccati equation L,
control variance σ 2 , discount factor γ.
Output: Trained parameters Ψ.
1: Pretrain parameters Ψ by minimizing H in Eq. (8) using
sequences of measurement and control vectors D.
2: while End condition is satisfied do
3:
Initialize the solution of the Riccati equation P = Q.
4:for ` ∈ {1, · · · , L} do
5:Iterate Eq. (18) to solve the Riccati equation.
6:
end for
7:
Obtain optimal gain matrix F by Eq. (17).
8:
Randomly sample initial measurement vector y1 .
9:for t ∈ {1, · · · , T } do
10:
Calculate the mean of the control distribution
π(yt ; Ψ) by Eq. (20).
11:
Sample control ut according to the control distribution in Eq. (23).
12:
Calculate netagive reward E(yt , ut ) by Eq. (19).
13:
Obtain next measurement vector yt+1 from thesystem with sampled control ut .
14:
end for15:
for t ∈ {1, · · · , T } do
16:
Calculate cumulative discounted rewards rt (yt , ut )by Eq. (21).
17:
end for18:
Calculate J in Eq. (22) and its gradient in Eq. (24)using sampled measurement vectors yt , control vectors ut , and cumulative discount rewards rt for t ∈{1, · · · , T }.
19:
Update model parameters Ψ using J and its gradientusing a stochastic gradient method.
20: end while

Figure 3 shows examples of measurement vector sequences with random control. For both systems, measurement vectors were obtained by yt = xt + , where 
was Gaussian noise with mean 0 and standard deviation
10−2 . For the cost function in Eq. (5), we used zero target
measurement vector y∗ = 0, control cost R = 10−2 , andtimesteps T = 100.
B. Proposed method setting
We used four-layered feed-forward neural networks with
128 hidden units, ReLU activation, and Koopman space K =64 for encoder φ and decoder ψ. We pretrained parameters
Ψ as well as parameters of decoder ψ by minimizing H
in Eq. (8) with regularization hyperparameter η = 1 using
100 sequences of measurement and control vectors D withrandom initialization and random control as described in
Section IV-B. For the optimization, we used Adam [11] withlearning rate 10−3 , batch size one, and 1,000 training epochs.
Then, parameters Ψ were tuned by maximizing J in Eq. (22)using the policy gradient method as described in Section IVD. The maximum number of iterations for solving the Riccati

TABLE I
AVERAGE T EST COSTS AND THEIR STANDARD ERRORS .

(a) FitzHugh-Nagumo
Fig. 3.

(b) Lorenz

Ours
NDMD
NDMD+CNDMD+DDMD
DMD+RLRL

FitzHugh-Nagumo
83.1 ± 2.9
443.9 ± 90.6
343.2 ± 4.9
940.3 ± 293.4
312.8 ± 1.5
211.7 ± 15.3
85.7 ± 2.6

Lorenz5151.8 ± 786.8
83152.3 ± 521.2
86526.7 ± 24.0
84631.8 ± 1171.8
73073.3 ± 1982.8
37590.1 ± 3059.8
24310.1 ± 4131.8

Measurement vector sequences with random control.

equation in Eq. (18) was 20. The discount factor in Eq (21)was γ = 0.99. The variance of control σ 2 in Eq. (23) was
set to half of the bounded width of the control variable, i.e.,
2.5 for FitzHugh-Nagumo, and 50 for Lorenz equations. For
the optimization, we used Adam with learning rate 10−3 ,batch size 10, and maximum number of iterations 5,000. 50
simulation runs were evaluated for each iteration for early
stopping.
C. Comparing methods
We compared the proposed method with the following fivemethods: NDMD, NDMD+C, NDMD+D, DMD, DMD+RL,
and RL, which were based on the Koopman operator theory
and/or reinforcement learning. The comparing methods weretrained in the same way with the proposed method.
NDMD is neural dynamic mode decomposition [29],where system identification in the Koopman space was conducted as in Section IV-B assuming the neural network-based
encoder and decoder. The control sequences were obtainedby an LQR as in Section IV-C. NDMD corresponds to the
proposed method without reinforcement learning proceduresin Section IV-D.
NDMD+C is NDMD with a regularizer to improve controllability as [7]. A linear dynamical system is controllable if controllability matrix C =[B AB A2 B · · · AK−1 B] has full row rank. Since
the number of non-zero singular values is equal
the rank of
Pto
K
1
1
the matrix, we added regularization term K
k=1 s̃k to H
skP
in Eq. (8), where s̃k = K s 0 is the normalized singulark0 =1 kvalue, and sk is the singular value of controllability matrix
C.
NDMD+D is NDMD with a regularizer to improve costfunction approximation as [17] by making distances in the
Koopman space close to thosePin thespace.
Pmeasurement
PT
TN
We added regularization term n=1 t=1 t0 =1 kk ynt −ynt0 k2 − k φ(ynt ) − φ(ynt0 ) k2 k2 to H in Eq. (8).
DMD is dynamic mode decomposition, where system
identification in the Koopman space was conducted without
neural network-based encoders and decoders assuming theKoopman space was the same with the measurement space
gt = yt . The control sequences were obtained by an LQR.
DMD+RL is the DMD with reinforcement learning procedures in Section IV-D. DMD+RL corresponds to the
proposed method without encoders and decoders.

RL is reinforcement learning using the policy gradientmethod with a policy of a feed-forward neural network. The
policy network takes a measurement vector as input, andoutputs a control vector. We used a four-layered feed-forward
neural network with 128 hidden units, and ReLU activation.
RL corresponds to the proposed method with a black-box
policy that considers neither Koopman spaces nor LQRs.
D. Results
Table I shows test costs averaged over ten experiments.
Figure 4 show examples of controlled dynamics by the
proposed method, NDMD, and RL. The proposed method
achieved the lowest test cost with both systems. The test
cost of NDMD was high because system identification of
nonlinear dynamical systems was difficult, and the costin the Koopman space could not approximate that in themeasurement space appropriately. Even with regularizers(NDMD+C/D), the performance was not improved well. On
the other hand, the proposed method directly minimized
the cost in the measurement space with the reinforcement
learning framework. The test cost of the DMD was highbecause it could not model the nonlinear dynamical systems.
By incorporating the reinforcement learning framework with
DMD+RL, the performance was improved compared with
DMD. However, the performance of DMD+RL was worse
than that of the proposed method. This result indicates that
it is important to use nonlinear transformation by encoders
to the Koopman space for controlling nonlinear dynamicalsystems. RL achieved better performance than the othercomparing methods due to its direct minimization of the costin the measurement space, and the high representation powerof neural networks. With RL, the policy network was fullymodeled by a neural network as a black-box function. In
contrast, with the proposed method, the policy network wasmodeled by a neural network incorporating the knowledge on
the Koopman operator theory and linear quadratic regulators.
Therefore, the proposed method obtained better controllers
than RL.
VI. C ONCLUSION
We proposed a method to learn a controller for nonlinear
dynamical systems. With the proposed method, a policy
network is modeled based on a linear quadratic regulator
in a Koopman space, and it is trained by minimizing the
expected cost with reinforcement learning. Although our
results are encouraging, we must extend our approach inseveral future directions. First, we will apply it to otherFitHugh-Nagumo

Lorenz(a) Ours(b) NDMD(c) RL
Fig. 4. Examples of controlled dynamics of (Left) FitzHugh-Nagumo and(Right) Lorenz equations by (a) our method, (b) NDMD, and (c) RL.

nonlinear dynamical systems, such as fluid and robots.
Second, we plan to improve our proposed method using
reinforcement learning techniques, which includes actorcritic methods [5]. Third, we will incorporate other control
methods for linear dynamical systems than linear quadratic
regulators in our framework based on the Koopman operator
theory and reinforcement learning.
ACKNOWLEDGEMENT
This work was partially supported by JST CREST Grant
No. JPMJCR1913.
R EFERENCES[1] O. Azencot, N. B. Erichson, V. Lin, and M. W. Mahoney. Forecasting
sequential data using consistent Koopman autoencoders. In International Conference on Machine Learning, 2020.
[2] D. Bruder, B. Gillespie, C. D. Remy, and R. Vasudevan. Modelingand control of soft robots using the Koopman operator and model
predictive control. In Robotics: Science and Systems XV, 2019.
[3] R. FitzHugh. Mathematical models of threshold phenomena in thenerve membrane. The Bulletin of Mathematical Biophysics, 17(4):257–
278, 1955.
[4] E. Greensmith, P. L. Bartlett, and J. Baxter. Variance reduction
techniques for gradient estimates in reinforcement learning. Journalof Machine Learning Research, 5:1471–1530, 2004.
[5] I. Grondman, L. Busoniu, G. A. Lopes, and R. Babuska. A surveyof actor-critic reinforcement learning: Standard and natural policy
gradients. IEEE Transactions on Systems, Man, and Cybernetics, PartC, 42(6):1291–1307, 2012.
[6] S. J. Guastello. Chaos, catastrophe, and human affairs: Applicationsof nonlinear dynamics to work, organizations, and social evolution.
Psychology Press, 2013.

[7] Y. Han, W. Hao, and U. Vaidya. Deep learning of Koopman
representation for control. In IEEE Conference on Decision andControl, pages 1890–1895, 2020.
[8] B. Huang, X. Ma, and U. Vaidya. Feedback stabilization using
Koopman operator. In IEEE Conference on Decision and Control,
pages 6434–6439, 2018.
[9] T. Iwata and Y. Kawahara. Neural dynamic mode decomposition
for end-to-end modeling of nonlinear dynamics. arXiv preprint
arXiv:2012.06191, 2020.
[10] E. Kaiser, J. N. Kutz, and S. L. Brunton.
Data-driven discovery of Koopman eigenfunctions for control. arXiv preprint
arXiv:1707.01146, 2017.
[11] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization.
In International Conference on Learning Representations, 2015.
[12] D. E. Kirk. Optimal control theory: an introduction. Courier
Corporation, 2004.
[13] B. O. Koopman. Hamiltonian systems and transformation in Hilbert
space. Proceedings of the National Academy of Sciences of the UnitedStates of America, 17(5):315–318, 1931.
[14] M. Korda and I. Mezić. Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control. Automatica,
93:149–160, 2018.
[15] H. Kwakernaak and R. Sivan. Linear optimal control systems,
volume 1. Wiley-interscience New York, 1972.
[16] K. Lee and K. T. Carlberg. Model reduction of dynamical systems on
nonlinear manifolds using deep convolutional autoencoders. Journalof Computational Physics, 404:108973, 2020.
[17] Y. Li, H. He, J. Wu, D. Katabi, and A. Torralba. Learning compositional Koopman operators for model-based control. In International
Conference on Learning Representations, 2019.
[18] W.-m. Liu, H. W. Hethcote, and S. A. Levin. Dynamical behaviorof epidemiological models with nonlinear incidence rates. Journal of
Mathematical Biology, 25(4):359–380, 1987.
[19] E. N. Lorenz. Deterministic nonperiodic flow. Journal of Atmospheric
Aciences, 20(2):130–141, 1963.
[20] B. Lusch, J. N. Kutz, and S. L. Brunton. Deep learning for universal
linear embeddings of nonlinear dynamics. Nature communications,9(1):1–10, 2018.
[21] X. Ma, B. Huang, and U. Vaidya. Optimal quadratic regulation of
nonlinear system using Koopman operator. In American Control
Conference, pages 4911–4916, 2019.
[22] A. Mauroy, Y. Susuki, and I. Mezić. The Koopman Operator in
Systems and Control. Springer, 2020.
[23] I. Mezić. Spectral properties of dynamical systems, model reduction
and decompositions. Nonlinear Dynamics, 41(1-3):309–325, 2005.
[24] J. Morton, A. Jameson, M. J. Kochenderfer, and F. Witherden. Deep
dynamical modeling and control of unsteady fluid flows. In Advancesin Neural Information Processing Systems, pages 9258–9268, 2018.
[25] J. Nagumo, S. Arimoto, and S. Yoshizawa. An active pulse transmission line simulating nerve axon. Proceedings of the IRE, 50(10):2061–
2070, 1962.
[26] J. L. Proctor, S. L. Brunton, and J. N. Kutz. Dynamic mode
decomposition with control. SIAM Journal on Applied Dynamical
Systems, 15(1):142–161, 2016.
[27] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van
Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam,
M. Lanctot, et al. Mastering the game of go with deep neural networks
and tree search. Nature, 529(7587):484–489, 2016.
[28] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradientmethods for reinforcement learning with function approximation.
Advances in Neural Information Processing Systems, 12:1057–1063,
1999.
[29] N. Takeishi, Y. Kawahara, and T. Yairi. Learning Koopman invariant
subspaces for dynamic mode decomposition. In Advances in Neural
Information Processing Systems, pages 1130–1140, 2017.
[30] L. Weaver and N. Tao. The optimal reward baseline for gradient-based
reinforcement learning. In Conference on Uncertainty in Artificial
Intelligence, pages 538–545, 2001.
[31] E. Yeung, S. Kundu, and N. Hodas. Learning deep neural network representations for Koopman operators of nonlinear dynamical systems.
In American Control Conference, pages 4832–4839, 2019.
[32] P. You, J. Pang, and E. Yeung. Deep Koopman controller synthesis for cyber-resilient market-based frequency regulation. IFACPapersOnLine, 51(28):720–725, 2018.