DeSKO: Stability-Assured Robust Control with a Deep Stochastic Koopman Operator
Authors: Minghao Han
Jacob Euler-Rolle
Robert K. Katzschmann
Venue: ICLR 2022 Poster (2022)
Keywords: Koopman Operator
Robust Control
Robotics
Model Predictive Control
Soft Robotics

Abstract: The Koopman operator theory linearly describes nonlinear dynamical systems in a high-dimensional functional space and it allows to apply linear control methods to highly nonlinear systems. However, the Koopman operator does not account for any uncertainty in dynamical systems, causing it to perform poorly in real-world applications.
Therefore, we propose a deep stochastic Koopman operator (DeSKO) model in a robust learning control framework to guarantee stability of nonlinear stochastic systems. The DeSKO model captures a dynamical system's uncertainty by inferring a distribution of observables. We use the inferred distribution to design a robust, stabilizing closed-loop controller for a dynamical system. Modeling and control experiments on several advanced control benchmarks show that our framework is more robust and scalable than state-of-the-art deep Koopman operators and reinforcement learning methods. Tested control benchmarks include a soft robotic arm, a legged robot, and a biological gene regulatory network. We also demonstrate that this robust control method resists previously unseen uncertainties, such as external disturbances, with a magnitude of up to five times the maximum control input. Our approach opens up new possibilities in learning control for high-dimensional nonlinear systems while robustly managing internal or external uncertainty.

Published as a conference paper at ICLR 2022
DESKO: STABILITY-ASSUREI ROBUST CONTROL OF NONLINEAR SYSTEMS WITH A DEEP STOCHASTIC KOOPMAN OPERATOR
Minghao Han 1.2 Jacob Euler-Rolle", Robert K. Katzschmann'
Department of Control Science and Engineering, Harbin Institute of Technology 2 Soft Robotics Lab, ETH Zurich (minhan, jacob, rkk rkk]@etha @ethz. ch
ABSTRACT
The Koopman operator theory linearly describes nonlinear dynamical systems in a high-dimensional functional space and it allows to apply linear control methods highly nonlinear systems. However, the Koopman operator does not account for any uncertainty in dynamical systems, causing it to perform poorly in real-world applications. Therefore, we propose a deep stochastic Koopman operator (DeSKO) model in robust learning control framework to guarantee stability of nonlinear stochastic systems. The DeSKO model captures a dynamical system's 's uncertainty by inferring distribution of observables. We use the inferred distribution to design a robust, stabilizing closed-loop controller for dynamical system. Modeling and control experiments on several advanced control benchmarks show that our frame- work is more robust and scalable than state-of-the-ar deep Koopman operators and reinforcement learning methods. Tested control benchmarks include soft robotic arm, a legged robot, and a biological gene regulatory network. We also demonstrate that this robust control method resists previously unseen uncertainties, such as external disturbances, with magnitude of up five times the maximum control input. Our approach opens up new possibilities in learning control for nigh-dimensional nonlinear systems while robustly managing internal external uncertainty.
INTRODUCTION
Modeling is crucial to the development of intelligent machines and robots. It is needed to predict the behavior of a dynamical system, analyze system's interior properties, realize reliable planning and control (Müller et al.) (2007). For many systems of interest, such as a soft robotic arm that can interact with an unknown object (George Thuruthel et al. 2018), the governing equations are either unknown concise model.
complex to be useful (Bakker et al. 2020). Thus, it essential
accurate but
The need for fast model that is able to represent complex systems has led to recent advancements in data-driven modeling of complex dynamics, and learning world models. These advancements have been achieved by constructing surrogate neural network (NN) models (Chua et al.. 2018 Ha & Schmidhuber 2018 Zhang al. 2019). By formulating the modeling problem as supervised learning task, the neural network can be trained to infer the systems' dynamics based on the dense state and action input (Nagabandi et al. 2018b) or high-din digh-dimensional image observations (Hafner et al. 2019). However, nonlinear systems are often approximated by complex and over-parameterized neural networks (Morton et al. 2019). the learned models are restrictive when they are used for dynamical analysis and controller design. Optimal control problems based on NN models are typically non-convex, and they need to be solved by using inefficient population
optimization methods (Bharadhwaj et al. 2020), such as random shooting (Zhang et al. 2019 Nagabandi et al.
201 cross entropy methods (Chua et 2018 Hafner al. 2019 Wang & Ba 2019).
Recently, the Koopman operator theory has attracted much attention. The Koopman operator can model complex nonlinear systems by linearly propagating observables in infinite dimensional functional space (Koopman) 1931b). This propagation allows us apply the well-established linear control and analysis theory to nonlinear systems. Based on system's 's runtime data, a linear operator, which propagates the observables in time, is then learned. Based on this idea, practical algorithms, like

---

Published as a conference paper at ICLR 2022
dynamic mode decomposition (DMD) and extended dynamic mode decomposition (EDMD). were developed by selecting a finite set of parameterized linear and nonlinear observable functions. These
methods have been applied to the modeling of soft robotic arms (Bruder et al.
2020
Haggerty et al.
2020), multirotors (Folkestad et al. 2020), and mobile robots (Shi & Karydis 202T
More recently.
Deep- DMM exploits deep learning techniques to automate the design of observable functions (Lusch
et al. 2018 Otto & Rowley 2019; Han et al. 2020. Yeung al., 2018 Morton et al. 20188. Deep- DMD automatically learns a richer set of observable functions than DMD or EDMD, and it can also be more accurate, even though it uses fewer observables (Bakker et al.) 2020). However, these methods focus modeling nominal systems with clean data sets, while noise, either in the process or observation, has not yet been accounted for. Furthermore, it is challenging to ensure closed-loop stability in controller design that is based on learned Koopman operator model (Bakker et al. 2020).
In this work, we learn and control uncertain nonlinear dynamics using the deep stochastic Koopman operator (DeSKO) model. Instead of encoding the observables deterministically, DeSKO infers the probabilistic distribution of observables and learns linear operator that can propagate the inferred distribution forward in time. Furthermore, we propose efficient and robust model predictive controller based on the DeSKO model. In terms of robotic control benchmarks, we show that our proposed method outperforms state-of-the-art deep Koopman operators (Lusch et al. 2018 Han et al. 2020) in modeling and in stabilizing systems that have been corrupted by process and observation noises. Furthermore, we show that the proposed framework is scalable; it can handle high-dimensional and complex systems, including legged robots, soft robotic arms, and biological gene networks.
The outline of this paper is as follows. In Section 2 we introduce and discuss the related works. In Section 3, we introduce the basic notations and concepts of the Koopman operator (KO), and present our proposed DeSKO model. In Section 4 we discuss our robust control framework and prove its stability. In Section 5 we evaluate the DeSKO model in terms of modeling, control, and robustness in eight different environments, and we compare it to baseline methods. Finally, in we conclude the paper and discuss future challenges.
Section 6
2 RELATED WORKS
Following the Koopman operator theory formalized in Kooomann (193 various practical algorithms have been proposed to extract the Koopman spectral properties from state-transition data. DMD and EDMD adopt a hand-crafted set of linear and nonlinear functions to encode the observables and obtain the linear operator through least square optimization (Schmid) 2010 Tu et al.) 2014 Williams et al. 2015). A straightforward approach to automate the design of observable functions to parameterize the observable functions as neural networks and use deep learning techniques for training (Lusch et al. 2018: Otto & Rowley| 2019; Yeung et al.) [018 Morton et al. 2018 Takeishi et al. 2017). Unlike DMD and EDMD, DMD approaches do not suffer from the curse of dimensionality; that is, the number of observables does not grow exponentially with the number of states. Furthermore, Deep-DMD can also express a richer class of observable functions than DMD or EDMID. Graph neural networks and block-wise Koopman operators in (Li et al. 2020) to model objects composed of similar building blocks, such as ropes.
Koopman operators
introduced
While the described works have greatly advanced the field of modeling and control by applying the Koopman operator, several challenges still remain. Azencot et al. (2020) proposed a Koopman autoencoder framework that can learn a Koopman model from noisy data for long horizon prediction, but only considered autonomous systems without control input. Morton et al. (2019) proposed to measure uncertainty with an ensemble of Koopman models sampled from the encoded distribution. However, their work is focused on deterministic systems with clean datasets. Furthermore, how to guarantee stability with a learned Koopman model remains an open question. Mamakoukas al. (2020); Kolter & Manek (2019); Pan & Duraisamyl (2020) studied how to learn a stable dynamics of embeddings on the latent space, while we are focused on stabilizing the original system with the learned model. Our method differs from previous works by learning a stochastic model from noisy data, and providing a robust control framework with a stability guarantee. We show that this framework can easily generalize to systems with process and observation noises.
Model-based Learning Control In recent years, model- reinforcement learning has dominated a large portion of model-based learning control studies (Pascanu et al.) 2017: Hamrrckk et al. 2017 Racanière et al.; (017). Several researchers have combined learned neural network models with
model predictive control (Chua et al., 20188) Zhang et al. 2019 Hafner et al. 2019 Nagabandi et al.
2018al Wang & Bal 2019). In this work, we utilize the Koopman operator's 's linear propagation
2

---

Published as a conference paper at ICLR 2022
property, and conduct model predictive control efficiently. Furthermore, also show that the Koopman operator substantially facilitates analyzing and assuring stability of the closed-loop system.
Gaussian process-based methods are another popular class of learning control methods (Kocijan et al. 2004, [Berkenkamp al. 2017 Vinogradska et al. 2016 [Berkenkamp & Schoellig 2013). As type of con-maraettric method, Gaussian process-basee based methods are efficient in modeling with small data sets. Indeed, Gaussian process models have been applied to various robotic systems (Chang et al. 2017, Cao af., 2017 Kahzan et al. 2019 Chang et al. 2020p. Nonetheless, Gaussian process- based approaches are typically limited to low-dimensional systems and small data sets (Wang et al. (2020). In this work, present a framework that is scalable to high-dimensional robotic control tasks.
MODELING
In this section, we will first introduce the basic notations and concepts of the Koopman operator (KO) Koopman 1931a). Then the derivation of the DeSKO model established.
3.1 THE KOOPMAN OPERATOR
For the sake of clarity, we introduce the KO using notations that are similar to the ones in Bakker et al. (2020). Consider the nonlinear discrete time dynamical system
T++1 = ( u4)
where f is nonlinear differentiable function. The system can be lifted to cictinte-dimensionn function space F composed of all square- inteerrblee real-valued functions within the compact domain X x U C Ru++m Elements of F are called observbles. On this space, the flow of the systems is characterized by the Koopman operator K. F- F, which is an dimensional linear operator that satisfies
Kob(Ze; u,) o f u4)
where vi denotes the observable function. However, the
function space F and the KO IC are impractical. Therefore, finite dimensional function space F. C F, spanned by linearly independent basis function vi IR" + Reth is used instead, where h is the dimensionality of the observable function specified by the designer. Here, denotes the index of each observable function. The linear property of the KO enables the use of linear control techniques for efficient control. The Koopman equation can be written as
Anp(x.) - Bu, = 1/ [1I, we)
where the KO is split into the Koopman matrix A E Ax and the control matrix B E Ax For the reader's convenience, we also have also included the details for learning Koopman matrices with a set of observables in the Appendix.
The Koopman operator conducts deterministic inference of observables and ignores the existence of any uncertainty that may stem from the system and the lack of data about the system. A controller design based on such a defective model would inevitably perform suboptimally. and even result in failure. In this work, we interested in learning and controlling a nonlinear system that has been corrupted by additive noise:
1++1
f( u+) + w.
where w; Pwo denotes random noises subject to the distribution par In this case, the transition of states is probabilistic. and it denoted by p(#(+1 |ze, ue). In the next section, we will discuss how to overcome this challenge by encoding and propagating the distribution of observables with Koopman operators.
3.2 DEEP STOCHASTIC KOOPMAN OPERATOR
The deep stochastic Koopman operator (DeSKO) consists of two building blocks: probabilistic neural network that encodes the distribution of the observables, and a KO that propagates the distribution into the future. In the following section, we will describe the construction, implementation, and optimization of the DeSKO.
Probabilistic neural network To account for uncertainty, we exploit a probabilistic model po (xx)) to encode the distribution of observables given the current state. The probabilistic model is parame- terized by a probabilistic neural network. The output neurons of the probabilistic NN characterize
3

---

Published as a conference paper at ICLR 2022
Recursive Propagation
(e(x))
p(,,
Koopman Matrix A p(Preal*,
Observation Matrix Xx+1
dg (xce
U4 x
Control Matrix
Figure I: Structure of the Deep Stochastic Koopman Operator (DeSKO). With the help of two neural networks, the DeSKO model encodes the system state 2; into the parameters of Gaussian distribution observables, which are mean vector (t (10();) and the variance vector og (24y). Then, the learned Koopman matrix A and control matrix B propagate the distribution to the next time step. This can recursively produce series of distribution predictions over a given time horizon. Finally, the learned observation matrix C maps the observables to the state space.
the parameters of the distribution to capture the uncertainty caused by the process and observation noise. In our particular case, we design two NNs to output mean vector Me (2(x)) and diagonal covariance vector o (x(x)) Together, these vectors describe multivariate Gaussian distribution, which is defined as Pa ( (ry) . (1og (xx), ara (xx)) (see also Figure A Gaussian distribution is often chosen for continuous- valued states, and it reasonable choice if the uncertainty in the system is unimodal (Chua et al. 2018). While a Gaussian distribution is effective our experiments, other tractable distributions can be used instead. In order to expressively encode the dynamics, the distribu- tion parameters can be represented with an arbitrarily high complexity by using nonlinear functions that depend on the current state. This makes it feasible to incorporate NNs into the probabilistic model.
Propagation of observable distributions As shown in Figure given a series of control input {u,, u.+1...... M4+Th }. the distribution of observables encoded by the probabilistic NNs is recursively propagated into the future with the Koopman matrix A and control matrix B. According to ( the KO propagates the observables pointwise the functional space F. By conducting pointwise mapping. KO is also able infer the future distribution of observables. The use of KO to conduct distribution propagation can also be justified from the perspective of the Frobenius-Peree Operator (FPO) and has been discussed in the Appendix. Finally, the decoder maps the observables back to the state space and produces the prediction results. The decoder is designed to be linear, and parameterized by an observation matrix m This facilitates controller design, which will be discussed in the next section.
Optimization procedure To optimize the probabilistic neural network and the Koopman operator, including the observation matrix, the single-step prediction loss is given as
B,, C,0) = Ep (xx) | - C(Av) Bu)ll
where denotes expectation. However, the model trained with single-step optimization may not satisfactorily perform long-term predictions, since it suffers from the accumulation of prediction errors during recursive propagation. To this end, we train the model to minimize the following multi-step prediction loss:
((A,B,C,O) ED Σ
Cabraal
where D denotes the data set composed of multiple steps input trajectories uo.7 and the resulting state trajectories Po.r. In (3). H denotes the horizon of forward prediction. The distribution of the observable v/z+ is inferred through recursive propagation with the KO based on the current state
I, and series of control inputs 4- We exploit the reparameterization trick (Haarnoja et al.) stable training process (Xu al. 2019). By injecting the observables can be approximated by (zz)) This could produce a lower variance estimator of the gradient, and also enables the straightforward exploitation of KO in forward propagation. The optimization problem is
2018) in the calculation of B to achieve
a Gaussian noise vector E; ~ (O 1), the expectation
N E (xz +
4

---

Published as a conference paper at ICLR 2022
summarized
min C(A, C, ED Σ 100+. C0i+x1l A,B,C,,
(4)
s.t. With = A + Bu++k-1
- vo + (xt), G ~0,11)
The parameters in (4) are updated using gradient descent.
Entropy Constraint To further improve the robustness of the DeSKO model, we introduce con- straint on the entropy of the observable distributions during optimization (4). During the training process probabilistic models, the entropy of the learned distribution naturally falls as the model becomes more certain about the underlying dynamic. However, the lack of data in some states may cause the learned model become overly confident about the encoding. To this end, we prevent the model from overfitting by constraining the average entropy of the learned distribution above a minimum entropy threshold, i.e., Ep log p( th [x,) H. where 7 denotes the minimum entropy threshold. Note that the constraint is valid for the average entropy, rather than a pointwise constraint on the state space, thus the entropy can vary at different states. In our implementation, we exploit the Lagrange method to conduct the constrained optimization. The entropy constraint is multiplied by a Lagrange multiplier and added into the loss function. During the training of NNs, we also adjust the weight of the entropy constraint by updating the Lagrange multiplier using gradient ascent.
4 CONTROL
In this section, we will present the robust control framework for the proposed DeSKO model. We exploit the model predictive control (MPC) framework to achieve robust and stabilizing control guarantees. First, a nominal MPC controller (Borrelli et al. (017) is designed to solve a finite horizon stochastic optimal control problem, minimizing the expectation of cumulative stage cost. As DeSKO model encodes Gaussian distribution of the observables, the nominal dynamic of expectation of observables could be characterized by
ju+ Ajr + Be:
(5)
where fie the nominal mean vector encoded by the DeSKO model and e denotes the nominal control input. Then the nominal MPC solves the following deterministic optimal control problem
H-1
V (Ax) = min Σ
|[C 11 + llc,+1ll + ||C
(6)
coIH-1
k=0
s.t. Pc+++1 Apt+ + Boi+k, CL-R E
(7)
where Q, R and P are known positive definite weight matrices, and r denotes the reference signal.
Ideally, after applying optimized control input ot the system, the mean vector encoded the next time step n (#1+1) should equal to Ait + Bet. However, this can hardly be true due to the existence of noise. The actual dynamic of the mean vector is dominated by
Me+1 = ( (f u)) + we)
(8)
With the learned KO, the evolution the actual mean vector on the observable space is defined as
M4+1 Apr + Bu, + g(w;)
(9)
where g(w:) = u (f ue) wx) - ( uz)) characterizes the unknown effect of w, on the observable dynamics.
To compensate for the control enror caused by the uncertainty, stabilizing feedback controller K introduced to drive the actual state μe to the nominal trajectory fa. At every time step, the action input is given by
Ur c + K ((uo(w)) (ix)
(10)
In this work, the feedback controller K is obtained by using the linear quadratic regulator (LQR).
To ensure the closed-loop stability of the nominal system (5) the following constraint needs to be
satisfied (Borrelli al. 2017 Mayne et al. 2000).
|[C(A A BK 11 |C - -1C l/ i|
(11)
5

---

Published as a conference paper at ICLR 2022
In fact, by exploiting the LQR controller K, this constraint can be easily satisfied. The matrix P is determined by solving the following discrete time Riccati equation,
P P= QC + A PA A PB(R+ B² PB) B PA
(12)
and the K given by
K (R+ BR+
(13)
The overview of the control procedure summarized in Algorithm
4.1 STABILITY WITH EXACT KOOPMAN OPERATORS
First, we investigate the case where the exact Koopman operators are available. To establish the stability guarantee, make the following assumptions:
Assumption 1. The probabilistic NN μ* Lipschitz continuous, [| + y) - po(x))l - Lllyll- Assumption 2. The random noise has bounded energy, i.e,, a finite constant b exists such that Eu Bw]] b.
With the assumptions above, we can establish the stability guarantee follows:
Proposition 1. Consider system (2) controlled by the Robust MPC controller (6- and 10) Then the closed-loop system is uniformly ultimately bounded with bound Lb where 3 denotes the 1-3 maximum eigenvalue of The closed-loop transition matrix Ax = A+ BK and a == ||C|]
The detailed proof of the proposition is deferred the Appendix.
4.2 STABILITY WITH APPROXIMATED KOOPMAN OPERATORS
Proposition shows that the controller is stabilizing with the exact Koopman operator. However, in practice, the exact Koopman matrices A", B", C* are generally infeasible and only the sub-optimal solutions A, B, C' be obtained. Now, will show that stability could be assured even though the Koopman matrices are approximated with bounded prediction error.
First, the nominal system 5 is conceptual system constructed with the approximated Koopman matrices and thus its dynamic and the resulting MPC problem remain the same as (5). (6) and (7). On the other hand, the evolution of the mean vector is characterized with the exact Koopman matrices,
P++1 = A μr + B²u un + g(we)
C* μt
We define the dynamic residual caused by the sub-optimal approximation (A* (A*-A)pp AApp + (B* B) B),,, and the reconstruction residual d, == (C* C)) Then the above system can be rewritten as
Pe+1 Apt: + Bu, + g(w,) +
T3 = Ct4 + de
Assumption 3. There exist positive constants 7.7 E R+ such that ||e]| S and [|]]] S 1)
With the Assumptions
3 establish the stability guarantee as follows:
Proposition Consider system controlled by the Robust MPC controller Then the closed-loop system (2) is uniformly ultimately bounded with bound L where
and 70P.
= |C|]
Detailed proof of the above proposition is referred to Appendix. Proposition shows that with the controller given by (10). the stability of the system (2) can still be guaranteed even with approximated Koopman matrices, though the uniform ultimate bound is inevitably larger than the ideal case in
Proposition I
Remark 1. Assumption is equivalent to assuming that ||A - A 1BB" Bl). and ||C* Cll are bounded respectively, and the state space X. and action space U bounded. To simply the notations, we chose the form in Assumption 3
In addition to the stabilization tasks, we are also interested in how to achieve optimal tracking control with the DeSKO model. By plugging in a reference signal and modifying the input regulation accordingly, the proposed method could be extended to deal with set-point and dynamic tracking problems. The algorithmic and implementation details are detailed in the Appendix.

---

Published as a conference paper at ICLR 2022
Algorithm Robust MPC with DeSKO
Require: Weighting matrices Q, R, state feedback matrix K, prediction horizon H Initialize fi < (0(
1. fort=
do
Solve 6
obtain C
Apply 21 c + K (Ho( - Pe) to the system
Pz+1 Añe Beg
end for
EXPERIMENTS
In this section, we will evaluate the performance of the DeSKO model in terms of modeling, control and robustness. Specifically, evaluate the following aspects: (a) Convergence of the proposed training algorithm with random parameter initialization; (b) Model performance of the DeSKO compared to other baselines in learning and predicting diverse dynamics; (c) Reliability of the control framework achieving successful performance and stability guarantees; (d) Robustness the controller when faced with uncertainties unseen during training, such as external disturbances; and (e) Scalability of the proposed method high-dimensional complex systems.
We illustrate four simulated robotic modeling and control problems to show the general applicability of DeSKO. First of all, the classic control problem of CartPole balancing from the control and Reinforcement Learning (RL) literature (Barto al. 1983) is illustrated. Then, we consider more complicated high-dimensional continuous control problems of robots, such as the legged robot HalfChectah and the soft robotic arm SoPrA. We simulate the HalfCheetah in the MuJoCo physics engine Todorov et al.) 2012) and the SoPrA (oshimitsu et al. 2021) in the DRAKE simulation toolbox (Tedrake & the Drake Development Team 2019). Lastly, we apply DeSKO to autonomous systems cell biology, i.c., biological gene regulatory networks (GRN) (Élowitz Leibler 2000). To further investigate how does DeSKO perform when faced with uncertainty, we introduce process and observation noise to the CartPole and GRN examples, thus four additional variants of the nominal systems are included as benchmarks. The environments are detailed the Appendix. We compare
HP
DUSEO
-4
(a) CartPole
(b) CartPole (Obs Noise) (c)
CartPole(Proeess
(d) GRN
Noise)
R.P
RCP
nEn
ER0
EXO
360
DIEKO
-10
(e) GRN (Obs Noise) (f) GRN (Process Noise)
(g) Halfcheetah
(h) SoPrA
Figure 2: Cumulative prediction error on the validation set. The Y-axis indicates the cumulative mean-squared prediction in log space over 16 time-steps and the X-axis indicates the training time steps. The shadowed region shows the confidence interval (one standard deviation) over 10 random seeds.
the proposed method with three state-of-the-art baseline methods in terms of modeling and control. (1) The Deep Koopman Operator (DKO) (Lusch et al.) [2018] 1.018HHs. Han al. (020) learns a neural network as the observable function and the corresponding Koopman operator for forward propagation. In combination with LQR and MPC, achieves better control performance than reinforcement learning methods such as deep deterministic policy gradient (DDPG) (Lilicrap et al, 2019) on OpenAl gym benchmarks (Brockman al. 2016). (2) An ensemble of ten Multilayer Perceptrons (MLP) (Chua al. 2018) is trained
baselme for modeling. Each model fully connected NN that maps the current state and action to the next state. The models are trained to minimize the cumulative prediction error prediction horizon. The uncertainty could be quantified by the range of
7

---

Published as a conference paper at ICLR 2022
predictions of the models. (3) Soft actor-critic (SAC) Haarnoja et al. 2018) is the state-of-the-ar model-free reinforcement learning algorithm. Even though the sample complexity of model-free approaches is much higher than model- based ones, they can typically converge to solutions with better performance. SAC updates the controller to minimize cumulative stage costs, thus implicitly optimizes for a stabilizing controller. Both SAC and DKO could deal with action constraints explicitly. DKO and MLP also used the multi-step loss for training with the same horizon as DeSKO.
For each environment, a training set composed of 40000 state-action pairs and validation set of 4000 state-action pairs were collected. The actions were collected by uniformly sampling over the action space. Both methods were trained minimize the cumulative prediction error a time horizon of 16, and at each update step, a batch of 256 data-points was randomly sampled for the madient-descent update. The same learning rate 0.001 and decay strategy were used for both methods. SAC iteratively interacts with the environments and updates the control policy. For each environment, 1000k steps of cate-action-reweee pairs were collected for training.
5.1 MODELING EVALUATION
The cumulative prediction error during training on the validation set is shown in Figure 2 As shown in the plots. DeSKO achieved better or the same prediction performance than DKO. In particular, in all of the noisy systems, DeSKO achieved better performance; it also scaled better to high-din nigh-dimensional systems such as HalfCheetah. Nonetheless, MLP achieved better prediction error due to its advantage of huge approximation capacity. However, the key drawback of MLP models the large computational burden in both forward rollout and backward gradients, making it impossible to obtain an analytical optimal controller or action input.
319
NWW
(a) CartPole
(b) CartPole (Ohs Noise) (c) CartPole
(Process
(d) GRN
Noise)
SAC
(e) GRN (Obs Noise)
(I) GRN (Process Noise)
(g) Halfcheetah
(h) SoPrA
Figure 3: State trajectories (a-g) and mean- square tracking error (h). The y-axis indicates the average state trajectory (a-g) or tracking error (h) and the x-axis indicates the time steps in each episode. The shadowed region shows the confidence interval (one standard deviation) over ten random seeds. In (a-g), only the dimensions where reference signal is tracked are shown.
5.2 CONTROL EVALUATION
The control performance of DeSKO and the baselines is compared. A partial state set-point tracking task assigned to each testing environment. Further details can be found in the Appendix.
To evaluate the performance, we observe the angular position of the pole in CartPole, the concentration of protein in GRN, and the speed in the x-direction in Halfcheetah. For the SoPrA, we record the sequence of mean mean-square tracking error, because the reference signal is douesdimensional.
As shown in Figure 3 DeSKO attained the best control performance with low average tracking error
and variances across different trials in (a-f). In (g) and (h), DeSKO performed comparable or slightly worse than SAC. In comparison, DKO failed at (a.b.f) and produced large variances in the state trajectories in the GRN systems (d,e).
5.3 ROBUSTNESS EVALUATION
is well-known that optimal controllers designed for learned models could be fragile when faced with unknown impulsive disturbances. Thus, we are also interested in investigating the robustness of the DeSKO controller when faced with unknown external disturbances. To show this, introduce
8

---

Published as a conference paper at ICLR 2022
periodic external disturbances (every 20 steps) with different magnitudes in the CartPole and GRN systems and observe the performance of each controller. These two environments are chosen as test- beds for robustness due to their higher fragiliity to disturbances when compared to HalfCheetah and SoPrA. In CartPole, the pole may fall over when interfered by an external force, ending the episode in advance. For this reason, we measure the robustness of controllers with the death-rate, i.e., the probability of falling over after being disturbed. For GRN where the episodes are always of the same length, we measure the robustness controllers by the variation in the cumulative tracking error. Under each disturbance magnitude, the policies are tested for 100 trials and the performance is shown in Figure 4
SAC
SAC
SAC
(a) CartPole
(b) CartPolc (Obs Noise)
(c) CartPole (Process Noise)
DHG
Deo
DRO
8
1
50
23
12
20
44
30
291
(d) GRN
(e) GRN (Obs Noise)
(I) GRN (Process Noise)
Figure 4: Death rate and cumulative tracking error in the presence of persistent disturbances with different magnitudes. The x-axis indicates the magnitude of the applied disturbances. The y-axis indicates the death rate in the CartPole systems (a-c) and the cumulative tracking error in log space in GRN systems (d-e). All of the trained policies are evaluated for 100 trials in each setting.
As shown Figure DeSKO attained the best robustness in all of the tests. In the CartPole examples, the DeSKO controller can resist disturbances with magnitude up to five times (100N) of the maximum control input (20N), without any failure (see (a.b)). In (f), the tracking shown in the plots, and the zoomed-our view of (f) can be found in the Appendix.
of DKO is too high to be
5.4 ABLATION ON THE ENTROPY CONSTRAINT
We are curious to what is the effect of the entropy constraint on the robustness of the resulting controller. To show this, we trained three models in CartPole with different minimum entropy thresholds H. and constantly disturb the CartPole to evaluate the robustness of the con- troller. The death rate of the resulting controllers is shown in Figure 5 As observed in Figure increasing the value of H helps the resulting controller to be more robust to unknown disturbances, which validates the effect of the entropy constraint.
Entropy Entrosy Entrosy=10
22c
140
Figure 5: Ablation study on the Entropy threshold H.
6 DISCUSSION AND CONCLUSION
In this paper, we introduced and discussed an efficient model learning approach called the deep stochastic Koopman operator (DeSKO). By using deep neural networks to encode the distribution of observables, the DeSKO model is able to infer and propagate the uncertainty in dynamical systems. We developed robust model predictive control framework based on the learned DeSKO model; the framework guarantees closed-loop stability the controlled systems. Our experiments show that DeSKO can be applied to nigh-dimensional complex nonlinear systems, and it outperforms existing deep Koopman operator models and RL algorithms in terms of modeling and control. Our control experiments showed that DeSKO was more robust than the baselines when faced with large external disturbances.
In the future, it could be interesting to investigate the convergence proof of algorithms based on the deep Koopman operator. New Koopman representations that have nonlinear control inputs are also interesting. Another potential future research topic could be control tasks with state constraints.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank Lixian Zhang, Miriam Filippi, Lewis Jones, Elvis Nava, Mike Yan Michelis, and Hehui Zheng for their useful comments and insights. DeSKO is supported by the program of China Scholarship Council (No.2020061 100055). We also want to thank the generous gift from Credit Suisse the ETH Foundation enabling Soft Robotics Research.
REFERENCES
Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney, Forecasting sequential data using consistent koopman autoencoders. In International Conference on Machine Learning. pp. 475-485, PMLR, 2020.
Craig Bakker, Arnab Bhattacharya, Samrat Chatterjee, Casey J. Perkins, and Matthew R. Oster. The Koopman Operator: Capabilities and Recent Advances. In 2020 Resilience Week (RWS), pp. 34-40, Salt Lake City, ID, USA. October 2020. IEEE. ISBN 778-1-72218-69333 doi: 10.1 109/RWS50334. 2020.9241276. URL
leee wrt//oooggeeoooo /tw.cuummn//241202
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834 846, 1983.
Felix Berkenkamp and Angela Schoellig. Safe and robust learning control with Gaussian processes. In 2015 European Control Conference (ECC), pp. 1496-2501, July 2015. doi: 10. 10.1109/ECC.2015. 7330913.
Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe Model-based Reinforcement Learning with Stability Guarantees. arXiv:1705.08:: 08551 [cs, stat]. November 2017. URL htt.htttp///xxi...oo
wltp///s/s/////.....// 08551 arXiv: 1705.08551.
Homanga Bharadhwaj. Kevin Xie, and Florian Shkurti. Model-predictive control via cross cross-entropy and gradient-based optimization. In Leurning for Dynamics and Control, pp. 277-286. PMLR, 2020.
Francesco Borrelli, Alberto Bemporad, and Manfred Morari. Predictive control for linear and hybrid systems. Cambridge University Press, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01555 2016.
Daniel Bruder, Xun Fu, R. Brent Gillespie, C. David Remy, and Ram Vasudevan. Koopman-based Control Soft Continuum Manipulator Under Variable Loading Conditions. arXiv:2002.0140) [cs], February 2020. URL U http:////// arxiv. org/ abs/ 2002.0002222 014C7
arXiv: 2002.01407.
Gang Cao, Edmund M.-K. Lai, and Fakhrul Alam. Gaussian Process Model Predictive Control of an Unmanned Quadrotor. Journal of Intelligent & Robotic Systems, 18(1):147-162, October 2017. ISSN 0921 -921- 1573-0409. doi: 10. 10000/00146-007-000000
URL http://link
springezs com 10. 1007/00000 10846 017 0549-V
Alexander H. Chang, Christian M. Hubicki, Jeff J. Aguilar, Daniel I. Goldman, Aaron D. Ames, and Patricio A. Vela. Learning to jump in granular media: Unifying optimal control synthesis with Gaussian process-based regression. In 20/7 IEEE International Conference on Robotics and Automation (ICRA), pp. 2154-2160, May 2017. doi: 10.1109/CRA.2011..99997
Alexander H. Chang, Christian M. Hubicki, Jeffrey Aguilar, Daniel I. Goldman, Aaron D. Ames, and Patricio A. Vela. Learning Terrain Dynamics: A Gaussian Process Modeling and Optimal Control Adaptation Framework Applied Robotic Jumping. IEEE Transactions Control Systems Technology, pp. 1-16, 2020. ISSN 1558-0865. doi: 10.1 00.10999CS11222000000966 Conference Name: IEEE Transactions Control Systems Technology.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement leamn ing in handful of trials using probabilistic dynamics models. arXiv preprint 1rX.v:1805.1211 12114, 2018.
10

---

Published as a conference paper at ICLR 2022
Michael B Elowitz and Stanislas Leibler. A synthetic oscillatory network of transcriptional regulators. Nature, 403(6767):355-333) 2000.
C. Folkestad, D. Pastor, and J. W. Burdick. Episodic Koopman Learning of Nonlinear Robot Dynamics with Application to Fast Multirotor Landing. In 2020 IEEE International Conference on Roboties and Automation (ICRA), pp. 9216-9222, May 2020. doi: 10.1
ISSN: 2577-087X.
Thomas George Thuruthel, Yasmin Ansari, Egidio Falotico, and Cecilia Laschi. Control strategies for soft robotic manipulators: A survey, Soft robotics, 5(2):149-163, 2018.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 2455-2460 2467,, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. International Conference on Machine Learning, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, pp.
PMLR, 2019.
David A. Haggerty, Michael Banks, Patrick C. Curtis, Igor Mezić, and Elliot Hawkes. Modeling, Reduction, and Control of a Helically Actuated Inertial Soft Robotic Arm via the Koopman Operator. arXiv:201 1.0093 [cs, eess], November 2020. URL http:://ttttttte www./v..vorr...oooooo org abs/ 2011. 07939 arXiv: 2011.07939.
Jessica B. Hamrick, Andrew J. Ballard, Razvan Pascanu, Oriol Vinyals, Nicolas Heess, and Peter W. Battaglia. Metacontrol for Adaptive Imagination Based Optimization. arXiv: Stti::7700.0661 [cs], May 2017. http::ttt::aaat
117000 02670 arXiv: 1705.02670.
Yiqiang Han, Wenjian Hao, and Umesh Vaidya. Deep Learning of Koopman Representation for Control. arXiv:2010.0PP 07546 [cs, eess], October 2020. URL /ttp:://axxi..o///bbbbbbs 07546 arXiv: 2010.07546.
Juraj Kabzan, Lukas Hewing, Alexander Liniger, and Melanie N. Zeilinger. Learning-Ban Baase Model Predictive Control for Autonomous Racing. IEEE Robotics and Automation Letters, 4(4):3333- 3370, October 2019. ISSN 2377-3766. doi: 10. R19. 292267777..777. Conference Name:
IEEE Robotics and Automation Letters.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6990 [cs], 2017. URLhttt: /ttp://aaxvvvv/va
1412. 6980
Stefan Klus, Péter Koltai, and Christof Schütte. On the numerical approximation of the Perron- Frobenius and Koopman operator. Journal of Computational Dynamics, 3(1): September 2016. ISSN 2158-2491. doi: (99934/jdd220160000 URL BRRLtttpp/aaaxi..rogtp http://aaww....gaa////////////// 05997
arXiv: 1512.05997.
J. Kocijan, R. Murray-Smith, C.F. Rasmussen, and A. Girard. Gaussian process model based predictive control. In Proceedings of the 2004 American Control Conference, volume 3, pp. 2214-2219 vol.3, June 2004. doi: (0.23919/ACC.20) 1383790. ISSN: 0743-1619.
J Zico Kolter and Gaurav Manek. Learning stable deep dynamics models. Advances in Neural Information Processing Systems, 12:11128-11131 2019.
B. O. Koopman. Hamiltonian Systems and Transformation in Hilbert Space. Proceedings of the National Academy of Sciences of the United States of America, 1765)315-318, May 1931a. ISSN 0027-8424. URL https:: /wwww//w....m/mmmo ncbi www.mnnniii...mmmm.mmmmmm.mmm .ih. PMC10760527
ess
Bernard O Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the national academy of sciences of the united states of america, 17(5):315, 1931b.
11

---

Published as a conference paper at ICLR 2022
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning Compositional Koopman Operators for Model-Based Control. arXiv:1910.08264 [cs, math, stat], April 2020.
URL http:hhtpaxxii..oo/ org/ abs 08264
arXiv: 1910.08264.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv: 02971 [es, stat], 2019.
abs/ 1509 02971
Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Deep learning for universal linear embed- dings of nonlinear dynamics, Nature Communications, 91)):955,, December 2018. ISSN 2041- 1723. doi: 10. 10.1038/<41467-018-077777 s4 541467-018-07200-
URL http: /www www.nnnture...m www.eee.ommaae comparicee/m/tt
Giorgos Mamakoukas, Ian Abraham, and Todd D Murphey, Learning data-driven stable koopman operators. arXiv preprint arXiv:2005.0422 2020.
David Q Mayne, James B Rawlings, Christopher V Rao, and Pierre OM Scokaert. Constrained model predictive control: Stability and optimality. Automatica, 36(6):789-4 -114, 2000.
Jeremy Morton, Freddie D. Witherden, Antony Jameson, and Mykel J. Kochenderfer. Deep Dynamical Modeling and Control of Unsteady Fluid Flows. arXiv: 805.07472 [cs], November 2018. URL
httt:://arxii org/ / wwws1805..001.
arXiv: 1805.07472.
Jeremy Morton, Freddie D. Witherden, and Mykel J. Kochenderfer. Deep Variational Koopman Models: Inferring Koopman Observations for Uncertainty-Awaare Dynamics Modeling and Control.
arXiv:1902.0977 [cs, stat], June 2019. URL
arXiv: 1902.09742.
Dominic Müller, Manfred Reichert, and Joachim Herbst. Data-driven modeling and coordination of large process structures. In OTM Confederated International Conferences" On the Move to Meaningful Internet Systems", pp. 131-149. Springer, 2007.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt dynamic, real-world environments through meta-reinforcemment learning. arXiv preprint arXiv: I 1803. 1347, 20.
Anusha Nagabandi, Gregory Kahn, Ronald Fearing, and Sergey Levine. Neural network dynam- ics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018b.
Samuel E. Otto and Clarence W. Rowley. Linearly Recurrent Autoencoder Networks for Learning Dynamics. SIAM Journal on Applied Dynamical Systems, (8(1))558-559) January 2019, ISSN 1536-0040. doi: (0.1137/18M117788
URL.httss://eppe
s iam.a.mmm.. /ttp:///ss...saaaaaoooooooooo/// 1137/
1871177846
Shaowu Pan and Karthik Duraisamy. thysics-informan probabilistic learning of linear embeddings of nonlinear dynamics with guaranteed stability. SIAM Journal on Applied Dynamical Systems, 19 (1):480-509, 2020.
Razvan Pascanu, Yujia Li, Oriol Vinyals, Nicolas Heess, Lars Buesing, Sebastien Racanière, David Reichert, Théophane Weber, Daan Wierstra, and Peter Battaglia. Learning model- planning from scratch. 1707v:1707.061 [cs, stat], July 2017. URL htllhttp:/// pttt:://arxivv org/ /abs////////// /1707
06170 arXiv: 1707.06170.
Sébastien Racanière, Théophane Weber, David P Reichert, Lars Buesing, Arthur Guez, Danilo Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imagination- augmented agents for deep reinforcement learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 5694-5705, 2017.
Peter J. Schmid. Dynamic mode decomposition of numerical and experimental data. Jour- nal of Fluid Mechanics, 656:5-28. August 2010. ISSN 0022-1120, 1469-7645. doi:
10. 10.10177/50221110100000011
JtRL:https:/ http:://wwwwwwwmw/eeeeeee www.cccmbbieeeee www.abiiige..g///ooooooooeeoeooo wttp:////////p/ccc////o/oooo////
Lu Shi and Konstantinos Karydis. Enhancement for robustness of koopman operator- data- driven mobile robotic systems. arXiv preprint 1rX0v:2103.008/2, 2021.
12

---

Published as a conference paper at ICLR 2022
Aivar Sootla, Natalja Strelkowa, Damien Ernst, Mauricio Barahona, and Guy-Bart Stan. On periodic reference tracking using batch-mode reinforcement learning with application to gene regulatory network control. In 52nd IEEE conference on decision and control, pp. 4086-4091. IEEE, 2013,
Natalja Strelkowa and Mauricio Barahona. Switchable genetic oscillator operating in quasi-stable mode. Journal of The Royal Society Interface, 7(48):1071-1082, 2010.
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode decomposition. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1130-1140, 2017.
Russ Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019. U.Lpttts:://rrak mit.eemmiieeee....
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, IEEE, 2012.
Yasunori Toshimitsu, Ki Wan Wong, Thomas Buchner, and Robert Katzschmann. Sopra: Fabrication & dynamical modeling of a scalable soft continuum robotic arm with integrated proprioceptive sensing. arXiv preprint arXiv: 10726, 2021.
Jonathan H. Tu, Clarence W. Rowley, Dirk M. Luchtenburg, Steven L. Brunton, and J. Nathan kutz. On Dynamic Mode Decomposition: Theory and Applications. Journal of Computational Dynamics, 1(2):391 421, 2014. ISSN 2158-2505. doi: 10.3934/jcd.20144 2.31....1. URL http:
0041
arXiv: 1312.0041.
Julia Vinogradska, Bastian Bischoff,, Duy Nguyen-Tuong, Anne Romer, Henner Schmidt, and Jan Peters. Stability controllers for gaussian process forward models. In International Conference on Machine Learning, pp. 545-554, PMLR, 2016.
Sifan Wang. Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. arXiv:2007.144 145277 [cs, math, stat], 2020. URL URL httt:://txxxiii org/ abs/ 2007.14527
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. arXiv preprint 2019,
Matthew O Williams, loannis G. Kevrekidis, and Clarence W. Rowley. A Data-Driven Ap- proximation of the Koopman Operator: Extending Dynamic Mode Decomposition. Jour- nal of Nonlinear Science, 2576):137-11346 December 2015. ISSN 0938-8974, 1432- 1467. doi: 10. 111107770000332----25555 URL httt:://link spr waw...cco... 10.00//000000 80C332-013 000322-15-9258-5
Ming Xu, Matias Quiroz, Robert Kohn, and Scott A Sisson. Variance reduction properties of the reparameterization trick. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 2711-2720 PMLR, 2019.
Enoch Yeung, Zhiyuan Liu, and Nathan O. Hodas. A Koopman Operator Approach for Computing and Balancing Gramians for Discrete Time Nonlinear Systems. In 2018 Annual American Control Conference (ACC). pp. 337-344, Milwaukee, WI, June 2018. IEEE. ISBN 978- 1 5388-5522-666 doi: 10.239 19/AC1.201111 84317388 URL Btllltttss/:/eexpppeeee wwwpppoorrereeeeeee......o wrtg//ocu/eoooutt..oo... 8431738/
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, pp. 7444-7453. PMLR, 2019.
13

---

Published as a conference paper at ICLR 2022
A THE FROBENIUS-PERRON OPERATOR
Unlike KO, the Frobenius-Perie Perron operator propagates the probabilistic distribution of observables, thus could better capture the uncertainty in the dynamic.
The Frobenius-Perron Operator P (X) ( is able to characterize the evolution of probabilistic distributions (Klus et al. 2016), and it is defined as
J Py(a)du(o)= J, g(x)d)(=)
(14)
1(A)
(X, B, (i) is the probabilistic density function (PDF). The measure space (X, B, 1i) composed of the state space X, the o-algebra B, and a (probabilistic) measure μ. The FPO P can be considered a linear, infiniteddimension dynamical system shown in (2) (Klus et al. 2016). Furthermore, the FPO is the adjoint operator of the KO in the following sense Klus et al. 2016):
where y E LI(X) L1 (X,
representation of the nonlinear, dimen-damensional
(vb, Pg) - (())P)((w)))())> (KWS(s))(())dd(()) (Kv,g)
(15)
where (z denotes the duality pairing between the L1 and Loo functions. (15) shows that any probabilistic distribution g can be propagated with the Koopman operator. In this work, we will exploit the above relation to propagate the implicit uncertainty by using the KO.
B PROOF OF PROPOSITION 1
Proof. The stability proof is composed of two parts, the stability of the nominal system and the stability of the enror system between the actual system ( and the nominal system (5). First, prove
the stability of the nominal system (5). By solving the problem (6) ( the optimal control sequence
(Gp 10
(16)
and the resulting optimal state trajectory
[A Pi - P
(17)
at instant are obtained. By appending the control signal produced by the feedback controller K Kit to (16), a suboptimal solution at next time step + given by
the Stiper .. -4 111 K KPt m+)
(18)
and
(Pite-Pe Pe P4
I|c 1je: - P4+ AK :
(19)
where Ax : A + BK denotes the closed-loop transition matrix. Based on this suboptimal solution, we can now prove that the optimal value function V*(ū) decreasing along the trajectory. Due to the suboptimality of (18) and (19). one has
H
(*(+++1 <
a(pi+k klt (Hm) + g(Ai K KP mi) ) + p(Axpit
(20)
1=1
= V + a(pi+Ho Kūt+H)) + p( ( M4+ - g g( cep))
(21)
where qis,, a)= = 1[ |cc 1 cost. Thus it follows that
denotes the stage cost and p (it) [|C || denotes the terminal
(*(+++) v*( < -9(App %(p))
and the optimal value function V*(-) is valid Lyapunov function. Therefore the expectation of the nominal state Er,,- = Ci converges to zero as f -> pc, that is the nominal state is mean square stable.
Second, let's 's consider the dynamics of the error system e := μ4 ji defined by the difference of 9 and 5p. Substituting the controller (10) into the error system, it follows that
C++1 (A + BK)e, g(w.)
(22)
14

---

Published as a conference paper at ICLR 2022
Iterate the dynamics of the enror system 22 from the initial time instance to t with ct+1 the initial instance
Akg(w:) + A +.. +.... Akw(wn A'ei According to Algorithm |
e1 equals to zero as Pu = (a()))) Then the L2 norm of the error state is given by
||e || = [[Akg(w)) + A ((r-)) +
(
v [[ + (1) + .... [Akg(w))]
Σ (1])( + (
+... 9 (g(
Taking the expectation over the random noise w/, and using the fact that the random noise signal at different time instances independently distributed, it follows that
E || - _E (l) - (lll(( (1((())))) 1)]l + ..... + S (g(w)) )ll
According to Assumption (T). we can further infer that
Bl|ex+1| w. well /82 LIF // +... + 8r LB wwll
S BLb + 82 L.b + ................................ + 3
(3 Lb
33
where the second inequality is a direct result of Assumption 2 As / + oc, the expectation of the error state norm bounded by 3Lb 1-3- The state of the original system is given by Ez; = C(.c + Eez), thus
the effect of the error state upon the original state bounded by 9r T-Ø"" where := ||C||. Because the
nominal state is mean square stable and the error between actual and nominal state is bounded, the system (2 is proven to be uniformly ultimately bounded.
C PROOF OF PROPOSITION 2
Proof. As the nominal system remains the same in Proposition 1 the proof for mean square stability of the nominal system is identical well. We would focus on proving the uniform ultimate boundedness of the error system.
In presence of the approximation residuals, the dynamic of the error system e 14 piais given as follows,
C++1 = (A BK)e, + g(w;) +
(23)
Iterate the above equation (22 from the initial time instance to f, one has
Ake11 Aga Aga-1 +...4 Age Akg(wi) + A +... Akg (w1)
C++1
Σ c
Σi (kkww)
(24)
According Algorithm I of the error state is given by
the initial instance e equals to zero as ÅI ns (z())). Then the L2 norm
|ez+1|l
Σ EK
Σ
Akk((w))l
| eall Σ |(w))(
$ Σ 3 + Σ (>1l(())))
Taking the expectation over the random noise wy, it follows that
E
Σ |exll Σ gh [l((w)))l
Σ B E
(B 3T)(Tb 7)
1-8 B
15

---

Published as a conference paper at ICLR 2022
where the second inequality is direct result of Assumption 2 and Assumption As + the expectation of the error state norm is bounded by _Lb+2 Because the nominal state is mean square I-8
stable and the error between the actual and nominal state is bounded by constant (((b+>) the 1-8
system ( is proven be uniformly ultimately bounded.
D ROBUST OPTIMAL TRACKING CONTROL
We concerned with two types of tracking problems, the static set-point tracking problems and the dynamic tracking problems where a time varying reference signal given. We will start by showing how to achieve set-point tracking with the proposed controller.
For the set-point tracking problems, the nominal MPC controller is formulated follows
H-1
min Σ ||Cr++ - -rl2 + ||ee+ -u.li || rlp
(25)
R=0
s.t. P0+++1 Ait+ + BOO+K Cc+b E U
where refers the set-point reference signal and u, is the set-point control input that sustains the state at the reference. Ug could be obtained by solving the following optimization problem
min ||u=|1? Pa,Me
Cū. C(A + Bus) Ci.
where ü.s denotes the unknown nominal set-point observable. Plugging the optimal control input of of (25) into the controller (IO) yields the used control input. The set-point controller is used in the SoPrA and GRN examples, where the reference signal is constant.
For dynamic tracking problems like the locomotion of HalfCheetah, the reference signal r, is time varying and the corresponding MPC is formulated as follows
H-1
(*())) = min Σ
CC retall + ||c++1ll |[C - T1+ 1|
(26)
R=
s.t. fu+++1 = Au + Bei+k, C++b E U
There does not exist set-point control u. thus the input regulation is the same as in (6)
E EXPERIMENTAL SETUP
The experimental evaluation occured in OpenAl Gym (Brockman et al., 2016) or the Drake simula- tor (Tedrake & the Drake Development Team 2019). A snapshot of the adopted environments in this paper can be found in Figure
(8) Cartpole
(b) HalfCheetah
(c) SoPrA
Figure 6: Snapshot of the environments. The CartPole and HaflChectah are simulated in OpenAI gym. The SoPrA soft robotic arm is visualized using Drake.
16

---

Published as a conference paper at ICLR 2022
E. CARTPOLE INVERTED PENDULUM ON A CART
We developed modified version of CartPole in (Brockman et al. 2016) with continuous action space instead discrete action space. The system contains a horizontally moving cart and has an inverted pendulum attached it. The cart is fully actuated while the inverted pendulum is unactuated. In this experiment, the controller is expected to maintain the pendulum in its upright, vertical orientation. The action is the horizontal force applied upon the cart (a E [0 20]). 2-hreshold and Burshhold represents the maximum of position and angle, respectively, WChreshold 10 and Ophreshold 20" The episode ends if [8] > fhrrehhlld and the episodes end in advance. The episodes for control evaluation of length 250. For robustness evaluation in Section [.3] apply impulsive disturbance force F on the cart every 20 steps, of which the magnitude ranges from 80 to 150 and the direction opposite to the direction of control input.
E.2 HALFCHEETAH TWO-LEGGED RUNNING ROBOT
HalfCheetah legged robot locomotion task adapted from OpenAl Gym (Brockman et al. 2016). The task is to control a two-legged simulated robot to run at the speed m/s. The control input is the torque applied on each joint, ranging from -1 to 1. The episodes for control evaluation are of length 200.
To achieve dynamic locomotion, a reference signal is first produced for the DeSKO and DKO controllers. In our case, we trained model-free RL agent using DDPG (Lillicrap et al. 2019) to run forward at the desired speed and record its state trajectory as the reference signal. Nonetheless, this reference signal is suboptimal and could be improved by using model-based planning methods. In the meantime, SAC is trained directly with the reward to rum forward at 1m/s without the need for reference signal.
E.3 SoPRA SOFT CONTINUUM ROBOTIC ARM
SoPrA is pneumatic two-segment soft continuum robotic arm (Toshimitsu et al. (021), built and simulated with the Drake simulation (Tedrake & the Drake Development Team| [2019). The pose of the SoPrA arm is described by just two configuration variables & and # per segment, which are the relative angle of the plane of bending and the curvature, as is shown in Figure 7 In order to eliminate a singularity in the representation, the following parameterization is adopted
0 0 cos ())
0, 0 sin(
and the pose vector g 10,,,, 0g 1z. 2z.2, 2,.2 where the subscripts indicate the indexes of the segments. The state composed of the pose and its derivative, i.e., [q. q). The controller adjusts the pressure in the six air chambers of SoPrA. Each segment contains three air chambers. Further modeling details can be found in (Toshimitsu et al. 2021). The episodes for control evaluation are of length 250.
E.4 SYNTHETIC BIOLOGY GENE REGULATORY NETWORKS
The gene regulatory networks (GRNs) considered here are in the nano-scale and their physical properties are vastly different compared the other examples. Particularly note is that GRNs can exhibit interesting oscillatory behavior.
In this example, we consider a classical dynamical system in systems/synthetie biology which we use illustrate the reference tracking task hand. The GRN a synthetic three-gene regulatory network where the dynamics of mRNAs and proteins follow an oscillatory behavior (Elowitz & Leibler 2000). A discrete-time mathematical description of the GRN, which includes both transcription and
17

---

Published as a conference paper at ICLR 2022
m4
0:/2
0
0,
0
174
y
0,-1
Ø
Figure 7: Configuration space of the two-segment soft robotic arm SoPrA, reproduced from Toshim itsu al. (202T). 0, describes the curvature for each bending segment th describes the relative angle the plane of bending for a segment i. m, is the mass of segment
translation dynamics, is given by the following set of discrete-time equations:
z1 + 21) + dt
(t) +
a1
+ 41
+ Et)),
Ki + xz]()
x2(t ((t+1) = =2()) + dt
(t) +
02
+ U/2
+ Et)),
K₂ + x2(()
23 = 23(1) + dt
(8 +
03
+ u3
+ Et(),,
(27)
K3 + xz](t
<a(1 +)(++1) =4(1) di
(1) + (1) (1) Ea(), (-og0s (2r2(t) (t), (-cyzo(t) (t))) (t)) + (o(t).
25(t =5(() + dt
2t 26 =t) + dt
Here, 3i, 22, 23 (resp. F4, m.) denote the concentrations of the mRNA transcripts (resp. proteins)
of genes 1, 2, and 3, respectively. &. Vi are i.i.d. uniform noise ranging from 1-8, 8). i.e., & ~ u(-6, 8). During training,
0 and for evaluation d set to 0.5 and respectively in Section 5.3 a1, 02 0g denote the maximum promoter strength for their corresponding gene, 71 72, ? denote the mRNA degradation rates, cı, c2, C3 denote the protein degradation rates, 31, $2, B3 denote the protein production rates, and Ki, K2, K3 are the dissociation constants. The set of equations in (7)) corresponds to topology where gene is repressed by gene 2, gene 2 is repressed by gene 3, and gene 3 is repressed by gene 1. dt the discretization time step.
In practice, only the protein concentrations are observed and given as readouts, for instance via fluorescent markers (e.g., green fluorescent protein, GFP or red fluorescent protein, mCherry). The control scheme u; will be implemented by light control signals which can induce the expression of genes through the activation of their photo-sensitive promoters. To simplify the system dynamics and it is usually done for the GRN model (Elowitz.c & Leibler 2000), consider the corresponding parameters of the mRNA and protein dynamics for different genes to be equal. More background on mathematical modeling and control of synthetic biology gene regulatory networks can be referred (Strelkowa Barahona 2010; Sootla al. 2013). In this example, the parameters are as follows:
Vi: K; = 1,a; 1.6, = 0.16, 3: = 0.16, c; = 0.06, dt =1
In Figure 8] single snapshot of the state temporal evolution without is depicted. We uniformly initialized between 0 to 5, i.c., (0) ~ ((, 5), which is the range we train the policy in Section 5 persistent oscillatory behavior is also exhibiting similar to the snapshot in Figure 8
18

---

Published as a conference paper at ICLR 2022
mRNA
15.0
mRNA
mRNA
125
Protein 1
10.0
Protein 2
Protein 3
25
200
Figure 8: A snapshot of the natural oscillatory behavior of a GRN system consisting of 3 genes. The oscillations have period of approximately 150 arbitrary time units. The task is to control the concentration of Protein to track set-point reference signal. The X-axis denotes time and Y-axis denotes the value/concentration of each state.
140000
DeSKO
DKO
SAC
tracking.ed errorr
100000
80000
60000
40000
20000
0.15
0.20
0.30
0.45
magnitude
Figure 9: Zoomed-out view of subfigure (f) in Figure shown above, the tracking error of DKO
controller is significantly fragile to disturbances.
F ADDITIONAL SIMULATION RESULTS
To further show the effectiveness DeSKO, we also inject noises in the Halfcheetah and SoPrA environments and test the modeling and control performance of DeSKO and the baselines.
DX0
DeSKO
1.00
4.70
(a) Halfcheetah (Observation Noise) (b) SoPrA (Observation Noise)
Figure 10: Cumulative prediction error on the validation set. The Y-axis indicates the cumulative
mean- mean-squared squared prediction error in log space over 16 time-steps and the X-axis indicates the training time steps. The shadowed region shows the confidence interval (one standard deviation) over 10 random seeds.
19

---

Published as a conference paper at ICLR 2022
SAC
SAC
DeG
DKD
DusKO
DesKO
200
100
200
(a) Halfchcetah (Ohservation Noise)
(b) SoPPA (Ohservation Noise)
Figure 11: Mean-squar tracking error. The y-axis indicates the tracking error and the x-axis indicates
the time steps taken during training. The shadowed region shows the confidence interval (one standard deviation) over ten random seeds.
In terms of modelling, DeSKO is still consistently better than DKO, and even outperforms MLP on SoPrA with observation noise. In terms of control, DeSKO also shows better performance than DKO and competitive performance with SAC.
G REAL-WORLDD EXPERIMENT
To further validate the applicability of DeSKO to real-world control tasks, we tested its control performance real SoPrA arm (Toshimitsu et al. 2021). as shown in Figure [1]
The arm was built with silicon and has chambers pneumatically driven by series pumps. Each chamber can hold up to 600 millibars of pressure. The state of the arm is characterized by the end position and velocity of each segment, and measured by motion capture system. The control frequency was set to be 100Hz. We collected a data set of 200000 steps of state-action pairs by randomly actuating the chambers, which took about 40 minutes.
Figure 12: Snapshot of the real SoPrA arm.
G.1 RESULTS
We train the DeSKO model and the baselines with the same dataset collected in the previous step. Figure 13 shows that both methods can converge during training, and MLP could still achieve the lowest prediction error. Then, we exploit the learned model to design controller and implement it on the real SoPrA arm on dynamic trajectory tracking problems. The reference trajectories are three letters on the x-y plane (parallel to the ground), and the robot is expected to follow the trajectories with its end-point at the speed of lcm/s. The hyperparameters of the controllers their best performance. As shown in Figure 14 DeSKO outperforms DKO in the tracking accuracy in both scenarios. SAC is not evaluated in the real world, because it requires far more data than the model-bat based methods and could likely damage the robot during exploring.
tuned reach
20

---

Published as a conference paper at ICLR 2022
MLP
DKO
DeSKO
0.2
0.0
50
Figure 13: Cumulative prediction error on the validation set. The Y-axis indicates the cumulative mean-squared prediction error in log space 16 time-steps and the X-axis indicates the training time steps. The shadowed region shows the confidence interval (one standard deviation) over 10 random seeds.
R
40
(a) DeSKO-S
(b) DeSKO-R
(c) DeSKO-L
4.0
(d) DKO-S
(e) DKO-R
(f) DKO-L
Figure 14: Trajectory tracking with DeSKO and DKO. The above figures show the end-poin position of the SoPrA arm on the x-y plane, controlled by the DeSKO and DKO controllers to track the given letter trajectories.
H HYPERPARAMETERS
The data set is split into the training set composed of 40000 data points, and the validation set composed of 4000 data points. Each data point contains the current state and action input, and the resulting state, i.e. {at, ur x++1)- The data is collected by randomly sampling actions from a uniform distribution over the action space. The DeSKO model is trained with stochastic gradient descent. In our implementation, the ADAM solver (Kingma Ba 2017) used for optimization. At each step during training, batch of 256 data points is sampled from the training set and used for the model update.
21

---

Published as a conference paper at ICLR 2022
Table 1: Hyperparameters of DeSKO
Hyperparameters
Value
Size of data set DD
40000
Batch Size
256
Learning rate
1e-3
Prediction horzion
16
Structure of pa(-)
(256,128,64 64)
Structure of (-) Activation function
128,,1)),64)
ReLU
Dimension of observables
20
Entropy threshold H.
-20
22


Reviews:
[{"review_id": "U9zkL1LKCm3", "reviewer": "Reviewer_R938", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n+ The proposed method is elegant and intuitive, and the resulting algorithm practical and easy to implement.\n\n+ The empirical evaluations are extensive and convincing, with the method performing favorably against strong baselines.\n\n+ The paper is well written and very clear\n\nWeaknesses\n\n- The theoretical results implicitly assume that the learned Koopman embedding (encoded in equation (6)) is exact in the absence of noise across the entire state-space.  This is easily seen by setting $w_t=0$ in equation (10).  Of course, this is never the case in practice, as only prediction error bounds on the parameters $(A,B)$ can be obtained, and these are distribution dependent: i.e., once we switch from the exploratory policy used to identify the parameters (such as random noise, as was used in the experiments) to the closed-loop control policy, we have no guarantees that equations (6) and (10) are valid.  This is one of the main challenges that imitation and reinforcement learning must deal with both practically and theoretically.  Therefore while the main theory result is correct, it is only correct under impractical assumptions and implies nothing about stability of the actual closed-loop system.  Further, the assumption that the lifted system can be written to be linear in the control input (i.e., that the control input enters as $Bu_k$ oin equation (1)) is also very strong, and not supported by the theory, see for example, section 3.2.2 of \n@article{korda2018linear,\n  title={Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control},\n  author={Korda, Milan and Mezi{\\'c}, Igor},\n  journal={Automatica},\n  volume={93},\n  pages={149--160},\n  year={2018},\n  publisher={Elsevier}\n}\n\n", "summary_of_the_paper": "This paper exploits the duality between the Perron-Frobenius and Koopman operators to formulate a Deep Stochastic Koopman Operator (DeSKO) to control an unknown system.  In particular, in contrast to prior work in this space that assumes a deterministic system, the paper proposes to learn a distribution over observables (encoded via a probabilistic neural network) and linear dynamics that propagate this distribution forward.  Theoretically, it is shown that under certain assumptions, a model-predictive-controller (MPC) consisting of a nominal policy and a stabilizing feedback gain around the nominal trajectory can guarantee robust stability.  Empirically, extensive experiments are conducted across 8 (4 nominal + 4 perturbed) environments, showing that the proposed approach leads to favorable prediction error (Fig. 2), control performance (Fig.3), and robustness (Fig. 4) when compared to other Koopman and Model-Free RL algorithms.  Further an ablation study on the effect of an entropy constraint that is introduced is also performed (Fig. 5).", "summary_of_the_review": "If the issues with the theoretical statements are adequately addressed, I find this to be a nice paper with convincing empirical results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C6vto9INRvn", "reviewer": "Reviewer_25j7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The learning and control of dynamical system from noisy data is not new, but this paper brings a novel angle into solving this problem - stochastic Koopman operator. The learning of the Koopman model and the proposed robust control provides a strong approach in terms of learning accuracy and reliabilily of controller. The proposed approach outperforms existing models and opens up a new perspective for the important problem of learning and control for dynamical systems. \n\n", "summary_of_the_paper": "This paper leverage the Koopman operator theory to represent a nonlinear dynamics as a high dimensional linear dynamics. The map from the state space to the high-dimensional space as well as the Koopman Matrix (linear transition matrix) are learned from data. Similar ideas has been explored in existing works. The contribution of this work is bringing stochasticity into the learning of Koopman models. The paper assumes the data are noisy and try to uncover a Koopman model with noisy data. Another contribution is the joint learning of a control matrix in the Koopman setting and a robust control framework for the proposed method. The proposed robust control framework is based on model predictive control and is provably stable (thanks to the linear transition of the observables). The proposed method is tested on rigid body systems (CartPole and HalfCheetah), a soft robotic arm (SoPrA) and cell biology (GRN). The proposed method outperform existing Koopman-based method (DKO) and RL method (SAC).", "summary_of_the_review": "Although the building blocks of this paper are not new, the paper leverage these ideas and empirically shows the advantage of the proposed framework. I recommend acceptance of this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B67D8l268be", "reviewer": "Reviewer_GCEV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work presents a new and elegant approach to modeling uncertainty in data-driven dynamical systems methods. The proposed method is simple and easy to code, yet it achieves impressive results on challenging benchmark control tasks. The guaranteed stability is a nice addition to this framework, which further facilitates the design and application of this approach to new tasks. The extensive evaluation of the approach shows its features in comparison to other methods, and independently. Finally, the versailty of framework and its simplicity opens up new possibilities for future research as partially listed by the authors in the discssion section.\n\nMy main concern regarding this work is the somewhat not detailed discussion of related work. Specifically, the following references are missing and should be added:\n\n1. \"Learning Koopman invariant subspaces for dynamic mode decomposition\", Takeishi et al. 2017\n2. \"Physics-informed probabilistic learning of linear embeddings of nonlinear dynamics with guaranteed stability\", Pan et al. 2020\n3. \"Forecasting Sequential Data Using Consistent Koopman Autoencoders\", Azencot et al. 2020\n\nMoreover, I would like to better understand the differences of the current approach with respect to Morton et al. 2019, and Pan et al. 2020. The former work considers distribution of observables, whereas the latter work has stability guarantees. Similarly, the Azencot et al. 2020 work shows several examples with sensor noise, and they also discuss empirical stability.\n\n\nminor comments:\n\t- The reference noa 2016 in Sec. 2 seems to be wrong.\n\t- Please specify the domain and range of A and B in Eq. (1)\n\t- Why $\\mathcal{K}$ is separable in the dynamics and control?\n\t- I believe you should change $L_\\infty$ to $L_2$ below Eq. (3)\n\t- I would move Sec. 3.1 & 3.2 to the appendix. \n\t- The contribution of the Perron-Frobenius to this work is unclear. Please elaborate.\n\t- The subscript/superscript notation in Eq. (5) is inconsistent with Sec. 3.1\n\t- The Figs. 2, 4 & 5, and possibly 3 should use a log-scale on the y-axis.\n\t- Why SAC is not in Fig. 2? Similarly, why MLP is not in Figs. 3 & 4?\n", "summary_of_the_paper": "This paper introduces a new control framework that is based on Koopman theory. A key advantage of the proposed approach is the guaranteed stability, yielding a robust closed-loop controller. The method is evaluated on several challenging control benchmarks, and it is compared to several state-of-the-art approaches. The results highlight the advantages of the approach. In particular, in the setting of large external disurbances, the proposed method achieves the best results. ", "summary_of_the_review": "This paper presents an elegant solution to the modeling of uncertainty in dynamical systems for the puprpose of designing closed-loop controllers. The model is backed by control theory and has a guaranteed stability. The method is evaluated on several tasks showing its properties.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9YYHoV705t", "reviewer": "Reviewer_sstK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of the paper:\n\n(1) Extends the Deep-DMD work by allowing uncertainties in the system dynamics.\n(2) By projecting the system dynamics to linear spaces, one can explore existing optimal control methods such as LQR for linear systems.\n(3) Achieve good control results (i.e. lowest tracking error or and in general more stable) in standard benchmarks. \n\nThe  main  weaknesses of the paper includes:\n\n(1) Can use more experiments to show a more convincing picture. \n\n(2) Compared with SAC, the proposed method does not show a clear edge in many cases. \n\n(3) Explanation of how the Perron-Frobenius operator evolves the system distribution is not entirely clear, i.e. why $f^{-1}(A)$ ?\n\n(4) How does equation 3 change the training process compared with the DKO baseline? I assume it has to do with the expected distribution $E_D$ in equation 4 and 5? Please clarify, and also draw the similarity/difference with DKO side by side. \n\n(5) Is multi-step loss also included in DKO training? If not, I don’t think it is a fair comparison. Please clarify. \n\n", "summary_of_the_paper": "In this paper, authors propose  \"Deep Stochastic Koopman Operator\" which can handle uncertainties in the system dynamics. The authors demonstrate that their approach can produce state loss similar to large capacity MLPs, yet maintains the linear structure in the projected space that is suitable to use linear MPC control methods such as LQR. The proposed method can achieve best tracking performance compared with previous non-stochastic version and SAC trained policies.\n", "summary_of_the_review": "The authors proposed a novel method that allows Deep Koopman Operators to handle noisy systems, and demonstrate that their approach beats various baselines. However, there are not enough experiment varieties, and the results are close to the SAC baseline (not by large margin).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was seen positively by all reviewers. The strength of the paper are:
- Intuitive and interesting combination of Koopman Operators and Optimal Control for Reinforcement Learning
- Convincing experiments on challenging benchmark tasks
- All of the issues of the reviewers (advantages to SAC, gaps in the theory and missing references) have been properly addressed in the rebuttal.

I therefore recommend acceptance of the paper.
Decision: Accept (Poster)