Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors propose  "Deep Stochastic Koopman Operator" which can handle uncertainties in the system dynamics. The authors demonstrate that their approach can produce state loss similar to large capacity MLPs, yet maintains the linear structure in the projected space that is suitable to use linear MPC control methods such as LQR. The proposed method can achieve best tracking performance compared with previous non-stochastic version and SAC trained policies.

main_review: The main strengths of the paper:

(1) Extends the Deep-DMD work by allowing uncertainties in the system dynamics.
(2) By projecting the system dynamics to linear spaces, one can explore existing optimal control methods such as LQR for linear systems.
(3) Achieve good control results (i.e. lowest tracking error or and in general more stable) in standard benchmarks. 

The  main  weaknesses of the paper includes:

(1) Can use more experiments to show a more convincing picture. 

(2) Compared with SAC, the proposed method does not show a clear edge in many cases. 

(3) Explanation of how the Perron-Frobenius operator evolves the system distribution is not entirely clear, i.e. why $f^{-1}(A)$ ?

(4) How does equation 3 change the training process compared with the DKO baseline? I assume it has to do with the expected distribution $E_D$ in equation 4 and 5? Please clarify, and also draw the similarity/difference with DKO side by side. 

(5) Is multi-step loss also included in DKO training? If not, I don’t think it is a fair comparison. Please clarify. 


summary_of_the_review: The authors proposed a novel method that allows Deep Koopman Operators to handle noisy systems, and demonstrate that their approach beats various baselines. However, there are not enough experiment varieties, and the results are close to the SAC baseline (not by large margin).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper leverage the Koopman operator theory to represent a nonlinear dynamics as a high dimensional linear dynamics. The map from the state space to the high-dimensional space as well as the Koopman Matrix (linear transition matrix) are learned from data. Similar ideas has been explored in existing works. The contribution of this work is bringing stochasticity into the learning of Koopman models. The paper assumes the data are noisy and try to uncover a Koopman model with noisy data. Another contribution is the joint learning of a control matrix in the Koopman setting and a robust control framework for the proposed method. The proposed robust control framework is based on model predictive control and is provably stable (thanks to the linear transition of the observables). The proposed method is tested on rigid body systems (CartPole and HalfCheetah), a soft robotic arm (SoPrA) and cell biology (GRN). The proposed method outperform existing Koopman-based method (DKO) and RL method (SAC).
main_review: The learning and control of dynamical system from noisy data is not new, but this paper brings a novel angle into solving this problem - stochastic Koopman operator. The learning of the Koopman model and the proposed robust control provides a strong approach in terms of learning accuracy and reliabilily of controller. The proposed approach outperforms existing models and opens up a new perspective for the important problem of learning and control for dynamical systems. 


summary_of_the_review: Although the building blocks of this paper are not new, the paper leverage these ideas and empirically shows the advantage of the proposed framework. I recommend acceptance of this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper exploits the duality between the Perron-Frobenius and Koopman operators to formulate a Deep Stochastic Koopman Operator (DeSKO) to control an unknown system.  In particular, in contrast to prior work in this space that assumes a deterministic system, the paper proposes to learn a distribution over observables (encoded via a probabilistic neural network) and linear dynamics that propagate this distribution forward.  Theoretically, it is shown that under certain assumptions, a model-predictive-controller (MPC) consisting of a nominal policy and a stabilizing feedback gain around the nominal trajectory can guarantee robust stability.  Empirically, extensive experiments are conducted across 8 (4 nominal + 4 perturbed) environments, showing that the proposed approach leads to favorable prediction error (Fig. 2), control performance (Fig.3), and robustness (Fig. 4) when compared to other Koopman and Model-Free RL algorithms.  Further an ablation study on the effect of an entropy constraint that is introduced is also performed (Fig. 5).
main_review: Strengths

+ The proposed method is elegant and intuitive, and the resulting algorithm practical and easy to implement.

+ The empirical evaluations are extensive and convincing, with the method performing favorably against strong baselines.

+ The paper is well written and very clear

Weaknesses

- The theoretical results implicitly assume that the learned Koopman embedding (encoded in equation (6)) is exact in the absence of noise across the entire state-space.  This is easily seen by setting $w_t=0$ in equation (10).  Of course, this is never the case in practice, as only prediction error bounds on the parameters $(A,B)$ can be obtained, and these are distribution dependent: i.e., once we switch from the exploratory policy used to identify the parameters (such as random noise, as was used in the experiments) to the closed-loop control policy, we have no guarantees that equations (6) and (10) are valid.  This is one of the main challenges that imitation and reinforcement learning must deal with both practically and theoretically.  Therefore while the main theory result is correct, it is only correct under impractical assumptions and implies nothing about stability of the actual closed-loop system.  Further, the assumption that the lifted system can be written to be linear in the control input (i.e., that the control input enters as $Bu_k$ oin equation (1)) is also very strong, and not supported by the theory, see for example, section 3.2.2 of 
@article{korda2018linear,
  title={Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control},
  author={Korda, Milan and Mezi{\'c}, Igor},
  journal={Automatica},
  volume={93},
  pages={149--160},
  year={2018},
  publisher={Elsevier}
}


summary_of_the_review: If the issues with the theoretical statements are adequately addressed, I find this to be a nice paper with convincing empirical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new control framework that is based on Koopman theory. A key advantage of the proposed approach is the guaranteed stability, yielding a robust closed-loop controller. The method is evaluated on several challenging control benchmarks, and it is compared to several state-of-the-art approaches. The results highlight the advantages of the approach. In particular, in the setting of large external disurbances, the proposed method achieves the best results. 
main_review: This work presents a new and elegant approach to modeling uncertainty in data-driven dynamical systems methods. The proposed method is simple and easy to code, yet it achieves impressive results on challenging benchmark control tasks. The guaranteed stability is a nice addition to this framework, which further facilitates the design and application of this approach to new tasks. The extensive evaluation of the approach shows its features in comparison to other methods, and independently. Finally, the versailty of framework and its simplicity opens up new possibilities for future research as partially listed by the authors in the discssion section.

My main concern regarding this work is the somewhat not detailed discussion of related work. Specifically, the following references are missing and should be added:

1. "Learning Koopman invariant subspaces for dynamic mode decomposition", Takeishi et al. 2017
2. "Physics-informed probabilistic learning of linear embeddings of nonlinear dynamics with guaranteed stability", Pan et al. 2020
3. "Forecasting Sequential Data Using Consistent Koopman Autoencoders", Azencot et al. 2020

Moreover, I would like to better understand the differences of the current approach with respect to Morton et al. 2019, and Pan et al. 2020. The former work considers distribution of observables, whereas the latter work has stability guarantees. Similarly, the Azencot et al. 2020 work shows several examples with sensor noise, and they also discuss empirical stability.


minor comments:
	- The reference noa 2016 in Sec. 2 seems to be wrong.
	- Please specify the domain and range of A and B in Eq. (1)
	- Why $\mathcal{K}$ is separable in the dynamics and control?
	- I believe you should change $L_\infty$ to $L_2$ below Eq. (3)
	- I would move Sec. 3.1 & 3.2 to the appendix. 
	- The contribution of the Perron-Frobenius to this work is unclear. Please elaborate.
	- The subscript/superscript notation in Eq. (5) is inconsistent with Sec. 3.1
	- The Figs. 2, 4 & 5, and possibly 3 should use a log-scale on the y-axis.
	- Why SAC is not in Fig. 2? Similarly, why MLP is not in Figs. 3 & 4?

summary_of_the_review: This paper presents an elegant solution to the modeling of uncertainty in dynamical systems for the puprpose of designing closed-loop controllers. The model is backed by control theory and has a guaranteed stability. The method is evaluated on several tasks showing its properties.
review_title: null

Meta-review: The paper was seen positively by all reviewers. The strength of the paper are:
- Intuitive and interesting combination of Koopman Operators and Optimal Control for Reinforcement Learning
- Convincing experiments on challenging benchmark tasks
- All of the issues of the reviewers (advantages to SAC, gaps in the theory and missing references) have been properly addressed in the rebuttal.

I therefore recommend acceptance of the paper.
Decision: Accept (Poster)