Task-Oriented Koopman-Based Control with Contrastive Encoder
Authors: Xubo Lyu
Hanyang Hu
Seth Siriya
Ye Pu
Mo Chen
Venue: CoRL 2023 Oral (2023)
Keywords: Learning and control
Koopman-based control
Represention learning

Abstract: We present task-oriented Koopman-based control that utilizes end-to-end reinforcement learning and contrastive encoder to simultaneously learn the Koopman latent embedding, operator, and associated linear controller within an iterative loop. By prioritizing the task cost as the main objective for controller learning, we reduce the reliance of controller design on a well-identified model, which, for the first time to the best of our knowledge, extends Koopman control from low to high-dimensional, complex nonlinear systems, including pixel-based tasks and a real robot with lidar observations. Code and videos are available: https://sites.google.com/view/kpmlilatsupp/.

Task-Oriented Koopman-Based Control with Contrastive Encoder
Xubo Lyu
Hanyang Hu Simon Fraser University hha1600sfu.co ca
Simon Fraser University xlv@sfu.cm ca
Seth Siriya
Ye Pu
Mo Chen
University Melbourne ssiriya@student, unimelb. edu.
University of Mclbourne ye pu@unimelb..... edu.
Simon Fraser University mochen@cs sfu.
Abstract: We present task-oriented Koopman-based control that utilizes end-to- end reinforcement learning and contrastive encoder simultaneously learn the Koopman latent embedding, operator, and associated linear controller within iterative loop. By prioritizing the task cost the main objective for controller learning, we reduce the reliance of controller design on well-identified model, which, for the first time to the best of our knowledge, extends Koopman control from low to nigh-dimensional, complex nonlinear systems, including pixel- based tasks and a real robot with lidar observations. Code and videos are available here.
Keywords: Learning and control, Koopman-baan based control, Representation learning
1 Introduction
Robot control is crucial and finds applications in various domains. Nonlinear and linear control are two primary approaches for robot control. Nonlinear control [1, 2, 3] is suitable for complex systems when a good nonlinear dynamical model is available. But such a model is not easy to obtain be sophisticated and time-consuming. Linear control [4, 5, 6]
and the nonlinear computation relatively simple implement and computationally efficient for systems with linear dynamics, but can exhibit poor performance instability in realistic systems with highly nonlinear behaviors. Based on Koopman operator theory [7]. Koopman Koasean-based
based control [8, 9, 10, 11] offers a data-driven approach that reconciles the advantages of nonlinear and linear control to address complex robot control problems. transforms (unknown) nonlinear system dynamics into latent space in which the dynamics are (globally) linear. This enables efficient control and prediction of nonlinear systems using linear control theory.
Numerous studies have been done on Koopman Cossean-based based control and they typically follow two-stage model-oriented process [11, 12, 13). The first stage is to identify a Koopman model - that is, a glob- ally linear model - from system data, which involves finding a Koopman operator and its associated embedding function to represent linearly evolving system dynamics the latent space. Classical methods use matrix factorization or solve least-square regression with pre-defined basis functions, while modern methods leverage deep learning techniques [14, 12, 11, 13, 15, 16], such deep ral networks (DNNs) and autoencoder frameworks, to enhance Koopman model approximation. In the second stage, a linear controller is designed over the latent space based on the Koopman model. Various optimal control methods for linear systems, including Linear Quadratic Regulator (LQR) [17, 13, 12] and Model Predictive Control (MPC) [18, 19, 20, 16), have been employed.
The model-oriented approach in the aforementioned works prioritizes Koopman model accuracy for prediction rather than control performance. While allows the model be transferred and reused across different tasks, it has certain limitations. Firstly, it involves sequential two-stage process, where the performance of the controller is highly dependent on the prediction accuracy of the Koop-
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

man model. Thus, slight prediction inaccuracies of the learned model can significantly degrade the subsequent control performance. Secondly, even if the model is perfect, the cost function parameters for the linear controller (e.g. Q and matrices in LQR controller) need careful manual tuning in both observed and latent space in order to have good control performance. These challenges are partic- ularly pronounced in problems with nigh-dimensional state spaces, thus restricting the applicability of Koopman-basee control to low-dimensional scenarios.
Contributions. In this paper, we propose a task-oriented approach with a contrastive encoder for Koopman-based control robotic systems. Unlike existing works that prioritize the Koopman model for prediction, our task-oriented approach emphasizes learning a Koopman model with the intent of yielding superior control performance. To achieve this, we employ an end-to-end rein- forcement learning (RL) framework to simaltaneously learn the Koopman model and its associated linear controller over latent space within single-stage loop. In this framework, we set the mini- mization of the RL task cost to be the primary objective, and the minimization of model prediction егто as an auxiliary objective. This configuration has the potential to alleviate the aforementioned limitations: (1) RL optimization provides a dominant, task-oriented drive for controller update, re- ducing its reliance on accurate model identification, (2) manual tuning of cost function parameters unnecessary as they can learned implicitly along with the controller in the end-to-end loop.
We adopt contrastive encoder the Koopman embedding function to learn the linear latent rep- resentation of the original nonlinear system. In contrast to the commonly-used autoencoder, the contrastive encoder is demonstrated to be preferable alternative, delivering latent embedding that well-suited for end-to-end learning, especially high-dimensional tasks such as pixel-based con- trol. To design the Koopman controller, we develop a differentiable LQR control process to be the linear controller for the Koopman latent system. This controller meadient-optmizzable allowing us integrate it into the end-to-end RL framework and optimize its parameters through gradient back- propagation. We empirically evaluate approach across various simulated tasks, demonstrating superior control performance and accurate Koopman model prediction. We compare approach with two-stage Koopman Koopman-based based control and pure RL, and deploy it on a real robot.
Cortrastive Encader
Kooaman Embediing Function
Ougput
Robot
L-xx
Koopman Model
Stare
Koopman Operaiio
Lreer Labent Dynam System
Sepertics
Control 4
Koopman Operatiua
Environment
Koopmenn Embedking Function
LQR Controller
u:
Pardaded 2011
Skatc
Update Contrelier Params
Gradeent-Reooreed DARE
Cast
Optim ser
LQR Controllr
B
R
Figure 1: Overview of our method. We adopt end-to-end RL framework to simultancously learn Koopman model and its associated controller. The Koopman model includes a contrastive encoder as the embedding function and a linear matrix as the operator. The Koopman controller is integrated into the loop as differentiable L.QR controller that allows for gradient-base updates. We optimize the entire loop by considering the task cost as the primary objective and incorporating contrastive and model prediction losses as auxiliary objectives.
2
Related Work
Koopman-Basse Control. B.O. Koopman [7] laid the foundation for analyzing nonlinear systems through an anfiite-dimensionall linear system via the Koopman operator. Subsequent works pro-

---

posed efficient computation algorithms such as dynamical mode decomposition (DMD) [21, 22] and extended DMD (EDMD) [23, 24] to approximate the Koopman operator from observed time-series data. Recent research has expanded the Koopman operator theory to controlled systems [9, 25], and explored its integration with various control techniques such as LQR [26], MPC [20, 18, 27, 19], pulse control [28]. The emergence of deep learning has further enhanced the learning of Koopman embedding and operator using neural networks and autoencoders [14, 15], enabling their integration with optimal control techniques [11, 12, 16].
Contrastive Representation Learning. Contrastive representation learning has emerged as a prominent approach in self-supervised learning in computer vision and natural language process- ing [29, 30, 31, 32, 33, 34, 35], where it employs an encoder learn latent space where the latent representation of similar sample pairs are proximate while dissimilar pairs are dístant. Recent works have extended contrastive learning RL for robot control. Partieularly, CURL [36] learns visual representation for RL tasks by matching embeddings of two data-augmented versions of the raw pixel observation in temporal sequence. The use of contrastive encoder on RL enables effective robot control directly from high-dimensional pixel observations.
Relations Our Work. Our work falls into the realm of using deep learning for Koopman- based control. In contrast to existing two-stage approaches [11, 12] involving model identification and controller design, Koopman model and controller in task-oriented way. We also draw inspiration from the use of contrastive encoder [36], and specifically tailor it as Koopman embedding function for nonlinear systems with physical states and pixel observations. Our approach enhances the Koopman- based control to be used high-dimensional control tasks beyond traditional low-dimensional settings.
propose single-stage, end-to-end RL loop that simultancously learns the
3 Problem Formulation
Consider an optimal control problem over a nonlinear, controlled dynamical systems
min Uo:T -1
D e cX uR) subject Xt+11 = f (Xx; uk),
where the state evolves each time step k following a dynamical model f. We aim find control sequence Up:r minimize the cumulative cost c(xx, ug) over T time steps. Koopman operator theory [7, 9] allows the lifting of original state and input space X and E U to latent embedding space E Z via set of valued embedding functions (X, U) + Z, where the evolution of latent embedding Ze g(xs, u&) can be globally captured by a linear operator IC, as shown in Eq. (2).
Kg (xg, un) A g (f (xx, uk), u
Identifying the Koopman operator IC as well as the embedding function g is the key to Koopman- based control. In practice, IC is often approximated using a finite-dimensional matrix K, and the choice of g is typically determined through heuristics or learning from data. Recent research [11, 12] has employed neural networks (b(-) to encode state x, and define the Koopman embedding function g(x, u) = (x) u]. Correspondingly, K decoupled into state and control components, denoted by matrices A and B, to account for (x)) and respectively. This results in linear time-invariant system with respect to (x) and u in Eq. (3), facilitating linear system analysis and control synthesis.
Kg(x;, u+) |A B
(xx)
u&] = Ax(x)) + Bu: (xx+1)
(3)
The goal of Koopman-based control is to identify the Koopman operator K = AA BJ the em- bedding function (x)) as well as a linear controller u = π (x) to minimise the total task cost.
4 Method: Task-Oriented Koopman Control with Contrastive Encoder
4.1
Contrastive Encoder as Koopman Embedding Function
Deep neural networks extensively employed flexible and expressive nonlinear approximators for learning Koopman embeddings in a latent space. Inspired by the success of contrastive learning,
3

---

we adopt a contrastive encoder to parameterize the embedding function (-). Specifically, for each state x; in the state set X' = {x, i = 0, 1, 1,2.... we create its associated query sample x and set of key samples x that include positive and negative samples x and {xj 1 + i}. x is generated by using different versions of augmentations X while {x} o i} are generated by applying similar augmentations for all the other states: (x; {xg j * i).
Following [32. 36), we use two separate encoders Vu, and 1h8, to compute the latent embeddings: z = u0 (x²), ), z = Va, (xx) and 2; = Ve, (x5 ). We compute the contrastive loss Lest over RL data batch B based on Eq. (4) to update encoders parameters gg. 6g and W, which is a learnable parameter matrix to measure the similarity between the query and key samples. Two encoders vho and Ve,, are used for contrastive loss computation, but eventually only th, serves as the Koopman embedding function, and we simplify its notation Vag We use t (z, u, 22, r, rr,d) to denote tuple with current and next latent state z', action u, reward -c and done signal d.
exp(z? W2,")
Last = RE log
(4)
exp(z? Wz,) + Ejti exp(2, W2,)
Different encoder structures and augmentation strategies are required to handle system states de- pending on how they are represented. For pixel-based states, we adopt convolutional layers as the encoder structure and apply random cropping for augmentation [32, 36]. For physical states, we utilize fully connected layers the encoder structure and augment the states by adding uniformly distributed, scaled random noise as defined in Eq. (5). x refers to element-wise absolute of x.
Ax U(-1px" x/p H); x
x +
Ax
(5)
4.2 Linear Matrices as Koopman Operator
The Koopman operator describes lincar-evolving system dynamics over the latent embeddings and can be represented by a matrix K. Following Eq. (3), we decompose K into two matrices and B, representing the state and control coefficient of linear latent dynamical system.
1k+1
Az Bug
To learn A and B. optimise model prediction loss C.. which is described by Mcan- -eann-Suareed- Error (MSE) as defined in Eq. (7). a is the latent embedding obtained through contrastive en- coder at k + 1 step. It supervises the predicted latent embedding at the k + step from Eq. (6).
f.
Az
Bu+l?; 2x+1 = =0((x+1)
(7)
4.3 LQR-In-The-Loop Koopman Linear Controller
Given Koopman embeddings z = (x) and its associated linear latent system parameterized
Algorithm 1: Iterative solution of DARE
I: Set total number of iterations
by K [A B) shown in Eq. (6), Koopman- based approaches allow for linear control syn- thesis over latent space Formally, consider the infinite time horizon LQR problem in Koop- man latent space that can be formulated as Eq. (8) where Q and R are state and control cost matrices. In practice, we choose to repre- sent Q and R as diagonal matrices to maintain their symmetry and positive definiteness. The LQR latent reference, denoted as Zyet, can be obtained from ((Xxet) if X provided. Zie can also be set to 0 if X is available. This is particularly useful in cases where the LQR problem does not have explicit, static goal reference, such as controlling the movement of a cheetah.
2: Prepare current A, B. Q, R; initialise PM = Q
3: for m = M, M 1, M 2
do
4: Pm A P m+ A A P B(R
B Pm 10-BBB Pm+AA+Q
5: end for
6: Compute linear gain:
G = (B P BB
7: Generate optimal control for latent embedding z:
u' = -Gz
min
(zz
Zee))
Q (zk Znet) u Ru; subject to z Az; + Bug,
(8)
k=

---

Solving the LQR problem in Eq. (8) involves solving the Discrete-ting time Algebraic Riccati Equation (DARE). One way this can be done to take a standard iterative procedure to recursively update the solution of DARE until convergence, as shown in Algo. 1. In practice, we find performing a small number of iterations, typically M 10, adequate to obtain a satisfactory and efficient approximation for the DARE solution. Thus, build LQR control policy TLQR over Koopman latent embedding while dependent set of parameters A, B, Q R. described by Eq. (9).
u ~ TLQR (z/G] G) - TI.QR T Iz R, - TI (2,,
Together with sho [x), Eq. (9) implies that the Koopman control policy FLQR is differentiable with respect to the parameter group Ω {Q, R. A, B, thu over the input x. Therefore, this process can be readily used in our pradient-based, end-to-end RL framework. During learning, we follow Algo. to dynamically solve an LQR problem (8) at each step & with current parameters Ω to derive a control u, for the robot. To optimize the controller #LQR towards lowering the task-oriented cost, we adopt well-known RL algorithm, soft actor-critic (SAC) [37], to maximize the objective of Eq. (10) via off-policy gradient ascent over data sampled from batch buffer B. Q1 Q2 are two Q-value approximators used in SAC. In principle, any other RL algorithms also be utilized.
Luae RAaB
min Qi(z, u) log T 2) a (x) 1-122
(10)
4.4 End-to-End Learning for Koopman Control
We summarise the previous discus- sions and present the end-to-end learning process for task-oriented Koopman control with contrastive encoder, illustrated in Fig. and Algo. 2. We repeatedly col- lect batches of trajectory data from task environment & and utilise three objectives to update the parameter group Ω (Q, A, }o) at each iteration. We take the RL task loss Lav from Eq. (10) to be the pri- mary objective to optimize all param- eters in Ω for achieving better control performance on the task. We use contrastive learning Lest and model prediction Lm losses from Eq. (4) and Eq. (7) as two auxiliary objectives to regularise the parameter learning. Lest is used to update v-)(-) to ensure contrastive Koopman embedding space, while Lm is used update A, B ensure an accurate Koopman model in the embedding space.
A Algorithm 2: End-to to-End Learning for Koopman Control
1: Initialise Koopman control parameters Q, R, A, A,B, vo
2: Reset task environment E.
3: Initialise a data replay buffer 2).
4: for iteration lj 0, 2,. do
5: Collect new roll-outs T, from by running policy m.or following Algo. 1, and save Tn to Sample batch of data B from D. Compute Lieey Less, L.m based on B.
6:
7:
8:
Update S iQ, R,, A, B, t/e based Lac-
9:
Update whe and A, B based on Ceu and C. respectively.
10:
end for
Cart pole Physical States
-2
(b) Visualization of pixel-based CartPole control.
-4
pale_ sale_angle
-6
cart vel
pole angvel
800
1000
Number Steps
(a) Controlled states of 4D CartPole system
(c) Visualization of cheetah 18D control.
Figure 2: Dynamical system behaviors obtained by learned Koopman controller.
5

---

Simulation Results
We present simulated experiments to mainly address the following questions: (I) Can method achieve desirable Koopman control performance for problems involving different state spaces with different dimensionalities? (2) Are we able to obtain well-fitted globally linear model in the latent space? For all control tasks, we assume the true system models are unknown.
5.1 Task Environments
We include three robotic control tasks with varying dimensions in their state and control spaces from DeepMind Control Suite Simulator [38]: (1) 4D CartPole Swingup. The objective of this task is to swing up cart-attached pole that initially points downwards and maintain its balance. To achieve this, we need to apply proper forces the cart. This task has 4D physical states of cart-pole kinematics as well as ID control. (2) 18D Cheetah Running. The goal of this task is to coordinate the movements of planar cheetah to enable its fast and stable locomotion. It has 18D states describing the kinematics of the cheetah's 's body, joints, and legs. The 6D torques are used as the control to be applied to the cheetah's 's joints. (3) Pixel-Based CartPole Swingup. The CartPole swingup task with the third-person image as state.
5.2 Result Analysis
We report the results in Fig. 2, 3, 4 to demonstrate the effectiveness of our method. Fig. 2 show- cases dynamical system behaviors by running learned Koopman controller. The state evolution and temporal visual snapshots the three tasks illustrate the successful control achieved by our method.Fig. fig. 3 shows the Koopman controller's performance by comparing its evaluation cost with the reference cost various learning stages. The reference cost, obtained from [36], considered the optimal solution to the problem. All experiments are tested over 5 random seeds. Across all three tasks, method can eventually reach within 10% of the reference cost and continues to make fur- ther progress. This indicates our method is generally applicable to both simple, low- www-dimensional systems and very complex systems involving high-dimensional physical and pixel states.
100
Reference Cazt
80
to Bererence
60
80
60
40
an
20
20
250
500
750
1000
1250
10000 Number of Episodes
5000
15000
1000
2000
3000
Number Episodes
Number of Episoodes
4D CartPole swingup
(b) 18D cheetah running
Pixel-based CartPole swingup
Figure 3: Mean and standard deviation of error between reference cost and our controller cost during learning.
nentt
2enext
pred
SNE Dim
10
Dim
ESN
-10
-10-
-20
-20
-20
-20
-20
-20
20
Dim
5SE Dim
(a) 4D CartPole swingup with mean (b) 18D cheetah running with mean (c) Pixel-hased CartPole with mean model 2.72 10
model error: 8.10 10-
model error: 3.71 10"
Figure 4: Distribution maps of 2D data points projected via t-SNE from latent trajectories. z. zenext denotes true trajectories while z.pred denotes predicted trajectories using learned Koopman model.

---

Fig. 4 shows the Koopman model's prediction accuracy in the latent space. We employ t-SNE [39] to project the latent trajectories from a 50D latent space onto a 2D map for improved visualization. The plot in Fig. 4 shows the true and predicted states from trajectories consisting of 1000 steps. Significant overlapping and matching patterns are observed the distribution the data points for the 4D CartPole and 18D cheetah systems. This, plus the model prediction error, indicates the potential of utilizing globally linear latent model capture the state evolution in both simple and highly complex nonlinear systems. However, for pixel-based CartPole control, the projected states do not perfectly match, suggesting difficulties in accurately modeling the pixel space. Nevertheless, our method still achieves good control performance, even with slight modeling inaccuracies. This highlights the advantage of our approach where the controller less affected by the model.
Comparison with Other Methods
6.1 Ours vs. Model-Oriented Koopman Control
We compare our method with model-oriented Koopman control (MO-Kpm), which often requires a two-stage process of Koopman model identification and linear controller design. We compare with the most recent work [12] and conduet analysis through the 4D CartPole-swingup task.
Controller More Robust to Model In- accuracy. Table. presents the perfor- mance of the Koopman controller under varying levels of Koopman model accu- racy. MO-Kpm experiences rapid in- crease in total control cost with slightly increasing model error. In contrast, our method demonstrates superior and con- sistent control performances, indicating its better control quality as well as less dependency on the model's accuracy. This advantage arises from designing the controller primarily based on task-oriented costs rather than relying heavily on the model. Thus, our method is applica- ble not only to low-dimense wiwdimenniionl systems but also to complex and high-dimensional scenarios, such as the cheetah and pixel-based CartPole, where MO-Kpm cannot obtain reasonable control policy.
Model Error
MO MO-Kpm
TO-K Ko- (Ours)
Total Cost
Cost Var
Total Cost
Cost Var
~0
-188.10 18
-872.18
10
100..67
42.75% 40.27%
-846.88
2.90 %
~0
-64.32
-784.01
7.42 %
Table 1: Total control cost and its variation under different lev- els model error using MO-Kpm and method.
MO-Kpm
Total Cost
10
1.25
Q: Diag(84.12 62.07, 65, 0.04, 0.04, O....
188.10
Q2 Diag 10, 10, 0.01 0.01, 0,
-109223
20
Q3 Diag(10, 60, 60, 0.01, 0.01, 0,0,
-70.65
0.75
Q1 = Diag 60, 60, 10, 10, 0.1
-124.80
0.50
TO- 1O-Kpm (Ours)
Total Cost
Q shown in Fig. 5
-846.88
0.25
Table 2: Manually tuned and learned matrices for latent LQR, and their associated control costs.
Figure Learned matrix
Automatic Learning of Q Matrix in Latent Space. One major challenge MO-Kpm is the difficulty in determining the state weight matrix Q for the latent cost function (Eq, (8)), especially for latent dimensions that may not have direct physical meanings. This challenge can lead poor when the identified model is perfect. Table. compares the control costs
control performance, obtained from several manually tuned Q matrices under the best-fitted Koopman model (10-4 level) with the learned Q using our method. Our approach enables automatic learning of Q over latent space and achieves the best control performance.
6.2 Ours vs. CURL
We compare method with CURL [36], model-free RL method that uses contrastive encoder for latent representation learning and neural network policy for control.
7

---

System Analysis using Control Theory. Our method differs from CURL in that we learn linear Koopman model, whereas CURL. does not. The presence of a Koopman model (parameterized by A, B Eq. 6) allows us to analyze the system using classical control theory and provides insights for optimizing the controller design. For the CartPole system, we perform stability analysis on both the 50D latent and the 4D systems, and draw the pole-zero plots in Fig. 6. We find that the learned system demonstrates the same inherent instability as the true system, with the true system's poles accurately reflected in the poles of the latent system (overlapping blue and red dots).
We analyze the controllability of the learned latent system and find its matrix rank to be 6, which indicates that a latent dimension of results in excessive uncontrollable states. Using this informa- tion, we apply our method with lower-dimensional 6D latent space and able to maintain the same control and model performance. Further decreasing the latent dimension to leads degraded control performance, suggesting that the controllability matrix rank is valuable for controller design. This demonstrates the benefit of having interpretable representation of the state space.
Latent Syctem Pries
True System Poles
Latent System Dimensions
Total Cost
Model Error
2
Dim Z) 50
-846.88
7.76 x 10-3
Dim Z) = rank rank(W22) = 6
-834,18
6.3 x 10
Dim (Z)
-253.80
5.4 10
CURL Control Performance
-841
Table 3:
Our method achieves comparable control cost CURL while providing more interpretable information about the system.
Figure 6: Pole-zero plot of true and learned latent CartPole systems.
Zero-Shot Sim-to-Real Evaluation
We deploy our algorithm trained from the Gazebo simulator to the turtlebot3 burger ground robot. We use 2D Lidar measurements as well as the odometry information as observation, and the linear LQR policy generates the linear and angular velocities as control. We aim to control the robot to navigate through narrow curved path without any collisions. We directly transfer the trained policy (which trained with only 40 episodes and each episode contains approximately 700 steps) to the hardware without any fine-tuning, demonstrating the applicability of approach a real robot.
(a) 4s
(b) 9s
(c) 15s
(d) 20s
(c) 26s
(f) 29s
(g) 30s
Figure 7: Snapshots of real robot curved trajectory different time stamps (seconds).
8 Conclusion and Limitations
In this work, we propose task-oriented Koopman- control with contrastive encoder to enable simultaneous learning of the Koopman embedding, model, and controller in iterative loop which extends the application of Koopman theory to high-dimensional, complex systems.
Limitation: End-to-ent RL sometimes suffers from poor data efficiency. Therefore, can be bene- ficial to leverage an identified model from model-oriented approaches to derive a linear controller to initialise end-to-end and improve efficiency. Furthermore, the method has only been validated through simulations and needs hardware deployment for more practical evaluation.

---

Acknowledgments
We thank reviewers for their invaluable feedback and express special appreciation our lab mate Rakesh Shrestha for his help the hardware setup. This project received support from the NSERC Discovery Grants Program, the Canada CIFAR AI Chairs program, and Huawei Tech- nologies Canada Co., Ltd. Ye Pu's research was supported by the Australian Research Council (DE220101527).
References
[1] J.JJ. E. Slotine, W. Li, et al. Applied nonlinear control, volume 199. Prentice hall Englewood Cliffs, NJ, 1991.
[2] B. Lantos and L. Márton. Nonlinear control of vehicles and robots. Springer Science &
Business Media, 2010.
[3] C. Dawson, Z. Qin, S. Gao, and C. Fan. Safe nonlinear control using robust neural Iyapunov- barrier functions. Conference on Robot Learning, pages 1724-1735, Auckland, New Zealand, December 2022. PMLR.
[4] H. Trentelman, A. A. Stoorvogel, and M. Hautus. Control theory for linear systems. Springer Science & Business Media, 2012.
[5] Y. Li, X. Chen, and N. Li. Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. Advances in Neural Information Processing Systems, 32, 2019.
[6] F. Rinaldi, S. Chiesa, and F. Quagliotti. Linear quadratic control for quadrotors wavs dynamics and formation flight. Journal of Intelligent Robotic Systems, 70:203-220 2013.
[7] B. O. Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences, 17(5)311-311, 1931.
[8] B. Jackson, J. H. Lee, Tracy, and Z. Manchester. Data-efficient model learning for control with jacobian-regullrizzed dynamic-mode decomposition. In Conference Robot Learning, pages 2273-2283, Atlanta, GA, November 2023, PMLR.
[9] J. L. Proctor, S. L. Brunton, and J. N. Kutz. Dynamic mode decomposition with control. SIAM Journal on Applied Dynamical Systems, 15(1):142-161. 2016.
[10] J. H. Tu. Dynamic mode decomposition: Theory and applications. PhD thesis, Princeton University, 2013.
[11] Y. Han, W. Hao, and U. Vaidya. Deep learning of koopman representation for control. In 2020 59th IEEE Conference Decision and Control (CDC), pages 1890-1895, Jeju Island, Republic of Korea, December 2020. IEEE.
[12] H. Shi and M. Q.-H. Meng. Deep koopman operator with control for nonlinear systems. IEEE Robotics and Automation Letters, 7(3):7700-7707 2022.
[13] P. Laferrière, S. Laferrière, S. Dahdah, J. R. Forbes, and L. Paull. Deep koopman representation for control over images (dkrci). In 2021 18th Conference on Robots and Vision (CRV), pages 158-164, Burnaby, British Columbia, May 2021. IEEE.
[14] B. Luseh, J. N. Kutz, and S. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature communications, 9(1):4950, November 2018.
[15] Y. Xiao, X. Xu, and Q Lin. Cknet: A convolutional neural network based on koopman operator for modeling latent dynamics from pixels. arXiv preprint wwXiv:2102.10200 2021.

---

[16] B. van der Heijden, L. Ferranti, J. Kober, and R. Babuška. Deepkoco: Efficient latent planning with a task-relevant koopman representation. In 202/ JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 183-189, Prague, Czech Republic, September 2021. IEEE
[17] G. Mamakoukas, M. Castano, X. Tan, and T. Murphey. Local koopman operators for data- driven control of robotic systems. In Robotics: science and systems XV, Freiburg im Breisgau, Germany, June 2019.
[18] Abraham, G. De La Torre, and T. D. Murphey. Model-based control using koopman opera- tors. arXiv preprint wrXiv:1709.01566 2017.
[19] E. Kaiser, J. N. Kutz, and S. L. Brunton. Data-driven discovery of koopman eigenfunctions for control. Machine Learning: Science and Technology, 2(3):035023, 2021.
[20] M. Korda and Mezić. Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control. Automatica, 13:149-160, 2018.
[21] P. J. Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of fluid mechanics, 656:5-28, 2010.
[22] Schmid. Application of the dynamic mode decomposition experimental data. Experi- ments in fluids, 50:1 123 1130, 2011.
[23] M. O. Williams, I. G. Kevrekidis, and C. W. Rowley. A data-driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 15:130711 2015.
[24] M. O. Williams, C. W. Rowley, and I. G. Kevrekidis. A kernel-based approach to data-driven koopman spectral analysis. arXiv preprint 141 2014.
[25] M. O. Williams, M. S. Hemati, T. Dawson, I. G. Kevrekidis, and C. W. Rowley. Extend- ing data-driven koopman analysis to actuated systems. IFAC-PapersOnLine, 49(18):704-709, 2016.
[26] S. L. Brunton, B. W. Brunton, J. L. Proctor, and J. N. Kutz. Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control. PloS one, 11(2): c0150171, 2016.
[27] M. Korda and Mezić. Optimal construction of koopman eigenfunctions for prediction and control. IEEE Transactions Automatic Control, 65(12):5114-5129 2020.
[28] A. Sootla, A. Mauroy, and D. Erast. Optimal control formulation of pulse-based control using koopman operator. Automatica, 91:217-224, 2018.
[29] S. Chopra, R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR 05), volume 1, pages 539-546. IEEE, 2005.
[30] A. v. d. Oord, Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018
[31] T. Chen, S. Korbblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[32] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages
2020.
10

---

[33] K. Kotar, G. Ilharco. L. Schmidt, K. Ehsani, and R. Mottaghi. Contrasting contrastive self- supervised representation learning pipelines. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9949-9959, virtually, October 2021. IEEE.
[34] X. Zhao, T. Du, Y. Wang, J. Yao, and W. Huang. Arel: Enhancing contrastive learning with augmentation-rooust representations. arXiv preprint arXivv2303.01009, 2023.
[35] Lilian Weng. Contrastive Representation Learning. Learning Blog article, 2021-05-31. URL https: // 28 May 2023.
lilianweng,
[36] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650, The Baltimore Convention Center, July 2020. PMLR.
[37] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICML), 2018.
[38] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancg, et al. Deepmind control suite. arXiv preprint arXiv:1801.00699, 2018.
[39] L. Van der Maaten and G. Hinton. Visualizing data using I-sne. Journal of machine learning research, 9(11), 2008.
[40] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, pages 2555-2565, Long Beach, California, US, June 2019, PMLR.
[41] D. Hafner, Lillicrap. M. Norouzi, and Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02/99,, 2020.

---

Appendix
Comparison with Other MBRL Methods
-200
planet
Evaluatten Cost
Fuslustton Codt
-20
Evalustion
-400
-400
400
-600
-600
-600
200
400
800
800
1000
200
400
600 Episodes
800
1000
200
400
000
800
2000
Number Episodes
Episodes
(a) Cart-Pole with Pixel Observation
(b) Cart-Polc with 4D state
(c) Cheetah Running with 18D state
Figure 8: Comparison with well-known model-based RL methods.
To benchmark approach against established moddll-base RL methods, we select two widely rec- ognized methods: PlaNet [40] and Dreamer (Version 2) [41]. This comparison spans three tasks in the main paper. Remarkably, across all three tasks, our method demonstrates clear superiority over PlaNet terms of both data efficiency and peak performance. Additionally, our method achieves competitive performance with Dreamer-v2 in two Cart-Pole experiments, along with comparable data efficiency in cheetah running tasks. It's important to note that this comparable performance is achieved while our method learns globally linear model, whereas Dreamer employs much larger nonlinear network approximate world model. Therefore, approach achieves substantial reduction in both computational demands and structural complexity while not compromising con- trol performance too much. It is also worth noting that our approach stands out from commonly used Model Based RL methods as it's rooted in Koopman theory, offering the advantage of enabling control theory analysis for the system.
Comparison with Other Encoders and Losses
Eveluation Cost
-200
AE recon_pred
Jeceocon
-00
Fualivatio Cost
-10
-400
Exclluation
-20
400
recon proal
-300
-600
-600
-400
-800
-600
-500
predo__pedd
200
600
1000
200
600
1000
2000
5000
4000
Number Episodes
Number Episodes
Number Episedes
(a) Cart-Pole with Pixel Observation
(b) Cart-Pole with 4D state
(c) Cheetah Running with 18D state
Figure Comparison with autoencoder and varying losses.
We perform experiments to validate our selection of the contrastive embedding function. To achieve this, we replace the contrastive encoder with a canonical autoencoder (AE), a commonly used method for learning condensed representations from high-dimensional observations. Specifically, we utilize the AE along with two types of loss functions: one involving only reconstruction loss, and another involving combination of reconstruction and one-step prediction loss. Importantly, keep all other aspects of method unchanged. Results are based on 3 random seeds.
As shown in Figure. 9, our approach achieves comparable control performance with the use of AE. This aligns with the notion that autoencoder excels in reconstructing and representing pixel- based observations. However, when tasks involve non-pixel observations, such as normal states, our approach still maintains significant control efficiency and performance, while the AE-based structure struggles to learn useful policy with ample data. Particularly, observed that the AE with only reconstruction loss slightly outperforms the one employing the combined loss, but still falls
12

---

short of achieving the performance obtained by our method using the contrastive encoder. These results provide validation for our choice of contrastive embedding function within our approach.
Ablations Study of Hyper-parameters
We undertook ablation study involving key parameters of the LQR solving iteration and Koopman embedding dimension. Results are the mean over random seeds and summarized in Table. 4.
We found that our approach remains robust regardless the specific number of iterations used for the LQR solution, as long as falls within a reasonable range. This suggests that achieving certain level of precision in solving the Riccati Equation contributes positively both policy and linear model learning. We also studied the impact of varying the latent embedding dimensions for the encoder. Our findings indicate that using smaller dimension that aligns with the encoder's intermediate layers yields consistently good results, while excessively increasing the embedding di- mension (d=100) can diminish control performance. One reason could be the reduced approximation capabilities of the encoder due to inappropriate high latent dimension. might also be because the latent state becoming overly sparse and failing to capture crucial information for effective control.
LQR iteration
Latent Dimension
iter=3
iter=
iter=10
d=30
d=50#
d=100
control cost
-863
-873
-835
-848
-873
-813
cartpole pixel
model error
0.075
0.058
0,269
0.165
0.058
0.038
control cost
-751
-762
-769
-777
-762
-667
cartpole state
model error
0.00047
0.00031
0.00042
0.00043
0.00031
0.0002
control cost
-436
-464
-448
-477
-464
-235
chectah run
model error
0.732
0.862
0.842
0.633
0.862
0.046
Table 4: Ablation results of final cost and model-fitiing error regarding LQR solving iteration and Koopman latent state dimension. The asterisk refers the parameter used in the main paper.
Interpretable Q Matrix in Latent Space.
One key distinction of our method from CURL [36] is the utilization of structured LQR policy in the latent space. In Fig. 10, we illustrate that the LQR policy parameters, especially the Q matrix, can capture the relative signifi- cance weights of latent embedding and their relationship to the original pixel states. We take the pixel-based Cart- Pole task as an example. The larger diagonal elements in the learned Q matrix correspond to visual patches that contain the CartPole object, which provides interpretable information that captures useful latent information related to the CartPole object's s area the image
1.0
10-
0.1
20
0.01
30
0.001
40
0.0001
0
10
20
30
40
Figure 10: Relations of learned weights in
latent Q matrix and original pixel states
13


Reviews:
[{"review_id": "nJSr4T6O7c", "reviewer": "Reviewer_rfrp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel task-oriented approach using a contrastive encoder for Koopman-based control of robotic systems.  \n\nThey employ an end-to-end reinforcement learning (RL) framework, simultaneously learning the Koopman model and its associated linear controller over latent space within a single-stage loop. The primary objective is to minimize the RL task cost, with model prediction error minimization as an auxiliary objective. \n\nIt use a contrastive encoder to learn the linear latent representation of the nonlinear system. They design the Koopman controller with a differentiable LQR solution process, allowing gradient optimization within the end-to-end RL framework for controller parameter optimization. \n\nThe paper presents empirical evaluations through simulations on various tasks and is compared with two-stage Koopman-based control and pure RL methods, offering a comprehensive assessment of its effectiveness.", "summary_of_recommendation": "The idea of implementing task performance oriented koopman control is intersting and has certain novelty. However, the evaluation tasks are relatively simple and lacks comparison experiments and analysis. I would recommend a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Q1 - The RL optimization part is unclear. The primary optimization objective Eq.10 is a typical max-entropy RL objective, while the paper describes the RL optimization as using SAC and it brings confusion. SAC is a typical off-policy actor-critic RL, but here it seems it is optimized in an on-policy manner by using the latent linear dynamics? If so, I think it is just optimized using the basic max-entropy RL instead of SAC. It would be nice to illustrate the RL optimization part more clearly in the appendix.\n\nQ2 - Is it possible to keep matrices A and B fixed during the RL optimization, such that A and B are only optimized for latent dynamics and Q, R is optimized for the task? If jointly optimizing \\OMEGA together, dynamic matrices A and B might also get affected. \n\nQ3 - As Koopman control essentially is a model-based solution, some comparison with other model-based RL solutions might be needed to see how beneficial if we take a data-driven Koopman control-based solution than simple model-based RLs.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The idea of applying koopman control for RL is not original, but the author propose to use RL simultaneously learn the Koopman model and its associated linear controller  instead  of only focusing on the accuracy of the dynamic model. The idea which is simple and effective that makes the model-based controller robust to the dynamic model's ion.\n\nadvantages:\n- Improves the model-oriented koopman control method's performance. The controller is robust to the learned dynamic model's prediction error. \n\nweakeness:\n- The experiment are evaluated on a limited selections of tasks, only 2 simple deepmind control suite tasks and one imaged-based cartpole control task. The dynamics and action dimensions are relatively simple in these tasks.\n\n- Lacks detailed analysis and comparison experiments. For example, this work does not compare the method with general model-based RLs (e.g. dreamver, etc). It neither presents the sample efficiency as it is important for model-based RLs.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "UTO1gFJ7fW", "reviewer": "Reviewer_n3f2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a method for learning control policies based on task-oriented Koopman operators. The main idea being that prioritizing Koopman operator model fidelity around the task has higher performance than simply emphasizing prediction loss of the Koopman operator. The main result would be the ability to construct linear control based on the Koopman operator around the task rather than over the whole state-space. The approach uses a contrastive encoder to construct the Koopman function embedding and integrates an LQR controller learning into the learning loop.", "summary_of_recommendation": "The work demonstrates a promising direction within robot learning that should be highlighted. However, the current work lacks sufficient robotics implementation. As such a weak accept is recommended as the work has potential, but requires more thorough robotics results and analysis to make a proper impact.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "What happens if you let the LQR controller iterate longer? \n\nDoes the time horizon of the problem matter?\n\nHow stable is the back prop through the LQR controller? Do we get divergence?\n\nWhat are key limitations of the work regarding implementation and prior knowledge?\n\nDo the specific nonlinearities of the embeddings matter in reconstructing the latent Koopman space or is all the improvement on the contrastive loss?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a novel approach to the problem of model-based control using Koopman operators. The concept of the contrastive encoder to model the Koopman embedding function is quite interesting and the interpretability of Koopman operator oriented around tasks rather than prediction error is an approach that has valuable merit in this area. \n\nA major strength of this work is enabling control-based interpretability of learned RL systems. This is often not a property of existing methods and should be highlighted. The ability to further adapt the LQR control problem in the loop is quite novel as a big issue with Koopman control is the ability to appropriately tune controllers based in the embedded space. \n\nHowever, one key weakness of the work is motivating analysis and experimental validation as to why the contrastive embedding is preferred over other canonical choices (e.g., normal MLP prediction loss). In fact one can perhaps claim that the true benefit is the task-oriented learning achieved with SAC that is key to ensuring the algorithm works well. As an improvement it is key that these kinds of choices have obvious empirical and theoretical backing, otherwise the choice of this approach seems arbitrary. It would benefit the paper to have such evidence that this is the right approach and what the expected improvements are so that one can validate the hypothesis experimentally. \n\nAnother minor weakness that can be readily addressed is the lack of an ablation study on the parameters e.g., prediction time T, Koopman embedding dimension, etc. These are valuable as it strengthens the reproducibility of the work and highlights limitations of the method and future directions.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "QGwIXVguE87", "reviewer": "Reviewer_XqmK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a novel method for task-oriented koopman based control. The proposed method utilises end-to-end reinforcement learning and contrastive encoder to simultaneously learn the Koopman latent embedding operator and associated linear controller. Overall, the paper is written very well but it is evaluated purely in simulation. Having said that, the provided evaluation and analysis is very comprehensive and overall pleasant to read. The proposed method has the potential to be scaled to physical systems and therefore I recommend it for acceptance. A disclaimer here is that I am not an expert on Koopman based control.", "summary_of_recommendation": "Overall I enjoyed reading this paper. I did not like that it does not have any physical world experiments but regardless I see the proposed approach to have potential for being impactful.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- This method seems scalable to real world scenarios, why are there no physical world evaluations. With those, I will be happy to increase my score.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths and Weaknesses\n### Strengths \n- well written and evaluated paper\n- an interesting idea\n- good results including on pixel based tasks\n\n### Weaknesses\n- no real world evaluation\n- no clear indication for such attempts\n- potentially very difficult to scale to complex physical tasks, e.g. like soft object manipulation", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "HEmVdGfBBk", "reviewer": "Reviewer_kvw1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a Model Based RL approach based on Koopman control.The paper proposes a contrastive learning approach to learn the koopman embedding as well as the linear dynamical system within the latent space. this setup can then allow to use classical control techniques to solve the control problem. The paper is clearly written and easy to understand the different parts of the paper and the algorithms. The experimental evaluation is fully in simulation on fairly simple control suite tasks. The results show that the proposed approach reaches near optimum performance on the tasks as well as decent predicting of the state space.", "summary_of_recommendation": "given the lack of more realistic experimental evaluation I believe this paper is not ready for publication at CoRL yet.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "see above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "the paper is clearly written and easy to follow and understand the algorithmic setup. I also think the approach is interesting and to a certain degree novel in the sense that it stitches together different components coming from optimal control and ML. The end to end approach is interesting especially also the structure around learning the contrastive embedding and linearising the system this way. However the evolution of the method is not mature enough yet for publication. besides of the evaluation being purely in simulation and on fairly easy tasks the paper also lacks comparison to more baselines. Other MBRL baselines, especially ones that combine learning approaches with optimal control should be compared against, it would be interesting to compare in terms of actual task performance to see whether there is improved performance and also whether there is a benefit in sample efficiency.  The paper only compares to total control cost which is a bit harder to interpret in terms of how useful the approach is.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a task-oriented koopman based control that features end-to-end RL and conrastive encoder to learn the Koopman latent embedding oeperator. 

The reviewer found that that the paper presents an interesting and novel ideas in a well-presented manner. The evaluation part is pointed out relatively weak, but the rebuttal better clarifies on this aspect.
Decision: Accept (Oral)