RoboKoop: Efficient Control Conditioned Representations from Visual Input in Robotics using Koopman Operator
Authors: Hemant Kumawat
Biswadeep Chakraborty
Saibal Mukhopadhyay
Venue: CoRL 2024 (2024)
Keywords: Feature extraction
Task Feedback
Control

Abstract: Developing agents that can perform complex control tasks from high-dimensional observations is a core ability of autonomous agents that requires underlying robust task control policies and adapting the underlying visual representations to the task. Most existing policies need a lot of training samples and treat this problem from the lens of two-stage learning with a controller learned on top of pre-trained vision models. We approach this problem from the lens of Koopman theory and learn visual representations from robotic agents conditioned on specific downstream tasks in the context of learning stabilizing control for the agent. We introduce a Contrastive Spectral Koopman Embedding network that allows us to learn efficient linearized visual representations from the agent's visual data in a high dimensional latent space and utilizes reinforcement learning to perform off-policy control on top of the extracted representations with a linear controller. Our method enhances stability and control in gradient dynamics over time, significantly outperforming existing approaches by improving efficiency and accuracy in learning task policies over extended horizons.

RoboKoop: Efficient Control Conditioned Representations from Visual Input in Robotics using Koopman Operator
Hemant Kumawat, Biswadeep Chakraborty, and Saibal Mukhopadhyay Georgia Institute of Technology (hkumawat6, biswadeep, smukhopadhyay6) @gatech. edu
A bsstract: Developing agents that can perform complex control tasks from high- dimensional observations is core ability of autonomous agents that requires un- derlying robust task control policies and adapting the underlying visual represen- tations to the task. Most existing policies need a lot of training samples and treat this problem from the lens of two-stage learning with a controller learned on top of pre-trained vision models. We approach this problem from the lens of Koopman theory and learn visual representations from robotic agents conditioned cific downstream tasks in the context of learning stabilizing control for the agent. We introduce Contrastive Spectral Koopman Embedding network that allows us to learn efficient linearized visual representations from the agent's visual data in a high dimensional latent space and utilizes reinforcement learning to perform off-policy control on top of the extracted representations with a linear controller. Our method enhances stability and control in gradient dynamics over time, signif- icantly outperforming existing approaches by improving efficiency and accuracy in learning task policies over extended horizons.
spe-
Keywords: Feature extraction, Task Feedback, Koopman Operator
Introduction
Building agents capable executing intricate control tasks using high dith--iimesional inputs, like pix- els is crucial in many real-world applications. Combining deep learning alongside reinforcement learning [1, 2, 31, various methods have been developed for learning representations from visual data in robotics for executing intricate control tasks. Most of these learning algorithms can be fundamen- tally organized into two interconneeted yet distinct research directions (refer to Figure 1 and Table 1): (1) Developing visual predictive models that learn the underlying dynamics [4, 5, 6, 7] in latent space by predicting the future visual observations. (Figure 1-A ) (2) Using deep RL to perform tasks with non-linear modeling in an interactive environment(8 9, 10] (Figure 1-B). The first approach involves creating predictive models of the environment which are then leveraged for tasks or gener- ating samples, which model-free methods can then use for learning. However, these models could have sub-optimal task performance. They may need either task-visual alignment [11, 12, 13] where pre-trained vision model's parameters are adapted the task via end-to-end training or manual task and control parameters tuning achieve good task performance. Alternatively, RL-based methods leam the representation by self-supervised auxiliary tasks but they may need huge amount of inter- actions with the environment as the dynamics are learned implicitly the form of policy networks. In general, all these approaches are not sample efficient i.e., they need to collect a large number of samples from environment interactions to achieve satisfactory task performance. Additionally, these models often use computationally expensive latent dynamics models like transformers, MLP, and RNN.
In our work, we aim to learn task-conditioned representations (Figure 1-C) from visual observa- tions in a sample-efficient way without sacrificing task accuracy by modeling the dynamics of the representations linearly. An important advantage linear systems is their generalized math-
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

ematical frameworks, in contrast to nonlinear systems which have no overarching mathematical frameworks for general characterization of systems. Linear systems are very well defined by their spectral decomposition, enabling the development of generalizable and efficient control algorithms. Prior works [16] have explored learning high semantic information from visual input with latent dynamics governed by dense Koopman. However such models are proven [17][18] to be computationally inefficient and require a large number of samples due to bigger parametric space with no stability guarantees of the sys- tem. contrast, we model the dynamies of the task embedding space with spectral Koopman decomposition which is proven learn rich linear representations for time series model- ing while also being computationally efficient. We hypothesize that if we can linearly model the task embedding space and identify a finite subset of relevant, stable (negative) eigenval- ues of the Koopman operator that governs its dynamics while conditioned on lask, we can learn rich agent representations for the task with lesser number of environmental interac- tions while also allowing to perform ef- ficient parallelized forward latent prediction. Our main contributions are as follows: (a.) We propose novel contrastive spectral Koopman encoder to map the visual input complex- valued task embedding space with dynamics governed by a learnable spectral Koopman op- erator. We then use reinforcement learning to learn this embedding space, spectral Koop- man operator, and its associated linear task con- troller with prediction as an auxiliary task. Our network is sample efficient and has better task performance compared to prior works as shown in Table 1. (b.) We conduct theoretical anal- ysis and examine the convergence behavior of our method and show that our method converges to the optimal task policy given sufficient environ- ment interactions. (c.) Through empirical tests on six simulated robotic tasks from deepmind control environment 19], we demonstrate our model's superior performance and sample efficiency against nonlinear and state dynamical models while being robust to sensory errors and external disturbances.
Ooservations
Obsensations
Robot Policy
Stage I: Visual Model Training conditined on prodiction
Stage il: Contral policy based trained vision model
A. Two Stage Polley Networks
Visual Ohservatios
Goal Chaaraatins
Taek Embedding Search
Pollcy
11 Robel Foliyy
Value Function
Task Embedding Space
Traditional RL Methods
Visual Observations
Goal Observatinn
Tesh Embedding Bearch
Task Conditioning
Ratee Polly
Task Embedding Space
Task Feedbback
Figure 1: A. Prediciion-conditineed vision mod- els used to learn task policies for robots. [14] B. Traditional RL methods [15] C. Our method where the visual representations are conditioned on the task policy.
Table 1: Evaluation of Task Performance in Reinforcement Learning: Linear vs. Non-Linear Mod- els. Task performance is visually indicated by arrows, with green up arrows denoting superior performance and red down arrows indicating inferior performance. The extent of performance vari- ation, either improvement or degradation, is qualitatively represented by the number of arrows.
Sample Efficiency
Task Performance
Robust to External Disturhances
Theoretical
Model
Dy Dynaiiss Model Type
Prior Work
Analysis
Predictive
Dense
[17, 23)
Model
Koopman
Nne-L Modele: MLP.
17, 24, 25, 26, 27, 28)
x
141
x
Lagrangian, Homilroiian
Models
Noa- Linea: MLP. Transformer,
(29, 30, 31,
1
Tink Conditioned
Dense Koopmam
[16] Ours
X
111
X
X
Models
Spectall Kroeman
111

---

2 Background
Contrastive Learning: Contrastive Learning has emerged as a powerful framework for learning representations that capture similarity constraints within datasets when there are no data labels. operates on principle analogous to performing a search within a dictionary, wherein the task involves identifying matches and mismatches akin to positive and negative pairs-as if they were keys specific query anchor point [32, 33, 9, 34, 35]. Methods like Curl [9] and To-KPM [16] employ contrastive encoders to directly learn representations from images and control the system. However, these methods require large number of samples (approximately 1000k) to achieve good task performance, while Curl attempts to learn the task without any explicit dynamic model [9].
Dynamical Models: Prior works model system dynamics from visual observations through deep learning networks, mapping high-dimensional observation spaces latent embeddings. These networks learn latent embedding evolution using methods like MLP[7], RNNs[36] neural ODE[37, 38), enabling accurate future state predictions by extracting relevant features and under- standing state relations. Methods like HNN [28] employ Hamiltonian dynamics, while Lag-VAB [24, 25] utilize Lagrangian dynamics for long-term prediction and control. However, these ap- proaches often face low prediction accuracy and require system state supervision, such as prior object segmentation [25, 39, 27, 26), to enhance Lagrangian model accuracy.
Self-supervised learning and RL: Multiple works [40, 41, 42] have utilized self-supervised learn- ing to develop world predictive models, which are subsequently employed for control tasks or spe- cific tasks. In [7], the authors learn a lincarized latent space, which is then controlled using optimal control methods. However, there is no feedback from the control to guide the task in learning control-oriented features. Reinforcement Learning (RL) based methods [43, 44, 45] for directly controlling systems from pixels have witnessed sigmificant advancements in recent years, leverag- ing deep learning to interpret complex visual inputs. However, one of the significant challenges in RL, especially when applied to environments with nigh-dimensional input spaces such as images (pixels), is sample inefficiency. This inefficiency leads to the requirement for an excessively large number of interactions with the environment for the agent to learn an effective policy.
3 Proposed Model
3.1 Problem Formulation
We consider an unknown time-invariant dynamical system of the form:
F(s(t), u(t))
where (l) E S IR" uul)= E 24 RT and & (O, Σg) are the system state, control input and system noise respectively. Function F(s(t), ut)) S l - S governs the transition of the states of the dynamical system and is assumed to be arbitrary, smooth and non-linear. We assume that we can only observe the system through the visual depictions x(t.. Our objective is to identify control sequence up.7 that minimizes the cumulative task cost function c(Tkk Ux.) over time steps by learning an implicit embedding function space. Further, the evolution of latent variable & g(z, u).g x U - Z is parameterized by neural network A. The network A is constrained to follow linear dynamics with a Koopman operator K. given by ( M1+1) (a u.) Azt + Bu, For control problems, task cost function as the optimal control problem in latent space given by: min
+ Z that maps the pixel space to some latent
formulate the Σ c(22, ug)
uo:7-1
subject to 2411 = A + Bug. We aim to Iearn this Koopman Operator and linear control policy u = (s) with the Koopman embedding function mapping the high dimensional input to latent space conditioned on the control cost the system.
3.2 RoboKoop Model Design
Figure 2 presents our overall methodology for learning a parameterized, linear Koopman embed- ding manifold via contrastive spectral Koopman encoder. It generates key and query samples corresponding to each observation at time t. Positive samples are created by applying image aug-
3

---

Constrastive Spectral Koopman Encoder Task Optimizer
k (oz)
Key Encoder
Contrastive
Data Aug
Loss
q= fn
Prediction Loss
Observation
Query
Encoder
Og
Control Encoder
Environment
LQR Controller
Reward
Soft Critic Loss
Control
DARE Insjectories
RL Optimzer
Cost
Figure 2: Our model RoboKoop with contrastive spectral Koopman encoding and RL guided control conditioning
mentations, such as random cropping [9], to the state 2t, whereas negative samples result from applying analogous augmentations to other states except zt. These samples are further processed by dedicated key and query encoders, which update their parameters based on the similarity mea- sure between the samples. The output of the query encoder is then used map the visual space to Koopman embedding space by bifurcating the query vector into real and imaginary components. This introduces complex-valle Koopman embedding space, characterized by spectral Koopman operator with learnable eigenvalues expressed as + jw. Utilizing this embedding space and its spectral Koopman operator, our approach focuses on deriving optimal control strategies and their parameters through the iterative optimization of Linear Quadratic Regulator (I.QR) problem for predefined finite time horizon, guiding the system towards a specified goal state. This goal state is also provided in visual space and subsequently mapped the Koopman embedding space through construct value functions through dual Q-value functions, serving
the key encoder. Moreover, as critics within the Soft Actor-Critic (SAC) algorithm framework. Model updates are performed following the SAC methodology [15], grounded on the cost incurred by the LQR controller. For training, we iteratively gather data batches from the environment, applying three specific losses to train our models: 1) SAC loss for the critic and Koopman parameters, 2) Contrastive loss for optimizing contrastive encoder parameters, and 3) Next latent prediction loss for regularizing the Koopman embedding space dynamics.
a.) Contrastive Spectral Koopman Representations In our work, we diverge from traditional tem- poral modeling techniques, which typically process a sequence of consecutive frames to understand scene dynamics [46]. Instead, inspired by contrastive learning approaches [9] [16]. we introduce a dual-encoder framework consisting of key and query encoders to extract nuanced visual represen- tations. Each encoder processes high-ding nigh-dimensional inputs to generate corresponding sets of key and query samples. To enrich the diversity of these samples, we employ random crop data augmenta- tion technique each input sequence X [zz]i 0, negative samples from the augmented states of X {ar}. These samples are then embedded via the encoders to produce embeddings =/. z;+ and 2,-
creating positive samples x, and
Building on this, we map the query embeddings into a complex Koopman embedding space, by splitting the query embedding vector into real and imaginary components, model continuous latent dynamics. This is complemented by control embedding that maps robot control inputs to complex- valued Koopman control space. We postulate that the latent and control spaces within it are

---

independent, adhering to model of continuous latent dynamics. This is mathematically represented as dz = (1) + Lu((), where the evolution of the system over time can be resolved through df
.o (((+=)) Lu(l)dl. Here, eKs denotes the matrix exponential. For practical applications, this continuous model can be converted into discrete form as 4t+1 = Kr, + Lu,, achieved by applying Zero-Order Hold (ZOH) discretization to the original equations, resulting in K = exp(tttt and L = (K)- (exp(Att) L. In scenarios where observations are uniformly sampled over time, this approach incorporates a learnable time step parameter A/ for adjustment.
=(8) - =
To enhance efficiency in latent space forecasting, we introduce a diagonalized Koopman matrix, K diag(>I, Am), facilitating the use of an x (r + 1) Vandermonde matrix V [17] for straightforward row-wise circular convolutions. This method reduces the computational load by avoiding complex matrix operations.
The future latent state predictions for latent 24 = 12) ...,..., 2f{ and control Ct = Lu, are given by
(((-D)...... ((++)) [ X7 1, 1
X7 [ct....,c4TT]]
(1)
where denotes the circular convolution operation and This formulation allows for efficient computation, particularly with Fast Fourier Transform (FFT) techniques. For model training, we use a contrastive loss function, Lest defined as
exp(2 W2+
Lest Et~B
log
exp(=?T W Σ, jzt exp(2 T
and prediction loss, Lpred, focused on Mcan- Lrean-Squared Error (MSE),
Lpred = EL~B | K2 - Lue ||2
(3)
These losses drive the learning of model parameters to accurately predict future latent states while learning representations from the visual input.
b.) Koopman Operator and Eigenspetrum Initialization: rately capture the system dynamics, it is imperative initialize the Koopman operator appropriately. The dynamic system's 's future state predictions can be expressed as:
To construct representations that aceu-
( -... [(4r)) [ x7 = [1,x
[ct -..-...,C c
where d = Me + iw; denotes the eigenvalues of the diagonal Koopman operator. From a linear stability perspective, we would like to place the real values of these μ4 in a negative real plane with large values so that the system reaches asymptotic stability faster. However, this configuration may lead exploding gradient issues, direct consequence of the exponential increase in gradient magnitudes proportional to the eigenvalues' real parts.
Lemma (Exponential Scaling of Gradient Norms): For the discrete dynamic system depicted in Equation 4, where Lpred = C, is defined as the latent prediction loss in Equation 3, let 2, represent the latent state and G = Lu, the control input at time step t. The gradient norms of L, with respect to the j-th components of 2j,t and Cj., exhibit exponential scaling compared the gradient norms at ac,
time step + 1, governed by the real parts of the system's 's eigenvalues, My: 825.1
8C,
Atps
025.11
where A/ denotes the time increment, and Hy the real part of the j-th eigenvalue, underscoring the link between the gradient norms of the loss function, the latent representations, the control inputs, and the system's eigenstructure.
To mitigate potential stability issues and constrain gradient magnitudes, limit the real parts of the cigenvalues, μj. within the interval |-0.5, -0.01], and initialize the imaginary parts, wj. in ascending order of frequency as w, = jr similar to [17]. This approach ensures comprehensive fre- quency mode capture within the system dynamics, fostering stable and efficient learning of dynamic representations.
c.) Task Conditioning with End-to-End learning To learn the contrastive spectral representations and spectral Koopman operator, we condition these networks to learn linear effective controller
5

---

similar to [16]. Given Spectral Koopman embeddings Ve, (z)) and its associated linear spectral latent system as shown in equation 4, we formulate a finite time horizon LQQ problem in Koopman latent space as min Ek=o (2* = Q( žςet) + u Ruy] subject to 2k++1 = 2; + Lux (8) where Q and are state and control cost diagonal matrices and Arer denotes the representation of goal imput. We solve the above equation in an iterative procedure recursively update the solution for small number of iterations, typically T 10, which is adequate obtain satisfactory and efficient approximation. The control policy is then given by 'LQRR (z|s) ( L, Q. R) Further to optimize this controller towards the task, we draw inspiration from [9][16], we maximize the following objective via two Q value estimators as used in soft actor-critic method [15]: LSAc = Eg-B imin;=1.2 Qi(2, u) a log [s))(
For end-to-end training of the network, iteratively collect trajectory data batches from the task environment E and use the three distinct loss objectives to train the networks in end-to-end fashion parameter. The primary objective, Lsae is used to optimize all the task parameters to learn efficient representations for the task. Additionally, we integrate contrastive learning loss Leat and model prediction loss Im to regularize the task parameter learning process and feature extraction.
Analytical Results
In this section, we aim to provide a theoretical analysis of the convergence behavior of our over network and show that our task network converges to an optimal policy given enough interactions with the environment. Specifically, we state the following theorems
Theorem 1: (Convergence of Contrastive Loss via Gradient Descent) Let Lem( be L-smooth contrastive loss function for encoder parameters 0 and assuming stochastic gradient descent (SGD) updates with learning rate satisfying Sobbins-Money conditions. If V g is an unbiased esti- mate of the gradient with bounded variance, then lim; & (0:)1l = O. (Proof)
Theorem 2: (Convergence Koopman Operator Approximations); Given (i) a discrete-timg time linear dynamical system with states E Rn and control inputs u IR" evolving according to Ug, where Atrue E T and Birue Ruxm are the true system matrices; and (ii) the Koopman operator approximation estimates A and B such that Ze+1 Rs Az + Bug, the minimization of Lm (. Z; Uk Zk+1) with respect to A and B converges to the true system matrices, ie., lim (A, B) = (Atrue, Btrue), where n represents the number of observations.
Zk+1 Atr
n1500
Theorem 3: (Convergence of the LQR Control Policy) Given discrete-tine linear system char-
acterized by state transition matrix A E Ra and control input matrix B E Raxm and the LQR problem aims to minimize a quadratic cost function J = E Qx + uy Ruy) with Q 2 0
and > 0, the DARE solution
P ATP.A-AAP,BR-BBBIIT " BBP,A+
converges to P*, ensuring that the optimal control gains G yield stable and optimal control policy.
Theorem 4 (LQR within SAC Optimizes Koopman Control Policy): Let Lsee be the SAC loss for a given policy Tsad ssa (ulz) integrated with the LQR control policy NI.R (z (G) in latent space Z. derived via the Koopman operator theory for nonlinear dynamical system. If the SAC loss Csae is Lipschitz continuous with respect to the parameter set Ω = (Q, R. A. A,B, roo and Lsee is bounded below, then applying gradient descent updates on Ω to minimize Lrae guarantees convergence to stationary point of Lewc.
Proofs for these theorems are provided in Appendix Section Building on these theorems, it is established that networks, given adequate number of samples from the training data, guaranteed to converge an optimal task policy.

---

Empirical Results
In this section, we conduct simulated experiments primarily to explore these two questions: Firstly, Can our methodology effectively achieve desirable task performance using linear spectral Koopman representations across diverse environments exhibiting varying nonlinear dynamics? Secondly, Is it feasible to develop a globally linear model that aligns closely with the latent space, demonstrating resilience to noise and external disruptions, while maintaining efficiency in sampling and computa- tion?
Experimental Settings and Baselines In our work, we implement the model that operates within spectral Koopman space, specifically designed with a 1 228-dimensional complex space. This model incorporates a control embedding that maps control inputs to a 128-dimensional space as well, and all the models are trained for 100k and 500k timesteps. We use five distinct robotic control tasks from the DeepMind Control Suite [19]. each characterized by varying state and action space dimen- sionalities: reacher easy, reacher hard, walker, ball in cup, cheetah, and cartpole. For comparative analysis, implement several baseline models. CURL [9] is model-free RL algorithm using contrastive encoder for latent space control learning. Our approach, while also using contrastive encoder. uniquely learns complex-value linear latent space. We also examine the To-KPM [16], which employs linear dense representations for dynamic modeling, whereas our model adopts a spectral Koopman form for efficient and stable system modeling. Additionally, we include model- based RI algorithms: TD-MPC [47] and PlaNet [30] use real-time planning with different variants of the Cross-Entrops Method (CEM), while Dreamer [29] employs background planning. SAC-State [15] serves upper performance bound, receiving direct state input from the simulator. Our perimental framework and the detailed analysis of these baselines thoroughly documented in Appendix A.I and A.2.
In RL, agents learn from interactions with their environment, guided by task rewards. These rewards provide scalar feedback, allowing agents to refine policies for maximizing future rewards. For instance, in the cart pole system, the agent receives fixed reward of +1 for every timestep the pole remains upright, within a predefined angle from the vertical. Therefore, the quicker the cartpole reaches vertical position, the more rewards it will accumulate. In this work, we set the reward as negative of the LQR cost. The greater the reward, the lower the control cost, enabling a faster achievement of the desired state. We report the mean reward and the variance of the distribution across random experiments for all the models.
Evaluation Metrie:
Table 2: Performance comparison at 100K and 500K steps
100K steps
500K steps
Model
Reacher easy
Walker
Cattpole
Chestah
Bull In Cup
Reacher easy
Walker
Cartpole
Cheotah
Ball In Cup
Control State Space Upper bound performance
Sac
123
26
975
961
in Space
TDMPC [47]
413 62
653 99
747 + 78
274 69)
675 + 221
722- 184
944
860
488 + 74
967 + 15
Carl [9]
460 65
482 +
547 73
260 27
741 + 102
929 44
897 + 26
841 45
518 + 28
957 6
DrQ [48]
213
612 164
759
344
9
942 71
921
868
660 96
963 +
Dreamer [29]
53
711
571
Planet [30]
1/10 a. 256
125 1:77
303 171
165 123
198 1-2
351 1
293 114
461 100
321 101
352 1-167
SLACK [49]
361 7:
319
512 110
842 5:
640 a 19
852 ::
To-PPM [16]
238 352
216
797 + 35
197 1
841+ 167
968
127 + 59
835 17
259 + 18
921 565
SAC+AF [15]
145 +
419
312
145
40
15
312 63
Koopman [50)
234
+:47
327
Ours
679 2. 300
640 4:34
864 4.2
305 7.3
940 3.36
969 = 7.9
959 2 15
874 1.7
4.9
967 2:
Results To operate in a sample-efficient regime, we trained our networks for only 100k steps, in con- trast to the 500k-step training regimen utilized for all baseline models. Table 2 provides a detailed comparison all the models for both 100k and 500k steps. For all the environments except Cheetah, our model achieves the highest reward, which notably corresponds to the negative of the control cost, outperforming all other models. We note that control models utilizing autoencoder-type prediction mechanisms, such Koopman AE, SAC-AE, Dreamer and Planet, underperform, with SAC-AE, despite its similar RL exploration strategy to ours, performing particularly poorly. To-KPM, with
7

---

Prediction
126
Operation
205
0.801
6-31
0.1
0.15
6.7
0.79
Model
Oura 12M
Koopeean Al 10M
Ours
Koopman A6
Cestrol
Prodecior
Figure 3: Computational Load SOTA dynam- ical models
Figure 4: Performance of models with external disturbances
its dense Koopman model, also yields comparatively lower rewards than our model. Furthermore, when comparing our model against state-space model saaee--pecificallyy. state-bases SAC -we -w observe that our approach surpasses the state-based controllers in terms of reward, even with a limited train- ing duration of 100k steps, and exhibits very low trial variance for rewards. This underscores the high expressive power and task-guided representation capabilities of model. In the cheetah task evaluation, however, DrQ and SLACv1. This exception may be attributed to these models' nonlinear dynamics models with non-linear next step prediction and similar encoder to our model, making them optimized for state space exploration through deep RI. For more ablation studies of our model, please refer to Appendix
model has the third-best reward in the 100K step evaluation, only behind
Computational Efficiency for Linear Non-Linear dynamics In this section, evaluate the computational efficiency of ofate-of-the-arr predictive models in conjunction with lincarized control mechanisms. Specifically, we implement the MLP dynamics model as described in [9] and the Dense Koopman model from [16] for comparison. Additionally, to assess our method against parallelizable and causally structured models, we incorporate the Transformer model [51, 31] and a recurrent model outlined in [29], which are utilized to simulate system dynamics over more than 100 future steps while concurrently learning a linear control strategy for the system. Figure 3 illustrates the Multiply-Accumulat (MAC) operations required by each model, from which deduce that our approach of using spectral Koopman necessitates the lowest computational effort for both control and prediction tasks. This comparison underscores the efficiency of our method in managing the computational demands associated with dynamic system modeling and control.
Performance under external disturbances: this section, we introduce an external force to the cart pole system during evaluation, denoted by F, which follows a uniform distribution F Uniform(Amare (amaxx O, dminn. The probability of applying this foree is represented by p. where amax and amin correspond the environment's maximum and minimum allowable control input values, spectively. Figure 4 illustrates the mean reward and Interquartile Mean (IQM) for the models under study. Notably, our models demonstrate robust control capabilities, maintaining high performance even when the probability of external disturbances is increased to 0.25. This resilience evidenced by a minimal degradation in performance when compared to alternative approaches.
Conclusion and Future Work
In this work, we introduce RoboKoop, a novel approach that integrates a contrastive encoder with spectral Koopman operators to learn visual representations guided by task learning. We demonstrate that RoboKoop surpasses current state-of-the-ar methods, achieving superior performance while maintaining sample efficiency and robustness to noise and external disturbances. Looking forward, aim further enhance sample efficiency by exploring alternatives to Soft Actor-Criti (SAC) methods, potentially reducing the number of samples required for effective learning.
8

---

Acknowledgments
This work is based in part on research supported by SRC JUMP2.0 (CogniSense Center, Grant: 2023-JU-3133) and the Army Research Office (Grant: W911NF-19-1-042
The views, opinions, findings, and conclusions expressed in this material are those of the author(s) and do not necessarily reflect the official policies positions of SRC, the Army Research Office, or the U.S. Government.
References
[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland. G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, Antonoglou, H. King. D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through
deep reinforcement learning. Nature, 518(7540):529-533 2015. 0a::10.1038.. 100.000//aauure1120000 URL https://doi. org/10. 1038/nature14236.
[2] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D, Silver, and D. Wierstra. Continuous control with deep reinforcement learning, 2019.
[3] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation, 2018.
[4] Schrittwieser, I. Antonoglou, Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, Lillicrap, and Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):044-60) Dec. 2020. ISSN 1476-4687.
0005886000000011-- URL http: http://xxxdddo.org/// http://tt.doi.or/11.10.0.0.
441586-020-03005
[5] D. Ha and J. Schmidhuber. World models. 2018. do1:10.221/77N00001177311 URL https: /etps://zenodee oot.aoddd.orrrreoor///0011111111
[6] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, and H. Michalewski.
Model-based reinforcement learning for atari, 2020.
[7] M. Watter, J. T. Springenberg. Boedecker, and M. Riedmiller. Embed to control: locally linear latent dynamics model for control from raw images, 2015.
[8] M. Jaderberg. V. Mnih, W. M. Czarnecki, T. Schaul, J. Leibo, D. Silver, and
K. Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks, 2016.
[9] A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, 2020.
[10] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifree, K. Kavukcuoglu, D. Kumaran, and R. Hadsell. Learning to navigate in complex environments, 2017.
[11] A. Pinto, A. Kolesnikov, Y. Shi, L. Beyer, and X. Zhai. Tuning computer vision models with task rewards, 2023.
[12] Y. Yu, Chung, H. Yun, Hessel, J. Park, X. Lu, Ammanabrolu, Zellers, I.. Bras, G. Kim, and Choi. Multimodal knowledge alignment with reinforcement learning, 2022.
[13] C. Kim, Y. Sco, Liu, Lee, Shin, H. Lee, and K. Lee. Guide your agent with adaptive multimodal rewards. In -hirth-seventh Conference Neural Information Processing Systems, 2023. URL https: https:// wttp:/eee//oooree htt.://wwwweeeoooooreeeaaaaaaaaa

---

[14] Y. Xiao, Z. Tang, X. Xu, X. Zhang, and Y. Shi. A deep koopman operator gperator-based based mod-
elling approach for long-term prediction of dynamics with pixel-level measurements. CAA/
Transactions on Intelligence Technology, 91):178-196, Feb. 2023. ISSN 2468-2322. doi:
12149 URL http: dww. www.d/.......................... ottp::///aa......oo//...0o... 1019:/02...99//1...12..
[15] Haarnoja, Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018.
[16] X. Lyu, H. Hu, S. Siriya, Y. Pu, and M. Chen. Task-oriented koopman based control with contrastive encoder. In 7th Annual Conference Robot Learning. 2023. URL https:// openreview weet/forummiddqQVoooo
[17] A. K. Mondal, S. S. Panigrahi, S. Rajeswar, K. Siddiqi, and S. Ravanbakhsh. Efficient dynam- ics modeling in interactive environments with koopman theory, 2023.
[18] A. Gu, A. Gupta, K. Goel, and C. Ré. On the parameterization and initialization of diagonal state space models, 2022.
[19] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. Riedmiller. Deepmind control suite, 2018.
[20] H. Shi and M. Q. H. Meng. Deep koopman operator with control for nonlinear systems, 2022.
URL atttps://axxivv org/abs/////eoooooooo..oo....eat 2tlpb//2220.0080
[21] Y. Han, W. Hao, and U. Vaidya. Deep learning of koopman representation for control. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 1890-1895, 2020. doi: 10. N0.1109/DDC422442000000000
[22] Y. Xiao, Tang. X. Xu, X. Zhang, and Shi. A deep koopman operator-based modelling approach for long-term prediction of dynamics with pixel-level measurements. volume 9, pages 178-196. 2024. dot.amps//db..ogg/.0000////.///. onlinelibrary. wiley. comm//oi/abs/1..1009990
URL attps://eetresearch 12149.
[23] E. Levina and Bickel. Maximum likelihood estimation of intrinsic dimension. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems, vol- ume 17. MIT Press, 2004.
www.eeeeednnsseeerrs......//o//e Paper. .df.
[24] S. Zhao, J. Song, and S. Ermon. The information autoencoding family: A lagrangian per- spective on latent variable generative models, 2018. URL https://arxivv. wttpi/v..rr/ab///////// 06514.
[25] Y. D. Zhong and N. E. Leonard. Unsupervised learning of lagrangian dynamics from images for prediction and control. CoRR, abs/2007.01926 2020. URL https::/ btttps://arxvv. org/obgg///.....oooo..ooeeeeee /bs/ 2007.01926.
[26] A. Sancher-Gonzaler, N. Heess, Springenberg. J. Merel, M. Riedmiller, R. Hadsell, and Battaglia. Graph networks learnable physics engines for inference and control, 06 2018.
[27] N. Watters, Tacchetti, Weber, R. Pascanu, P. Battaglia, and D. Zoran. Visual interaction networks, 2017. URL https://// http::::///aaai...//aaaa/a//////
01433.
[28] S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian neural networks. CORR, abs/1906.01563, 2019. URL http://arxiv http:://arriv.or/aaa/////o/ wwtp/org///////oeeo 01563.
[29] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination, 2020.
10

---

[30] D. Hafner, T. P. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learn- ing latent dynamics for planning from pixels. CoRR, abs/1811 1.0551, 2018. URL http: faxii.oorg..o... awsaaasssaa 04551.
[31] L. Chen, K. Lu, A. Rajeswaran, Lee. A. Grover, M. Laskin, Abbeel, A. Srinivas, and
I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021.
[32] X. Fang. Hardness-aware contrastive learning. PhD thesis, 2022. URL http://dx..o doi. org/10. 14T11//hheii--99011110066888
[33] Cui, 7.. Zhong, Liu, Yu, and Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 715-724, 2021.
[34] T. Chen, S. Kormllthh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In H. D. III and A. Singh, editors, Proceedings of the 37th Interna- tional Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020. URL pppoceedngsssssnsg..... mlr. press/v119/chet chen20 .html.
[35] R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 1299.. PMLR, 2021.
[36] A. Tariverdi, V. K. Venkiteswaran, M. Richter, O. J. Elle, J. Tørresen, K. Mathiassen, S. Misra, and O. G. Martinsen. recurrent neural-network based real-time dynamic model for soft continuum manipulators. Frontiers in Robotics and AI, 8, 2021. ISSN 2296-9144. 10i:30.3389/ crobt20211631130 URL wttp://e/ww.ooniieriiii.nn org/ att.aaattilleeeeee http//00...//fottt....oo 2021. 631303.
[37] H. Kumawat, B. Chakraborty, and S. Mukhopadhyay. Stemfold: Stochastic temporal manifold for multi-agent interactions in the presence of hidden agents. arXiv preprint 2rXiv:2401. 14522, 2024.
[38] Anonymous. STAGE net: Spatio-temporal attention-based graph encoding for learning multi- agent interactions in the presence of hidden agents, 2023. URL https://openreeieww net/ forum?id=tsj6rDII00
[39] P. W. Battaglia, R. Pascanu, M. Lai, D. Rezende, and K. Kavukcuoglu. Interaction networks for learning about objects, relations and physics, 2016. URL https: htwp://ww.oggaa//1/111.1 00222.
[40] H. van Hoof, N. Chen, M. Karl, P. van der Smagt, and Peters. Stable reinforcement learning with autoencoders for tactile and visual data. In 20/6 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3928-3934, 2016. 100/0.. 2001. 7759578.
[41] C. Finn, X. Tan, Duan, T. Darrell, S. Levine, and Abbeel. Deep spatial autoencoders for visuomotor learning, 2016.
[42] P. Agrawal, A. Nair, P. Abbeel, J. Malik, and S. Levine. Learning poke by poking: Experi- ential learning of intuitive physics, 2017.
[43] Reinforcement Learning and Stochastic Control. Cambridge University Press, 2022. doi: 10.1017797880000051110
URL http::/dds doi. wtt.:ddd.oggg/....07///000000000 008.
[44] R. Miftakhov, A. Al-Qasim, and Efremov. Deep reinforcement learning: Reservoir opti- mization from pixels. In IPTC, 2020. doi: dbt:10.25223ipoe2201111
URL http://dx.dof doi.
org/10. 2523 /es//ptc-22000 20151-ms..

---

[45] R. Miftakhov, I. Efremov, and A. S. Al-Qasim. Reinforcement learning from pixels: Wa- terflooding optimization. In American Society of Mechanical Engineers, 2020. doi:10.1115/ omae2020-1855 URL http: / /ttp//xxdddd// doi. org/18 10. 1111/11aae22000 18574.
[46] M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning, 2017.
[47] N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control, 2022.
[48] Kostrikov, Yarats, and R. Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels, 2021.
[49] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine Stochastic latent actor-critic: Deep rein- forcement learning with a latent variable model, 2020.
[50] H. Shi and M. Q. H. Meng. Deep koopman operator with control for nonlinear systems, 2022.
[51] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polo- sukhin. Attention is all you need, 2023.
[52] H. Kumawat and S. Mukhopadhyay. Radar guided dynamic visual attention for resource- efficient rgb object detection. In 2022 International Joint Conference on Neural Networks (JJCN), pages 1 1-8, 2022. doi:10.1 10L.L000IINNN5004020220999922229
[53] K. Samal, H. Kumawat, M. Wolf, and S. Mukhopadhyay, A methodology for understanding the origins of false negatives in dnn based object detectors. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2022. doi:10.1 (09/CCCNN550442222299999999
[54] K. Samal, H. Kumawat, P. Saha, M. Wolf, and S. Mukhopadhyay. Task-driven rgb-lidar fusion for object tracking in resource-efficient autonomous system. IEEE Transactions Intelligent Vehicles, 711::022-112, 2022.
12

---

A Model Details and Experimental Settings
13
A.1 Simulation Environment for Empirical Study
13
A.2 Model Hyper parameters
14
A.3 Algorithm Design
14
Baselines
15
B.1 CURL: Contrastive Unsupervised Representations for Reinforcement Learning [9]
15
B.2 TP- [16]
15
B.3 Planet [30]
16
B.4 Koopman AE [20]
16
C
Analytical Results
16
C.1 Convergence of Contrastive Learning
16
C.2 Stability and Convergence of the Koopman Operator Approximation
17
C.3 Convergence of the LQR Control Policy
18
C,4 Integration of LQR within SAC Framework Optimizes Koopman Control Policy
21
D Empirical Results
23
D.1 Eigenspectrum of our model
23
D.2 Ablation Study Spectral Koopman Operator Initialization
24
D.3 Ablation study on Performance under Imperfect Sensing
25
D.4 Ablation Study on Performance under Gaussian Noise
26
A Model Details and Experimental Settings
A.1 Simulation Environment for Empirical Study
In our research, introduce RoboKoop, algorithm distinguished by its sample efficiency, which processes pixel-based inputs to simultaneously learn linear dynamics and develop an effective con- trol policy. This algorithm demonstrates versatility across a broad spectrum of environments. We have rigorously tested RoboKoop against continuous control challenges within the DeepMind Con- trol Suite. Our selection these particular tasks grounded in several critical considerations:
1. Existing baseline methods exhibit suboptimal performance on these tasks, highlighting a gap that RoboKoop aims to fill.
2. Recent advancements have introduced both model-free and model-based strategies aimed at enhancing the sample efficiency of similar algorithms. Our work contributes to this ongoing dialogue by presenting an alternative approach.
3. The performance metrics obtained from these simulated tasks are highly indicative of real- world applicability, underscoring the practical relevance our findings broader contexts.
Cartpole Swingup This task is centered around the goal of swinging up a pole, initially in a down- ward orientation, attached moving cart, and then maintaining its upright position. Success in this task requires the precise application of forces the cart, navigating through 4D state space that represents the cort-pole system's kinematics, complemented by ID control space for force application.
13

---

Cheetah Run The objective here is to orchestrate the movements of a simulated planar cheetah to achieve rapid and stable running. This involves managing an 18D state space that encapsulates the kinematics of the cheetah's entire body, including its joints and limbs, while employing 6D torques as controls to manipulate
joints for optimal locomotion.
Reacher The Reacher task is designed to test precise motor control by requiring agent to maneu- ver a simulated two-joint robotic arm to target location in 2D plane. This task involves navigating through an 11D state space that includes the positions and velocities of the arm's 's joints, as well as the position of the target. The control space is 2D, representing the torques applied at each joint. Success in this task measured by the agent's 's ability to accurately and efficiently move the arm to the target position and maintain it there.
Ball in Cup In the Ball in Cup task, the objective is to control simulated robot arm to swing and catch ball attached to a string in a cup. This task is particularly challenging due to the non- linear dynamics involved in swinging the ball and the precision required to catch it in the cup. The environment's 's state space is 8D, capturing the positions and velocities the ball and the robot arm, as well as the angular position of the cup. The control space is 3D, representing the forces applied the robot arm to achieve the desired swing motion. Success in this task requires combination of dynamic coordination and precise timing.
Walker The Walker task involves controlling bipedal robot to achieve stable and efficient loco- motion. The state space for this task is 17D, encompassing the kinematic properties of the robot's body and legs, including joint positions and velocities. The control space is 6D, corresponding to the torques applied the robot's joints. The objective to navigate the robot through various terrains, maintaining balance and forward motion. Success in this task is determined by the robot's ability to move swiftly and stably without falling.
A.2 Model Hyper parameters
Table 3 provides comprehensive enumeration of the hyperparameters employed in our model, along with detailed descriptions of each parameter. For To-KPM [16] also, we use the same hyper- parameters as our model for a fair evaluation.
A.3 Algorithm Design
For an end-to-end training of the network, we iteratively collect trajectory data batches from the task environment E and use the three distinct loss objectives to train the networks an end-to-end fashion. The primary objective, Lucc used optimize all the task parameters to learn efficient representations for the task. Additionally, diction loss Imm to regularize the task parameter learning process and feature extraction. Algorithm describes the loss computation and parameter updates for the model.
integrate contrastive learning loss Lest and model pre-
Algorithm Model Training
0:
Initialize parameters: Koopman control Q, Rt Dynamics K, L, Encoder vhg.
Reset task environment E.
0: Initialize data replay buffer D.
for each iteration 17 do
0: Collect roll-outs T, by executing policy TiQr store in D.
Sample batch from D.
0: Compute Lea: Update the to minimize Lest-
Compute Lunc: Update Q, R, K, minimize frne (excluding vio).
Compute Lm: Update K, L minimize Lm.
end for=0
14

---

Table 3: Hyperparameters and Configuration Details
Name
Value
Description
Environment
transform image size
Initial size images before applying transforms. Number of frames stacked logether input. The resolution of input images to the network. Maximum size of the replay buffer.
Frame stack
3
Image size
84
Replay butfer capacity
100000
Agent
Hidden dim
1024
Dimension of hidden layers in neural networks. Discount factor for future rewards (t)).
Discount factor
0.99
Init temperature
0.1
Initial temperature parameter for SAC algorithm.
Alpha
0.0001
Learning rate for temperature parameter.
Alpha beta
0.5
Beta parameter for the Adam optimizer for temperature.
Actor lr
0.001
Learning rate for the actor network. Beta parameter for the Adam optimizer for the actor net- work.
Actor bcta
0.9
Actor update freq
Frequency actor network updates. Learning rale for the critic netwurk. 0.001
Critic Ir
Critic bela
0.9
Beta parameter for the Adam optimizer for the critic net- work.
Critic tau
0.01
Tau parameter for soft updates of the target networks. Frequeney target network updates.
Critic target update freq
Facnder feature dim
256
Dimensionality of the encoded features,
Control encode dim
128
Dimensionality of the encoded control input.
Encoder Ir
0.001
Learning rate for the encoder.
Encoder tau
0.05
Tau parameter for soft updates of the encoder. Number of layers the convolutional neural networks.
Num layers
Num fillers
32
Number of filters in the first convolutional layer.
Curl latent dim
128
Dimensionality of the latent space in CURL.
Koopman update freq
1
Frequency of updating the Koopman operator.
Koopman optim
0.001
Learning rate for optimizing the Koopman operator.
Koopman coeff
0.1
Coefficient fitting the Koopman operator.
Koopman horizon
5
Horizon length for Koopman predictions.
Training
Init steps
1000
Number of steps collected with random actions the of training.
Num train steps
150000
Total number of training steps.
Batch size
128
Batch size for training.
B
Baselines
This section delineates the comparative analysis baselines utilized in our study and elucidates how our approach diverges from them.
B.1 CURL: Contrastive Unsupervised Representations for Reinforcement Learning [9]
CURL, which stands for Contrastive Unsupervised Representations for Reinforcement Learning, employs contrastive learning to derive high-level features from raw pixels for reinforcement learning tasks. Our methodology, however, adopts spectral Koopman operator model to explicitly learn system dynamics, feature absent in CURL. This distinction permits an in-depth analysis of system stability and provides valuable insights into controller design. Unlike non-linear control policies that lack comprehensive system analysis, linear systems be thoroughly examined through eigenvalue analysis. We demonstrate this through pole analysis of the Koopman operators in Section 5, highlighting the methodological differences and advantages.
B.2 To-KPM [16]
To-KPM introduces a task-oriented approach that integrates a contrastive encoder with Koopman- based control. Unlike model, To-KPM relies dense Koopman operator, leading unstable poles and reduced sample efficiency due to the increased parameters required for learning the Koop-
15

---

man operator. These limitations are substantiated by the instability of poles (refer to Figures 4 and 5 in Section our paper) and underscore the efficiency our approach.
B.3
Planet [30]
Planet a model-basee agent that discerns environment dynamics directly from pixels, facilitating action selection through online planning within compact latent space. 'he latent space is structured around recurrent state-space model, which is computationally intensive, as evidenced in Section (Figure 6). Additionally, its emphasis on multi-step prediction in pixel space compromises sample efficiency, necessitating extensive interactions with the environment.
B.4 Koopman AE [20]
The Koopman methodology leverages soft actor-critic policy, underpinned by a regularized autoencoder (AE), learn latent space model atop AE features. Unlike Planet, this approach also explicitly models dynamics using Koopman operator. In contrast, method eschews the use of VAEs or AEs for pixel reconstruction, opting instead learn features via contrastive learning alone. This strategy ensures the prioritization of task-relevant features over the reconstruction of pixel space, enhancing task efficiency and model performance.
C Analytical Results
C.1 Convergence of Contrastive Learning
Definitions and Assumptions
1. Smoothness: The function Lest is assumed to be L-smooth with respect to 8, meaning it has Lipschitz continuous gradients:
| (oi) (02) s L110, 02l1. v0 0z.
Unbiased Gradient Estimates: The stochastic gradient VoLea is an unbiased estimate of the
true gradient:
B[VoCes(01] = (O).
3. Bounded Variance: The variance of the stochastic gradient is bounded by constant o²:
E V VeLcu (0)
< o²
Diminishing Learning Rates: The learning rate ai satisfies the Gobbins-Mone conditions:
oc
Σ a = 0c,
Σ af < co.
Convergence of Contrastive Loss via Gradient Descent
Theorem 1.:
Let Lor(0) be an L-smooth contrustive loss function for encoder parameters and assuming stochastic gradient descent (SGD) updates with learning rate 0n satisfying Robbins- Monro conditions. If V Lest is unbiased estimate of the gradient with bounded variance, then lim ofllV
Given the Lipschitz continuity of the and assuming the loss Les inherits this property with respect to 0, the Descent Lemma can be applied. The lemma states that for a Lipschitz continuous function ff with Lipschitz constant L.
Proof:
f(x Az) S f(x) Vf(x) Ar _1
Given the L-smoothness of Lests we have for any 0,, 02:
16

---

L
Laa(82) - (01) + (B))((11) (02 01) +
2 100 - 0111?
Substituting the gradient descent update 0z+1 = 02 an (oLc
Lea (0++)) Lca(0,)) -z VLe Vo (0:) + Lai
| VoLca(0t)1
2
Taking expectations both sides, and using the fact that ooCol VoLca (unbiased gradient estimates) and the bounded variance assumption:
E S E((ea( - ax Volc + Lai
(o² (V))cc(0)))1))
2
Rearranging the terms, aim to show that:
(1 Lay 2
Laça²
((Cesa(0++)))) (o B((Ca(0)))))) -BCee (0t+1)) +
2
Given oG satisfies the Robbins-Monn conditions and Los for sufficiently small an summing both sides over and applying the law of total expectation give:
00
Lo E (o Lox(O1) (0")
06
Lafoz
Σ
a1((1
where 0 is a local minimum of Cen-
Given the right-hand side is bounded (due to the boundedness of
Lest and the conditions at), and E a, Lau it follows from the quasi-martingale
convergence theorem and the Robbins-Monte conditions that:
lim (0.)) t
This implies that, in expectation, the gradient norm converges to O, indicating convergence stationary point. Now using the Polyak-t.ojssiewizz condition, can be shown that this is local minimum.
The exact form of fex and its gradient VoLes. The Lipschitz constants for the and Lest Conditions under which the stochastic gradient is unbiased estimate of the true gradient and has bounded variance. A suitable learning rate schedule ae that guarantees convergence.
C.2 Stability and Convergence of the Koopman Operator Approximation
Theorem 2:
Convergence of Koopman Operator Approximations: Given (i) discrete-time linear dynamical system with states E IR" and control inputs u E IRm evolving according to Zk+1 = Atrve + Brrne Up. where Atrue E Raxx and B,ywe Raxm are the true system matrices; and (ii) the Koopman operator approximation approach, which seeks to estimate matrices A and B such that 2g+1 & Azę + Bug, based loss function (A,,A, B; Zk, Ug, 4k+1), the minimization of Lm with respect to A and B over the observed data converges to the true system matrices, i.e.,
lim (A, B) (Atrues Birue); 20000
where represents the number of observations.
17

---

Proof
We model the evolution of the system's state as a linear regression problem, where:
next is the matrix of next states Bk+1+
is the design matrix composed of current states 2₁: and control inputs Us-
is the parameters matrix to be estimated, combining and B,
c is the error term.
The equation Zeext Xe + encapsulates this linear relationship.
The objective function to minimize the difference between the predicted next states and the actual next states, quantified by the Frobenius norm of their difference can be written as:
Pm
Znext
X
where || || denotes the Frobenius norm. To minimize L.m, we calculate the gradient of the loss function with respect θ and set it to zero: VeLm -2X (Znext X 0.
Solving this equation for + gives: θ (xxx) T Znext. This is the least squares solution, providing the best estimate of @ given the data.
With the assumption that the observations and Znext sufficiently cover the entire state and control input space and as the number of observations approaches infinity (N oo), the matrices X X and X Znex will converge to their expected values. This ensures that the estimated parameters O which combine A and B, converge to the true system matrices Atrue and Birue that govern the system's 's dynamics.
The solution involves setting the gradient of Lm with respect to to zero, leading to:
VeCm -2X (Znext X
Solving this equation yields the estimate for O:
C - 7 Znext
Given a sufficiently diverse and large dataset (n - oo), the estimates converge to the true system dynamics because the matrices X X and X Znest approach their expected values, ensuring the estimated parameters (A and B) converge to the true parameters (Atrue and Birue).
This proof assumes sufficient data coverage across the state and control input space, which guaran- tees the convergence of the Koopman operator approximations to the true system dynamics, thereby validating the theorem.
C.3 Convergence of the LQR Control Policy
Theorem 3: Convergence of the LQR Control Policy Given a discrete-ting time linear system char-
acterized by state transition matrix A Raax and control input matrix E Raxm and the LQQ problem aims minimize quadratic cost function J (x Qx: + u, Rug) with Q 2 and R 0, the iterative solution the Discrete-time Algebraic Riccati Equation (DARE)
P A P.A A. BBBABA
converges to the optimal solution p for the LQR problem, ensuring that the optimal control gains G B P''B" - AP'A A yield stable and optimal control policy.
Proof:
18

---

To prove the convergence of the Linear Quadratic Regulator (LQR) control policy, we focus on the discrete-time setting, where the goal is to design an optimal control policy that minimizes a given cost function. The essence of the proof involves showing that the solution to the Discrete-time Algebraic Riccati Equation (DARE) converges to unique positive semidefinite matrix, which then defines the optimal control gains.
We are given discrete-time linear system:
X = Ax Bug,
and aim to minimize the -oriitnn quadratic cost function:
J = Σ X Qx u Rus).
where Q 2 (positive semidefinite) and R (positive definite) are the state and control weight matrices, respectively.
The optimal control policy for this problem can be derived using dynamic programming, leading to the DARE:
P ATPA - PB -1 BB + Q,
where P is the solution that defines the optimal cost-to-go matrix.
The convergence of the LQR control policy essentially means proving that the iterative solution to the DARE converges to a unique positive semidefinite matrix P Here are the key steps:
Monotonicity and Boundedness:
To prove that the sequence {P:} generated by the Discrete-time Algebraic Riccati Equation (DARE) iterations monotonically decreasing and bounded below, thus ensuring convergence, let's delve into equations and inequalities that illustrate these properties. Consider the iterative update rule for DARE:
P P A PPAA-APBBBBBPP. -1 'B
where:
A and B define the system dynamics, is the control weighting matrix, which is positive
definite (R > 0),- - Q is the state weighting matrix, which positive semidefinite (Q 2 0), P, is the cost-to-go matrix iteration i.
To show that Pi+1 < P,, we need to establish that P, Pi+1 is positive semidefinite for each The Riccati update aims to minimize the cost function J, associated with using the control law derived from P.. Therefore, we define the cost reduction as AP, = P: P4+1. we seek to show that AP; (i.e., AP, is positive semidefinite).
Starting from the DARE update rule and rearranging terms gives us:
AP: = -; - A - BBP,A,
Given that R and P, is positive semidefinite, it follows that the right-hand side of the equation above is positive semidefinite. This is because the term inside the parenthesis, + B P,B, is positive definite, making its inverse also positive definite, and thus AP, is positive semidefinite, indicating that P < P..
19

---

The sequence is bounded below by the zero matrix, given that the cost-to-go matrices P; represent quadratic cost functions which are non-negative:
P.20 Vi,
implying that the sequence cannot decrease indefinitely and is bounded below by a matrix where all elements greater than equal Given the monotonicity and boundedness of the sequence [P.). it follows from the Monotone Convergence Theorem for matrices that the sequence converges to limit, say P*, which is the solution to the DARE and represents the optimal cost-to-go matrix:
lim P, =P,
where P satisfies the DARE and thus confirms the optimality and stability of the LQR control policy derived from it.
By establishing the monotonic decrease and boundedness below of the sequence (P,). have shown that this sequence converges to matrix P that minimizes the LQR cost function. This P is the fixed point of the DARE, providing the optimal cost-to-go estimate and ensuring the stability and optimality of the LQR control policy derived from it.
Fixed Point Convergence: Under the assumptions that B. Q. and R satisfy certain control- lability and observability conditions, it be shown that the iteration converges fixed point. To prove that the limit of the sequence (Pi). denoted as P satisfies the Discrete-time Algebraic Riccati Equation (DARE) and is thus fixed point of the iteration process, we employ the properties of convergence and continuity of matrix operations.
Given the iterative process:
PAI - A. -1 B
aim to show that, as i + 0c, P. P and that P satisfies the DARE:
P' T-AAP'AAAPPD +BBBB) -' 'BTP'A
From previous steps, we have shown that the sequence (Pr} is monotonically decreasing and bounded below, which guarantees convergence to a limit p due to the Monotone Convergence Theorem for matrices.
The operations involved in the iterative update rule, including matrix addition, multiplication, and inversion, are continuous functions of their arguments. This means that if a sequence of matrices (X,} converges to X, then the limit of continuous function f(X,) is f(X). The update rule can be seen as the application continuous function f to P,:
f(P.) = A P. A.B. BR.BBRBBBP.BB BTB P
Given the convergence P: P by continuity, have:
lim f(.) f( lim P.) f(P"). 1-000
10000
This implies:
= D'AAP'A-AAPBRR-BPPPP BBPAAA Q
20

---

which is precisely the DARE. By showing that P satisfies the DARE, we' we've proven that P is fixed point of the iteration process. This fixed point represents the solution to the DARE, establishing the optimality of the limit matrix P for the LQR problem.
Thus, by leveraging the properties of monotonicity, boundedness, convergence, and the continuity of matrix operations, we' ve demonstrated that the limit of the sequence {Pr). P satisfies the Discrete-time Algebraic Riccati Equation, making it the optimal solution and fixed point of the iterative process.
The convergence of the IQRR control policy to an optimal solution involves demonstrating that the iterative solution the DARE converges unique matrix that minimizes the cost function and that the corresponding control policy stabilizes the system. The proof relies algebraic properties of the Riccati equation, control theory, and the system's 's controllability and observability conditions.
C.4 Integration of LQR within SAC Framework Optimizes Koopman Control Policy
Lemma: Given a loss function L that is Lipschitz continuous with respect to the parameters Ω and bounded below, the sequence {St} generated by the gradient descent updates:
4 = Sr V
with a sufficiently small, fixed learning rate v Vn(((S²) 0.
0, converges to stationary point Ω where
Proof: Given that is Lipschitz continuous with Lipschitz constant L, we have for the gradient descent update:
(() < ((2,) + Val(a))
(0.+1 - 24) + ||22+1 22el
(5)
+00+1 - Ωr = -
(6)
7| V + Ln² [n((2))11
< L(.): #((A++1)
Choosing of Select 1) such that < 7. ensuring that:
C(S S ((())) - (11
Ln² (n--"²) [
Since L is bounded below, and L( s L())) for all f, the sequence (L()) is non-increasing and bounded. This implies convergence of the loss function values.
The reduction of the loss at each step is proportional the square of the norm the gradient. If the sequence (St) did not converge to stationary point, the gradient norm would not approach zero, contradicting the boundedness and convergence of the loss function values. Therefore, the gradient norm must approach zero, i.e., Vn((Ω*) 0, indicating convergence to stationary point.
Theorem 4: Let Liac be the Soft Actor-Critic (SAC) loss function for a given policy (ulz)))z) in- tegrated with the Linear Quadratic Regulator (LQR) control policy NI.R Gz) in a latent space Z. derived via the Koopman operator theory for a nonlinear dynamical system. If the SAC loss Cuse is Lipschitz continuous with respect the parameter S = A, vo) and Lsee is bounded below, then upplying gradient descent updates on Ω to minimize Lxae guarantees convergence stationary point of Exac
Proof: Assume Lsec satisfies the Lipschitz condition with Lipschitz constant L > 0, i.e.
[.soo()ti) Lsnc (52)) < L| D2ll.
21

---

for any S21, S2 in the parameter space.
Now, the update rule for the parameters via gradient descent is given by:
R Sr V (St,))
where 77 > 0 is the learning rate.
Using Lemma 1, given Sae is bounded below and Lipschitz continuous, the sequence {2t pro- duced by the gradient descent updates will converge to stationary point I2", characterized by:
V (2') 0.
Hence show the optimality and stability via LQR Integration. The integration of the LQR policy TLQR ensures that within the linear approximation of the dynamical system dynamics in the latent space Z, the SAC framework, enhanced with LQR, converges towards optimal control actions. The LQR component provides an optimal control policy for linearized dynamics around the current state and control, ensuring that the SAC algorithm's policy updates enhance both stability and optimality in control decisions.
For linear system 2 Az Bug, the LQR aims minimize the cost function:
CC
23 Qzi + up Rus).
k=
where Q 2 and R > 0. The optimal control law is i = KZz with K = (R +
B -B PA, where solves the Algebraic Riccati Equation (ARE):
P P A P BBB
Q
The SAC algorithm seeks optimize the policy Taee (u|z) by solving:
max E
7t (R(zk, ux) + (H((T((1xx)))
where denotes the entropy of the policy, promoting exploration, and a is the temperature param- eter that balances reward and entropy.
Integration means adjusting the SAC optimization include the LQR solution baseline regularization term. The objective becomes:
max Σ * (R(zk, u&) + aH((T(-Zzz) ALoR(Zz (u)))
where is a weighting coefficient, and JLOR is the LQR cost function introduced above. This formulation explicitly guides SAC policy towards the LQR's optimal policy within the linear approximation of the dynamics.
The optimal policy x² and the corresponding control law u from this integrated optimization prob- lem are given by (1) the policy x" that maximizes the augmented objective, and (2) the control law that minimizes the LQR cost, ensuring stability as P guarantees the eigenvalues of (A BK) lie within the unit circle, ensuring the system's stability.
The parameter update rule incorporating both SAC optimization and LQR regularization is given by:
22

---

0.20
0.75
Equrrzhaass
03
5.75
0.5
0135
3,00
66
0.90
0.75
-L.5
0335
-1.1 -356
-035
075
0.5
Figure 5: Eigenspectrum of To-KPM
Figure 6: Eigenspectrum of our model
84 Ω2 V/ (R.) (L)))
where Lscc and JLOR are differentiable with respect to Q2 ensuring that the gradient descent steps move the parameters towards minimizing the SAC loss while adhering to the LQR optimality cri- teria. Given the Lipschitz continuity and differentiability of Sas XJLLR. the updates guarantee convergence stationary point Ω where Vn (C (S*) >JLOR (S²)) 0, encapsulating both the optimal policy in the SAC framework and the stability provided by the LQR control law.
Thus, we ve shown how this combined approach integrating LQR within the SAC framework lever- ages LQR's optimality and stability, guiding the policy updates SAC towards enhanced control decisions. The integration explicitly incorporates the LQR's linear control optimality into SAC's nonlinear policy optimization, ensuring convergence towards optimal and stable control actions in the latent space Z. Thus, show that under the conditions of Lipschitz continuity and boundedness of the SAC loss function, gradient descent optimization of the combined SAC and LQR policies in the Koopman latent space converges to stationary point, optimizing the overall Koopman control policy. This integration not only leverages the strengths of both SAC and LQR but also ensures that the optimization process is theoretically grounded and guaranteed to reach point of stability and optimality.
D Empirical Results
In this section, conduct ablation study identify which components of our network contribute to its superior performance with limited number of training steps. First, to demonstrate the effect of nonlinearity, we use CURL[9] as a baseline. CURL features a contractive encoder similar to ours but employs nonlinear dynamics, unlike our spectral dynamics. For comparison with a linear dense model, we use ToKPM[16), which relies on dense linear dynamics as opposed to our spectral model. Throughout the ablation studies, we demonstrate that our model outperforms both baselines. For this section, we present the results for models trained for 150,000 steps, as the other baselines showed poor performance when evaluated at 100,000 time steps.
D.1 Eigenspectrum of our model
In Figures 5 and 6, we present the eigenspectrum contour plots for the To-KPM model and our proposed model, respectively. Analysis reveals that the eigenvalues the To-KPM model predom- inantly reside on the positive real axis, with an average value of approximately 0.4. Conversely, our model exhibits symmetric distribution of eigenvalues across the imaginary axis, featuring an equal proportion of positive and negative real eigenvalues. This distribution aligns with an increas- ing trend of eigenvalues per w jm. Within the framework of the Koopman operator theory,
23

---

Mean Reward vs. Rain Noise Level with IQM Reward Trendlines
x 10"
Mean Reward
4 102
3 10
2 102
Low
Medium
High
Rain Noise Level
Ours-IQM
Koopman AE-IQM
Koopman
To-KPM-IQM
Sac-AE-IQM
To-KPM-mean
Sac-AE-mean
Figure 7: Performance of models in rainy conditions.
negative cigenvalues signify that the system's observables exhibit exponential decay over time, as these eigenvalues are integral to the exponential term in the solution the linear system governed by the Koopman operator. Hence, negative eigenvalues are indicative of stable observable behaviors, whereas positive cigenvalues suggest exponential growth in observables, pointing to instability. The presence of positive eigenvalues in the To-KPM model undermines its ability to learn stable rep- resentations from images using a finite-dimensional Koopman operator, leading to inferior perfor- mance compared to our model, which benefits from balanced distribution of positive and negative eigenvalues.
D.2 Ablation Study on Spectral Koopman Operator Initialization
In this section, conduct ablation study evaluate various initialization strategies for the Koopman spectral method, as detailed in Section 3.2. Our baseline configuration sets the Koopman operator's real value at -0.2, with frequencies arranged in increasing order. To assess the impact of initialization on performance, we explore three additional designs: (a) learnable real values with increasing frequency, (b) constant real values with random frequency. and (c) learnable real val- ues with random frequency. This examination seeks identify the initialization method that most effectively enhances the accuracy and stability of the spectral Koopman method.
'abbe 4 presents the mean reward for all the models across cheetah and cartpole simulations. Our analysis reveals that the strategy of employing constant real values with increasing frequency for the imaginary component of the initialization yields superior results.
Table 4: Summary of Experimental Results for Different Model Initializations
Model Initialization
Cartpole Reward
Cheetah
14
Reward
Constant
Random
85
21.36
Learnable
Random
85
9,04
Learnable
Increasing freq
155
285
Constant
Increasing freq
874
311.19

---

Comparison of rewards under jerk
800
600
Scores
400
200
Low
Medium
High
Camera Jerk Level
IQM
Koopman
IQM
Koopman Sac-AE - Mean
To-KPM 1QM
Sac-AE - 1QM
To-KPM Mean
Figure 8: Performance of models with camera jerk
Comparison of rewards under Gaussian Noise
1000
800
Score
400
200
0.01
0.05
0.1
0.15
0.2
0.25
Noise Std
Ours 1QM
Ours
To-KPM IQM
To-KPM
Figure 9: Performance of Models under Gaussian Noise
D.3 Ablation study on Performance under Imperfect Sensing
This section delves into the resilience of our model when faced with imperfect sensing conditions during evaluation. We specifically examine its performance in challenging scenarios: a.) Rainy Environment with Structured Noise: Unlike Gaussian noise, rain noise[52] presents more struc- tured and challenging interference, making it difficult for common denoising techniques to effec- tively mitigate[[3][[4]. We evaluate the model's performance under three distinct levels of rain density: Low (0.03), Medium (0.75), and High (0.0125). The comparison encompasses predictive models equipped with dynamic predictors, and the outcomes are depicted in Figure 7. The results indicate a general degradation control performance across all models under test, except for ours, which notably excels by achieving a reward of approximately 400. This demonstrates our model's superior capability to maintain effective system control even in the presence of high noise levels. b.) Imperfect Sensing due to Camera Jerk: To further assess our model's robustness, we intro-
25

---

duce random camera jerks into the video input stream, simulating real-world sensing imperfections. Three levels camera jerk are considered: Low jerk (SSIM > 0.8), Medium Jerk (SSIM between 0.4 and 0.5), and High Jerk (SSIM < 0.3). Our findings from Figure 8 reveal that our model con- sistently outperforms the others under these conditions as well. However, it's noteworthy that performance gap between our model and the To-KPM model narrows the jerk intensity increases, with both models exhibiting similar performance metrics higher jerk levels. Conversely, meth- ods based autoencoders demonstrate significantly lower performance across all jerk conditions.
These evaluations underscore our model's robustness and adaptability to imperfect sensing scenar- ios, highlighting its potential for real-world applications where sensing conditions are often less than ideal.
D.4 Ablation Study on Performance under Gaussian Noise
In this experiment, we analyze the robustness of our model's control performance under the influence of Gaussian noise. We introduce zero-mean Gaussian noise the input images with increasing standard deviation. Figure 9 illustrates the comparative performance of our model and to-kpm [16] model in the presence of Gaussian noise. We exclude models that under performed significantly from this figure, as their performance was too low to be meaningful. Notably, our model demonstrates exceptional resilience, maintaining high performance even under substantial Gaussian noise in the visual input.
26


Reviews:
[{"review_id": "dJ2eWVEkkv", "reviewer": "Reviewer_MzWx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a new model called the Contrastive Spectral Koopman Embedding network, which is designed to learn efficient linear representations from high-dimensional visual data. This model uses reinforcement learning to perform off-policy control based on the extracted representations, employing a linear controller for this purpose.", "summary_of_recommendation": "The paper is mostly clear and provides thorough theoretical and empirical analysis. However, the limited scope of test environments and the similarities between algorithms used in this work and the previous work (specifically To-KPM [16]) makes it appear to be an incremental effort focusing mainly on reducing computation loads. These weaknesses limited the contribution, hence the weak reject recommendation.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1. In Equation (2), the W seems to be introduced without prior context or explanation. Could you please clarify where it originates from? \n\nAlso see weaknesses summaries in reviews above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NDAEtRG0RF", "reviewer": "Reviewer_n623", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors consider control of nonlinear systems through visual scene feedback. They propose to employ spectrally linearized state representation (using Koopman operator) and then RL to perform off-policy control. The claim is that through the choice of complex eigenvalue representation one can achieve sample efficiency through this such task-specific and structure-exploiting representation.", "summary_of_recommendation": "The novelty of the contribution is not clear; in addition, the source of improved performance is also not clear.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "Explain the impact on performance of contrastive vs spectral part\nExplain differences from those prior missing references\nExplain novelty and contributions in view of above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CxG18mHerb", "reviewer": "Reviewer_eUws", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces RoboKoop, a novel contrastive spectral Koopman representations for visual RL; RoboKoop outperforms baseline visual RL algorithms on several DeepMind Control Suite tasks.", "summary_of_recommendation": "A nice work on learning linear latent representation for visual RL; however, its applicability to real-world hardware control may be limited.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "My questions for rebuttal are listed in the review section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Acwicc3xRv", "reviewer": "Reviewer_5VsF", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Robot control based on image observation is a challenging problem regarding learning efficiency and robustness. This study proposes a Contrastive Spectral Koopman Embedding network that achieves high computational efficiency and agility to disturbances by explicitly representing the linear state space introduced by Koopman Theory to a diagonal canonical form.  This paper also proves the convergence of the policy learning in the LQR problem on linearized latent spaces based on Koopman operators. It demonstrates its computational efficiency and high performance through standard image-based RL benchmarks.", "summary_of_recommendation": "Please see above", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "1. Can the Elapsed time be evaluated about Figure 3? In real robots, the overall computation time, including overhead, needs to be evaluated rather than the number of computational operations.\n1. Can the robustness of the physics parameter to disturbances be evaluated about Figure 4?\nIn robotics applications, the robustness of Sim2Real against environmental changes is also subject to evaluation.\n1. Does the assumption hold in a real system, the continuous Objective functions assumed for the proof of Theorem 4?\nAccording to the motivation for which Fazel et al. (2018) derived Lemma 6, it is non-trivial to derive a proof of convergence based on the gradient method for even simple LQR by continuity of the Objective function since the Feedback System (A-BK) can be unstable. Does the proposed method address this point? And did such instability occur in the data of the actual learning process?\n\nFazel, M., Ge, R., Kakade, S. &; Mesbahi, M.. (2018). Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator. Proceedings of the 35th International Conference on Machine Learning", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to CoRL 2024. Reviewers particularly appreciated the computational and accuracy improvements gained by the proposed spectral Koopman embedding network, the detailed theoretic analysis, and the principled approach.

Reviewers raised concerns about the limited scope of experiments, generalizability, and some missing related work and comparisons. They were unclear about aspects of the training method and concerned that it may be too incremental.

Reviewers valued the authors' detailed and thorough responses, which were sufficient to raise at least one reviewer's initial recommendation. However, concerns linger about generalizability and, in particular, the applicability to real-robot deployment. The reviewers and AC encourage the authors to consider extending experiments to address these lingering concerns for a camera-ready revision.
Decision: Accept