On the Utility of Koopman Operator Theory in Learning Dexterous Manipulation Skills
Authors: Yunhai Han
Mandy Xie
Ye Zhao
Harish Ravichandar
Venue: CoRL 2023 Oral (2023)
Keywords: Koopman Operator
Dexterous Manipulation

Abstract: Despite impressive dexterous manipulation capabilities enabled by learning-based approaches, we are yet to witness widespread adoption beyond well-resourced laboratories. This is likely due to practical limitations, such as significant computational burden, inscrutable learned behaviors, sensitivity to initialization, and the considerable technical expertise required for implementation. In this work, we investigate the utility of Koopman operator theory in alleviating these limitations. Koopman operators are simple yet powerful control-theoretic structures to represent complex nonlinear dynamics as linear systems in higher dimensions. Motivated by the fact that complex nonlinear dynamics underlie dexterous manipulation, we develop a Koopman operator-based imitation learning framework to learn the desired motions of both the robotic hand and the object simultaneously. We show that Koopman operators are surprisingly effective for dexterous manipulation and offer a number of unique benefits. Notably, policies can be learned analytically, drastically reducing computation burden and eliminating sensitivity to initialization and the need for painstaking hyperparameter optimization. Our experiments reveal that a Koopman operator-based approach can perform comparably to state-of-the-art imitation learning algorithms in terms of success rate and sample efficiency, while being an order of magnitude faster. Policy videos can be viewed at https://sites.google.com/view/kodex-corl.

On the Utility of Koopman Operator Theory in Learning Dexterous Manipulation Skills
Yunhai Han', Madie Xie Ye Zhao', Harish Ravichandar
GGergia Institute of Technology (yhan389, manxie, yezhao, harish ravi aavvihandarr www.ehannnnnndaceee.o.aeo @gatech.
Despite impressive dexterous manipulation capabilities enabled by learning-based approaches, we are yet witness widespread adoption beyond well-resourced laboratories. This is likely due to practical limitations, such as significant computational burden, inscrutable learned behaviors, sensitivity ini- tialization, and the considerable technical expertise required for implementation. In this work, investigate the utility of Koopman operator theory in alleviating these limitations. Koopman operators are simple yet powerful control-theoretie structures to represent complex nonlinear dynamics as linear systems in higher dimensions. Motivated by the fact that complex nonlinear dynamics underlie dex- terous manipulation, we develop a Koopman operator-based imitation learning framework to learn the desired motions of both the robotic hand and the object simultaneously. We show that Koopman operators are surprisingly effective for dexterous manipulation and offer number of unique benefits. Notably, policies can be learned analytically, drastically reducing computation burden and elim- inating sensitivity to initialization and the need for painstaking hyperparameter optimization. Our experiments reveal that a Koopman operator-based approach can perform comparably to state-of-the- oftthe-rrttttttt imitation learning algorithms in terms success rate and sample efficiency, while being an order of magnitude faster. Policy videos can be viewed
Abstract:
http:///iitss.ogllee.ooo wtt..////wwwwwoodeeeecoooo
Keywords: Koopman Operator, Dexterous Manipulation
Introduction
Autonomous dexterous manipulation skills are necessary for robots to successfully operate in a phys- ical world built by and for humans. However, achieving reliable robotic dexterous manipulation has been a long-standing challenge [1] due to numerous factors, such as complex nonlinear dynamics, high-dimenssonal action spaces, and the expertise required to design bespoke controllers.
Over the past decade, learning-basee based solutions have emerged as promising solutions that can address the challenges acquiring dexterous manipulation skills. Indeed, these methods have been shown to be capable of impressive feats, such as solving Rubik s cubes [2], manipulating Baoding balls [3], re- trieving tool trays [4], and reorienting complex objects [5, 6]. However, existing learning approaches suffer from practical limitations that hinder their widespread adoption. First, implementing exist- ing algorithms requires significant technical expertise and familiarity with modern machine learning infrastructure (e.g., knowledge of complex learning algorithms and well-established deep learn- ing software frameworks). Second, training policies consume significant computational resources. Third, while existing approaches have achieved impressive SOTA performance, these results tend to require painstaking efforts to tune hyperparameters and architectures. Fourth, performance tends to be highly sensitive parameter initialization.
In this work, we investigate the utility of Koopman operator theory in alleviating the limitations of existing learning- baasid approaches as identified above. The Koopman operator theory helps represent arbitrary nonlinear dynamics in finite dimensional spaces as linear dynamics in an infinite- dimensional Hilbert space [7]. While this equivalence is exact and fascinating from a theoretical standpoint, it is not tractable. However, recent advances have enabled the approximation of this equivalence in higher but fiite-diieeniinnal spaces by learning the operator directly from data [8].
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

I Lifee Koopman Reference Dynamics
Inverse Dynamies Controlier
Denonstrations
Exeration
X,
5
RCO
Operster)
KODex simultancously encodes complex nonlinear dynamics of the desired motion of both the robot state (x,) and the object state (x,) as a linear dynamical system in a higher-dimensional space by learning a Koopman operator K directly from demonstrations. Further, KODex learns an inverse dynamics controller to track the robot reference trajectory (()(()))))) generated by the lifted linear system.
Figure 1:
We develop a novel imitation learning framework, dubbed Koopman Operator-based Dexterous Manipulation (KODex), to evaluate the utility of Koopman operator theory for dexterous manip- ulation (see Fig. 1). Specifically, we model desired behaviors as solutions to nonlinear dynami- cal systems and learn Koopman operators that define approximaeel--qquvvlennn linear dynamics in nigher-dimeesionall spaces. Note that it is insufficient to exclusively focus on the robot's 's motion as the objective of dexterous manipulation is centered simultaneously learns the desired motions of both the robot and the object from demonstrations.
the object's motion [1]. As such, KODex
To eliminate the need for an expertly-tune -tuned controller, KODex utilizes a learned inverse dynamics controller to track the reference trajectory generated by the learned dynamical system.
significant benefit of learning Koopman operators from data is that it lends itself analytical solution. As such, KODex is simple to implement and does not require expertise and familiarity with state-of-the-art (SOTA) machine learning infrastructure. Instead of painstaking hyperparameter optimization, we show that generic and task-agnostic polynomial lifting functions are sufficient for KODex learn diverse dexterous manipulation skills. Further, KODex offers consistent and predictable performance since the learning process is analytical and thus not sensitive to parameter initialization. Finally, given that KODex learns a linear dynamical system, one can readily inspect the learned behaviors using a wide array of control-theoretto tools.
We carry out extensive evaluations of KODex within the context of four dexterous manipulation skills on the simulated Adroit hand, an established experimental platform for dexterous manipula- tion [9]. Further, we compare KODex against SOTA imitation learning approaches in terms of gen- eral efficacy, computational efficiency, and sample efficiency. Our results demonstrate that KODex at least an order of magnitude faster to train than SOTA imitation learning algorithms, while achieving comparable sample efficiency and task success rates. These results suggest that Koopman operators can be effective, efficient, and reliable tools to learn dexterous manipulation skills and to reduce the barriers to wide-spread adoption.
2 Related Work
In this section, we contextualize our contributions within relevant sub-fields.
Learning Manipulation Skills as Dynamical Systems: Our work falls into the category of mnnaicca-ssystem-aasee imitation learning methods for manipulation [10]. Made popular by the Dynamics Movement Primitives (DMPs) [11], these methods model robot motions as solutions to learnable dynamical system. The past decade witnessed a plethora of approaches built upon the same principle (e.g., [12-17]), creating increasingly-capate
LiD tools for manipulation. Robust- ness to perturbations, high sample efficiency, and provable convergence are all but a few examples of the many advantages of dynamical-system- system-basee approaches. These approaches tend to be highly structured and leverage control-theorets and topological tools to learn complex desired motions with unparalleled sample efficiency. Recent work also embeded the dynamical systems structure into deep neural networks to enable end-to-end learning [18]. These approaches were primarily designed to capture low-dimensional end-effector skills for serial-link manipulators. In contrast, our work investigates the utility of Koopman operators in learning dexterous manipulation skills on high-DOF platforms.

---

Learning Dexterous Manipulation Skills: Deep Reinforcement Learning (RL) has been dominat- ing the field of dexterous manipulation recently, enabling an impressive array of skills [3. 19, 20J. A popular approach demonstrated that a multi-finger hand can learn to solve the Rubik's s cube OpenAl et al. [2]. Recently, Chen et al. [5] developed model-free RI. framework capable of reorienting over 2000 differently-shaped objects, and Khandate et al. [6] combined sampling- based planning and model-free RL methods on the same task. Despite impressive successes, RL-based algorithms suffer from poor sample efficiency and notoriously-diffiuul training procedures. In contrast, Imita- tion learning (IL) aims to improve sample efficiency by leveraging expert demonstrations [10, 21J. However, most existing IL-based methods (including those discussed in Section 2) focus lower- DOF manipulators and do not scale well to sigh-dimensiona systems. Indeed, there are at least two recent notable exceptions to this limitation: Xie et al. [4] developed highly structured IL method to learn dexterous manipulation skills from demonstrations in the form of dynamical sys- tem; Arunachalam al. [22] introduced mixed-reality framework to collect high-quality demon- strations and learn dexterous manipulation skills by leveraging visual representations and motion retargeting. Researchers have also combined IL with RL get the best of both approaches and has been able to achieve impressive performance (e.g., [9, 23]). A common attribute of all exist- ing learning approaches dexterous manipulation is that they rely on significant computational resources and user expertise for implementation and hyperparameter tuning. Further, the effec- tiveness of these approaches is highly sensitive to parameter initialization [24]. In stark contrast, KODex encodes complex skills as dynamical systems which are analytically extracted from demon- strations. As such, KODex incurs a significantly smaller computational burden and eliminates the dependence on painstaking hyperparameter tuning and unreliable numerical optimization. Further, unlike opaque deep neural networks, KODex learns linear dynamical systems that can be inspected via -ohtroc-theoretic tools.
Koopman Operators in Robotics: Recently, Koopman operator theory has proven beneficial in various robotic systems, such as differential drive robots [25], spherical and serial-link manipula- tors [26], autonomous excavators [27, 28), and soft robotic manipulators [29, 30]. However, the systems investigated in these works are low-dimensional. In contrast, our work is focused on evalu- ating the effectiveness of Koopman operators in learning skills for a high-dimensional system with complex dynamics (i.c., a mulli-fingeree hand). Further, prior works have not sufficiently inves- tigated the relative benefits of leveraging Koopman operators over SOTA neural network-based approaches, and the circumstances under which these benefits hold. In our work, we thoroughly evaluate KODex against SOTA imitation learning methods within the context of multiple dexterous manipulation tasks.
3 Preliminary: Koopman Operator Theory
We begin by providing brief introduction to Koopman operator theory [7].
Koopman Representation: Consider a discrete-time autonomous nonlinear dynamical system
x(t + 1) = =x(x(t)),
where x(t) X C R" is the state at time t, and F(-): RD + R" is nonlinear function.
To represent the the nonlinear dynamical system in (1) as linear system, we begin by introducing a set of observables using the so-called lifting function g X + O, where O is the space of observables. We can now define the Koopman Operator K., an infinitedimmessional operator on the lifting function g(-) for the discrete time system defined in (1) as follows
[Kg] ((F(x())) g(x(t 1))
If the observables belong vector space, the Operator KC can be seen as an infiniteddimensional linear map that describes the evolution of the observables as follows
g(x(t= + 1)) = Kg(x(())
(3)
Therefore, the Koopman operator K. linearly propagates forward the infinite-dimensional lifted states (i.e., observables). In practice, we do not benefit from this representation since it is infinite- dimensional. However, we can approximate AC using a matrix K E RPXP and define a finite set of observables s(t) E RP. Thus, we can rewrite the relationship in (3) as
((x(t + 1)) Kσ(x(1)) + r(x())),
(4)
3

---

where r(x(t)) R' is the residual error caused by the finite dimensional approximation, which can be arbitrarily reduced based on the choice p.
Learning Koopman Operator from Data:
The matrix operator K can be inferred from a dataset D = x((), x(T),, which contains the solution to the system in (1) for T time steps. Given the choice of observables --(), the finite dimensional Koopman matrix K is computed by minimizing the approximation error defined in (4). Specifically, we can obtain K from D by minimizing the cost function J (K) given below
1 T=T-1
J(K)
Σ
[x(((t))
Σ
1lo(x(t 1)) (o(x(t))l
(5)
t=1
Note minimizing (K) amounts to solving least-square problem, whose solution is given by [8]
t=T-1
K = AGI; T
(xx(++1)) G T Σ
o(x())) (6)
where G denotes the Moore-Penrose inverse' of G, and @ denotes the outer product.
4 Learning Koopman Operators for Dexterous Manipulation
We begin by introducing our framework model dexterous manipulation skills as nonlinear dynam- ies and discuss the importance of incorporating object states into the system (Section 4.1). Next, we describe how KODex learns the reference dynamics" for a given skill from demonstrations (Sec- tion 4.2). Then, we discuss how to learn a low-level controller, also from demonstrations, in order to faithfully track the reference trajectories generated by KODex (Section 4.3). Finally, we dis- cuss policy execution (Section 4.4). Further, an overall pseudo-code for KODex can be found in Appendix
4.1 Modeling Dexterous Manipulation Skills
A central principle behind KODex is that the desired behavior of a robot can be represented using a dynamical system. Note that, unlike other kinds of manipulation skills (e.g.. end-effector skills of multi-link manipulators), dexterous manipulation is explicitly concerned with how an object moves as result of the robot's motion [1]. As such, KODex captures the desired motion of the robot along
with that of the object. To this end, we define the state at time t as x(t) = [x, (t) (t) JT, where (l) X, C Rn and x!) E X, C T represent the state of the robot and the object, respectively, at time f. As such, the dynamical system we wish to capture is
x(1 1) F==
where F*(+): : Xr XX + Xr Xx denotes the true dynamics that govern the interdependent motions of the robot and the object. Note that this system is time-invariant Indeed, inneiinariannt dynamical systems provide natural way to capture manipulation skills that are more robust to intermittent perturbations than those that explicitly depend on time [10].
A key challenge in learning the dynamical system in (7) is that it can be arbitrarily complex and highly nonlinear, depending on the particular skill of interest. KODex leverages Koopman operator theory to learn a linear dynamical system that can effectively approximate such complex nonlinear dynamics. To this end, we first define a set of observables as follows
<(x(()) [xr (1) th (x),
(xx(()))) V
where v, E. + Rd and v. IR - m are vector-valued lifting functions that transform the robot and object state respectively. While there are no coupling terms between the robot and the object states in (8), note that the robot and object states still mix after the lifting operation.
could efficiently computed using thescipy. linalg pinv (G) function from Scipy library. We use the term "reference dynamics" to describe a fictitious dynamical system that encodes task -seciicc ideal trajectories the configuration space, and not the physical robot dynamics.

---

In our implementation. we use polynomial functions up to finite degree in our lifting function since polynomial functions allow for flexible definition of complex functions. However, it important to note that KODex is agnostic to the specific choice of observables. Further, we do not assume that we know the ideal set observables for any given skill. Instead, as we demonstrate in Section 5, KODex can learn different dexterous manipulation skills on the same space of observables.
4.2 Learning Reference Dynamics
We now turn to the challenge of learning the Koopman operator K from demonstrations. Let (t),
.... (t)) ing trajectories of state-torque pairs. Now, we can compute the Koopman matrix as K = AGT, where A and G can be computed by modifying the expressions in (6) follows
denote set of N demonstrations contain-
n= N (
6(x" (++ 1)) N(T(m)-1)
n= N= (TT()))
((x & Oxx N(T()) 1)
G=
Σ
(9)
It is worth noting that KODex can also leverage partial trajectories that do not complete the task, as long as all the state pairs (x" (t), (t++ are temporally consecutive. Additionally, we also record the actuated torque (l) at each time step for the controller design discussed in Section 4.3.
We use the learned reference dynamics to generate rollouts in observables space. However, we need to obtain the rollouts in the original robot states to command the robot. Since we designed (xx(t) such that both robot state x, is a part of the observables in (8), we can retrieve the desired robot trajectory (t)) by selecting the corresponding elements in (xx(t)).
Indeed, the data distribution in has considerable effect on the generalizability of the computed Koopman matrix K. Therefore, there is an inevitable trade-off between the number of demonstra- tions and the cost of data collection challenge shared by most imitation learning algorithms.
4.3 Learning a Tracking Controller
To track the desired trajectories generated from the learned reference dynamics, we learn an inverse dynamics controller C [31] [18]. Indeed, PD controller could be designed instead of learning tracking controller. However, one would have to painstakingly tune the control gains and frequency, and so for each task independently.
We use a multi-layer perception (MLP) as the tracking controller and train it using the recorded state-torque pairs (xx (t), (t + 1), (t)) by minimizing
R= (=T())
(xx, -
Coontol = Σ Σ
(10)
((T
The learned controller takes as input the current robot state x, (f) and the desired next state from the reference trajectory (t 1). and generates the torque r(t) required for actuation.
4.4 Execution
With the reference dynamics and the tracking control learned, we specify how to execute the policy in this section. Suppose x(1) (x,, xx)))) ( is the given initial state, generate the reference trajectory (t)) by propagating the learned reference dynamics x(t + 1) = K&(t). Further, at time step t, we pass the current robot state xt) and the desired next robot state X, + 1) to the learned controller C' to compute the required torque (t)).
Experimental Evaluation
We evaluated KODex along with existing approaches in terms of their general efficacy, computa- tional efficiency, sample efficiency, and scalability when learning dexterous manipulation skills.
5

---

Figure 2: We evaluate KODex on four tasks from [9]: Tool Use, Door Opening, Relocation, and Reorientation.
5.1 Experimental Design
Evaluation Platform: We conducted all our experiments the widely-used ADROIT Hand [9]
a 30-DoF simulated system (24-DoF hand + 6-DoF floating wrist base) built with MuJoCo [32].
Baselines: We compared KODex against the following baselines:
NV: Fully-connected neural network policy
LSTM: Recurrent neural network policy with Long Short-Term Memory (LSTM) units
NDP: Neural Dynamic policy [18]
NGF: Neural Geometric Fabrics policy [4]
Note that NN and LSTM are unstructured baselines which help question the need for structured policies. NDP and NGF are highly-structured SOTA imitation learning methods for manipulation.
We undertook several precautions to ensure a fair comparison. First, we designed the robot and object state space for all baselines and KODex to identical. Second, we carefully designed the baselines policies and tuned their hyper-parameters for each baseline method (Appendices E and F). Third, we trained each baseline policy over five random seeds to control for initialization effects. For all tasks, saved the baseline policies that performed the best validation set of 50 held-out demonstrations. Note that KODex utilizes an analytical solution and thus does not require parameter initialization or hyper parameter optimization.
Tasks: We evaluated all algorithms on set of four tasks originally proposed in [9] (see Fig. 2).
Tool use: Pick up the hammer to drive the nail into the board placed at a randomized height.
Door opening:
Given randomized door position, undo the latch and drag the door open. Object relocation: Move the blue ball to randomized target location (green sphere).
In-hand reorientation: Reorient the blue pen randomized goal orientation (green pen).
See Appendix B for the state space design of all tasks. For all tasks, the reference dynamics was 500HZ.
queried at 100HZ, and the controller
Metrics: We quantify performance in terms of Training time: Time taken to train a policy, ii) Imitation error: The L1 distance between generated joint trajectories and the demonstrations, and iii) Task success rate: Percentage successful trials (see Appendix C for success criteria).
Expert Policy: For each task, we trained an expert RL agent using DAPG [9] to generate 250 expert demonstrations (200 for training and 50 for validation). See Appendix D for further details.
Inverse Dynamics Controller: To standardize controller performance across methods, we trained common inverse dynamics controller for each task using 250 demonstrations (see Appendix G).
5.2 General Efficacy
In Fig. 3, we report the training time, imitation error, and task success rate for each method on each task, when trained on 200 demonstrations and tested on 10,000 testing instances.
Training time: As can be seen, KODex is order of magnitude faster than both unstructured baselines (NN, LSTM) and SOTA IL methods (NDP, NGF). Further, this trend holds across all the tasks. This is to be expected since KODex analytically computes the Koopman operator unlike all the baselines, which rely on gradient descent and numerical optimization.
Imitation error: In general, all methods (except NN) achieve low imitation enror for the Tool Use task with negligible difference across methods. the three remaining tasks, we see that struc- tured methods (NDP, NGF, KODex) considerably outperform the unstructured baseline (LSTM).

---

NN
LSTM
NDP
NGF
KODEX
Training
Imitation
0.20
Thaining Time (Soc)
400
Inaitaioon Etror (rad)
Task Success Rate (5)
100
75
500
0.10
200
50
100
0.05
25
Tool
Door
Reloc
Reorient
0.00
Tool
Dour
Reloc
Reurient
0
Tool
Door
Reloz
Rearient
Tasks
Tasks
Tasks
Figure 3: We report training time (left), imitation enror (middle), and success rate (right) for methods on each task when trained 200 demonstrations and evaluated on independent set of 10,000 samples. Error bars for baseline methods, show the standard deviation over five random sceds.
NN
LSTM
NDP
NGF
KODex
Thinng
Thinig
Thinnng
Thiinngg
300
Trining (50)
Trinny Times (Saz)
Trining 6
Trinug Times Timer(aap (Sa0
300
100
300
300
100
100
100
100
Number
Number Demonstirtions
Number
Initaion Ence Dunseet Size
Iniusion Erooe vs Duraset Size
Intution Erae Dusaset Size
Intesion Ervoe vs Duzaset Size
0.4
Initston Erey (ne)
0.4
Imilsiion Emos (rel)
Ereer (me))
Inilutio Emoo (re))
3.3
0.3
0.3
02
0.2
0.0
25
50
100
150
200
25
50
100
150
200
25
50
100
150
200
25
50
100
150
200
Nanberr of Demotolraaiota
Number
Nanber of Danetotratitt
Mamber
Taxt
Desest
Task
100)
100
100
100
Rtae (bae (8)
Taxk Stocess (tae (8)
(8))
Tosk Raae (8)
75
Stocens
50
30
24
Namber
Number Demonsiraions
Nomhor Dimmanttiaions
Number Demoonttaaions
(a) Tool Use (Hammer)
(b) Door Opening (c) Object Relocation
(d) In-hand Reorientation
Figure 4: The effects number of demonstrations on training time (top row), imitation error (middle row), and success rate (bottom row) for all methods on each task. Solid lines indicate mean trends and shaded areas show + standard deviation over five random seeds.
We have excluded the significantly larger imitation errors generated by NN from the plot to preserve the resolution necessary distinguish between the other methods. These results reinforce the effec- tiveness of structured methods in imitating demonstrations. Importantly, KODex is able to achieve imitation performance comparable the SOTA IL methods despite its simplicity, while remaining an order of magnitude more computationally efficient.
Task success rate: As expected, the NN policy performs significantly worse than all other methods. On the other hand, LSTM achieves impressive task success rates, even outperforming NDP two of the four tasks. This is in stark contrast its high imitation error. While counter-intuitive, this observation follows the recent finding that imitation error and task success rate might not necessarily be correlated [33]. We observe that KODex and NGF perform comparably, with achieving higher task success rate than the other in two of the four tasks. Importantly, KODex results in the most consistent and predictable performance due its lack of sensitivity to initialization.
7

---

5.3 Scalability and Sample Efficiency
To investigate scalability and sample efficiency, we trained policies on varying number of demon- strations ([10, 25, 50, 100, 150, 200]). In Fig. 4, we report the training time, imitation error, and task success rate for each method
function of the number of demonstrations when tested on the same 10,000 instances used to evaluate general efficacy.
Training time: We observe that KODex scales with the number of demonstrations significantly better than the baselines, as evidenced by its training time growing at considerably lower rates.
Imitation error and Success rate: We find that unstructured models (NN and LSTM) fail to demon- strate a consistent monotonic decrease (increase) in imitation error (task success rate) as the number of demonstrations increase. In stark contrast, structured methods (NDP, NGF, and KODex) are able to consistently drive down imitation error and improve task success rate. KODex almost consis- tently achieves the lowest imitation error and the highest task success rate with the fewest number of demonstrations, and is closely followed by NGF. These observations suggest that KODex tends to be comparably, if not more sample efficient than the baselines, thanks to the rich structure induced by the Koopman operator and the resulting effectiveness in capturing nonlinear dynamics. The only exception to this trend is the Object Relocation task, in which KODex requires 1,50 demonstrations to perform comparably to NGF. We speculate this is because the demonstrations for this task exhibit high variance the hand base moves across large space, and KODex requires more demonstrations to capture the reference dynamics.
5.4 Additional Experiments
Additional experiments reported in the appendix suggest that KODex learns policies that i) have inference time on par with SOTA baselines (Appendix H), ii) have zero-shot out-of distribbitt gen- eralization comparable to SOTA IL methods (Appendix I), iii) properties (Appendix J), iv) are not overly sensitive to the choice of basis functions (Appendix K), v) are nearly-stable linear dynamical systems that generate safe and smooth robot trajectories (Ap- pendix L), and vi) are significantly more efficient and scalable than a baseline BC method that directly learns state-action mappings (Appendix M).
robust to changes in physical
6 Conclusions
We investigated the utility of Koopman operator theory in learning dexterous manipulation skills by encoding complex nonlinear reference dynamics as linear dynamical systems higher-dimensional spaces. Our investigations conclusively show that Koopman-based framework can i) analytically learn dexterous manipulation skills, eliminating the sensitivity to initialization and reducing the need for user expertise, and ii) match or outperform SOTA imitation learning approaches various dexterous manipulation tasks, while being an order of magnitude faster.
Limitations and Future Work
While work offers promise for the utility of Koopman operators in dexterous manipulation, it reveals numerous avenues for further improvement. First, we did not deploy KODex on physical robots. Although our results on robustness changes in physical properties show promise, we plan to deploy KODex on physical platforms to translate our findings to hardware. Second, we only considered polynomial basis functions; other non-smooth functions (e.g., ReLU [34]) could be beneficial to manipulation tasks involving friction and contact. Third, KODex has limited out- of-distribution generalization, much like most existing imitation learning approaches. Future work can investigate if additional data collection [35] and learned lifting functions [36-38] alleviate this concern. Fourth, KODex relies on demonstrated action trajectories to learn the tracking controller and reduce human effort. It might be possible to instead use reinforcement learning [39], thereby enabling the ability learn from state-only observations [40]. Fifth, KODex could be evaluated on other domains and robotics tasks (e.g., the benchmark tasks in [41]) to further understand the trade- offs between KODex and other imitation learning approaches. Finally, Koopman operators can be used to learn system dynamics via self play to enable model-based reinforcement learning.
8

---

References
[1] A. M. Okamura, N. Smaby, and M. R. Cutkosky. An overview of dexterous manipulation. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 1, pages 255-262, 2000.
[2] OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning Dexterous In-Hand Manipulation. International Journal of Robotics Research (IJRR), 2020.
[3] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[4] M. Xie, A. Handa, S. Tyree, D. Fox, H. Ravichandar, N. D. Ratliff, and K. Van Wyk. Neu- ral geometric fabrics: Efficiently learning high-dimensional policies from demonstration. In Conference Robot Learning, pages 1355-1367. PMLR, 2023.
[5] Chen, J. Xu, and P. Agrawal. System for General In-Hand Object Re-Orientation. In Sth Annual Conference Robot Learning, 2021.
[6] G. Khandate, S. Shang, based Exploration for Reinforcement Learning of Dexterous Manipulation. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023, 1o..10.55607/RSS 2023.XIX.020
Chang. Saidi, Adams, and M. Ciocarlie. Sampling-
[7] B. O. Koopman. Hamiltonian Systems and Transformation in Hilbert Space. Proceedings of the National Academy of Sciences, 7(5):315-311 1931. 10.733../s/psas. 17.5.31..
[8] M. O. Williams, I. G. Kevrekidis, and C. W. Rowley. A data approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25(6)1307-11366, 2015.
[9] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demon- strations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
[10] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3(1):297-330, 2020.
[11] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation, 25(2):328- 373, 2013.
[12] S. M. Khansari-Zadch and Billard. Learning stable nonlinear dynamical systems with gaus- sian mixture models. IEEE Transactions Robotics, 27(5):943-957, 2011.
[13] K. Neumann and Steil. Learning robot motions with stable dynamical systems under diffeomorphic transformations, Robotics and Autonomous Systems, 70:1-15, 2015.
[14] H. Ravichandar, Salehi, and A. Dani. Learning Partially Contracting Dynamical Systems from Demonstrations, In Proceedings of the 1st Annual Conference on Robot Learning, vol- ume 78. pages 369-378, 13-15 Nov 2017.
[15] M. A. Rana, A. Li, H. Ravichandar, M. Mukadam, S. Chernova, D. Fox, B. Boots, and N. Ratliff. Learning reactive motion policies in multiple task spaces from human demon- strations. In Conference on Robot Learning, pages 1457-1468. PMLR, 2020.
[16] M. A. Rana, A. Li, D. Fox, B. Boots, F. Ramos, and N. Ratifff Fuclideanizing flows: Dif- feomorphic reduction for learning stable dynamical systems. In Learning for Dynamics and Control, pages 630-639, PMLR, 2020.

---

[17] N. Figueroa and A. Billard. Locally active globally stable dynamical systems: Theory, learn- ing. and experiments. The International Journal of Robotics Research, 41(3):312-347, 2022.
[18] Bahl, M. Mukadam, A. a. Gupta, and D. Pathak. Neural dynamic policies for end-to-end sensorimotor learning. Advances in Neural Information Processing Systems, 33:5058-5069, 2020.
[19] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. arXiv preprint arXiv:2210.045500 04887, 2022.
[20] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Roboties and Automation (ICRA), pages 3651-3657 IEEE, 2019.
[21] Osa, J. Pajarinen, Neumann, A. Bagnell, P. Abbeel, J. Peters, et al. algorithmic perspective imitation learning, Foundations and Trends@ in Robotics, 7(1-2):1-179, 2018.
[22] S. P. Arumachalam, I. Güzey, S. Chintala, and L. Pinto. Holo-dex: Teaching dexterity with immersive mixed reality, arXiv preprint arXiv:2210.0646. 2022.
[23] V. Kumar, A. Gupta, E. Todorov, and S. Levine. Learning dexterous manipulation policies from experience and imitation. arXiv preprint arXiv:1611.05000 2016.
[24] Henderson, Islam, Bachman, Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, 2018.
[25] I. Abraham, G. De La Torre, and T. D. Murphey. Model-based control using Koopman opera- tors. arXiv preprint arXiv:1709.011
2017.
[26] I. Abraham and T. D. Murphey. Active Learning of Dynamics for Data-Driven Control Using Koopman Operators. IEEE Transactions Robotics, 35(5):1071-1083, 2019. doi:10. 1009 TRR0.0019.923880..
[27] F. E. Sotiropoulos and H. H. Asada, Dynamic Modeling of Bucket-Soil Interactions Using Koopman- Lifting Linearization for Model Predictive Contouring Control of Autonomous Excavators. IEEE Robotics and Automation Letters, 7(1):151-158, 2022. doi: 109/LRA. 2021.3121136.
[28] N. S. Selby and H. H. Asada. Learning of causal observable functions for koopman-dfl lift- ing linearization of nonlinear controlled systems and its application to excavation automa- tion. IEEE Robotics and Automation Letters, 6(4):6297-6304, 2021. 101.:0.1100/LRA..001.. 3092256.
[29] D. Bruder, B. Gillespie, C. D. Remy, and Vasudevan. Modeling and control of soft robots using the koopman operator and model predictive control. arXiv preprint arXiv: 1902.0287, 2019.
[30] D. Bruder, X. Fu, R. B. Gillespie, C. D. Remy, and R. Vasudevan. Koopman Based Control of a Soft Continuum Manipulator Under Variable Loading Conditions. IEEE Roboties and Automation Letters, 6(4):6852 6859, 2021. doi: R1..0//.........009...
Saoppan--Basedd
[31] Hanna and P. Stone. Grounded action transformation for robot learning in simulation. In Thirty-first AAAI conference artificial intelligence, 2017.
[32] Todorov, Emanuel and Erez, Tom and Tassa, Yuval. Mujoco: A physics engine for model- based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026 5026-5033, 2012.
[33] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkami, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2 03298, 2021.
[34] A. F. Agarap. Deep learning using rectified linear units (relu). arXiv preprint
arXiv:1803.08375.0 2018.
10

---

[35] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[36] M. Weissenbacher, S. Sinha, A. Garg, and K. Yoshinobu. Koopman q-learning: Offline re- inforcement learning via symmetries of dynamics. In International Conference Machine Learning, pages
PMLR, 2022.
[37] B. Lusch, J. N. Kutz, and S. L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamies, Nature communications, 9(1):1-10, 2018.
[38] Li, H. He, J. Wu, D. Katabi, and Torralba. Learning compositional Koopman operators for model-based control. In International Conference on Learning Representations, 2020. URL pttps://openrevieee
weee//oormmiiieeee H11
[39] X. B. Peng, E. Coumans, Zhang, T.-W. Lee, Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.0077 2020.
[40] F. Torabi, G. Warnell, and P. Stone. Recent advances in imitation learning from observation. arXiv preprint 1905, 13566, 2019.
[41] A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.P. Berges, Abbeel, J. Malik, al. Where are in the search for artificial visual cortex for embodied intelligence? arXiv preprint 230v 18240, 2023.
[42] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10): 2222-2232, 2016.
[43] K. Van Wyk, M. Xie, A. Li, M. A. Rana, B. Babich, B. Peele, Q. Wan, I. Akinola, B. Sun- daralingam, D. Fox, al. Geometrie fabrics: Generalizing classical mechanics to capture the physics of behavior. IEEE Roboties and Automation Letters, 2022.
[44] C.-A. Cheng, M. Mukadam, J. Issac, S. Birchfield, D. Fox, B. Boots, and N. Ratliff. Rmpflow: geometric framework for generation of multitask motion policies. JEEE Transactions on Automation Science and Engineering, 18(3):668-955 2021.
[45] Ho and S. Ermon. Generative adversarial imitation learning, Advances in neural information processing systems, 29, 2016.

---

Appendices
A KODex Pseudo-code
The overall pseudo-code for KODex is shown below.
Algorithm 1: KODex
Demonstration Data Collection
Initialize D Ø3;
for [1,...,NN do
Generate (T)) trajectory states and torques ([x" ( (O]]TT to D;
Add end
Koopman Operator Approximation
Determine lifting function d(x()));
Compute K on D (6, 9);
Controller Design
Build controller C' as neural network with inputs as (t,,,,,,,t 1)) and output r(t):
Train C using state-torque pairs (x.))), (t 1), (t)) ) in (10);
Execution
Specify the initial states x(1); for & {l,..., 1} do
Predict the next robot states &, (t + 1) using K (3 8);
Read the current robot states x, (();
Generate the torque T() using C on (1), (1 + 1)) and execute it;
end
B
State Design
In this section, we show the state design for each task in detail. It should be noted that the motion capability of the hand for each task were suggested from the work [9] that originally introduced these tasks. For a decent implementation,
employed the same setting.
Tool use For this task, the floating wrist base can only rotate along the x and y axis, so we have x!( E X. C R Regarding the object states, unlike the other tasks, where the objects of interest are directly manipulated by the hand, this task requires to modify the environment it- self. As result, except for the hammer positions, orientations and their corresponding veloci- ties p. of toal ófool (R.), we also define the nail goal position p (RR3). Finally, we have xo(t) [pt ouoo p 6p00 mall E Xo C 15 As result, x(t) includes 41 states in total and we use = 100.
Door opening For this task, the floating wrist base can only move along the direction that is perpen- dicular to the door plane but rotate freely, so we have x, (t) E X, C 2828 Regarding the object states, define the fixed door position p which can provide with case-specific information (similar to paiil in Tool Use), and the handle positions handle (both R$), In order to take into consideration the status of door being opened, include the angular velocity of the opening angle (IR))). Finally, we have x!) = handle Vt, door] E X, C R.. As result, x(t) includes 35 states in total and we use 70. 70. Object relocation For this task, the ADROIT hand is fully actuated, so we have (t) E X" C. R330 (24-DoF hand 6-DoF floating wrist base). Regarding the object states, we define plurget and phlll as the target and current positions. Then, we compute phall = =pphal - tagget which is the component of hall in new coordinate frame that constructed by parget being the origin. We additional in- clude the ball orientation ohall and their corresponding velocities hhall ohail (all R²), Finally, we have
(I) balll of pt (hall) E X. c 12, As result, x(1) includes states in total and we use T 100.
In-hand reorientation For this task, the floating wrist base is fixed, so we only consider the 24-DoF hand joints. Therefore, we have (1) E X, C R221 Regarding the object states, we define geol
12

---

and of as the goal and current pen orientations, which are both unit direction vectors. Then, we transform of to a new rotated coordinate frame that constructed by 8gedl being I axis ([1,0.0). Note that the vector of after transformation is also a unit vector and it converges to x axis if the pen perfectly manipulated to goal orientation oBeal In addition, we also include the cen- ter of mass position pr and their corresponding velocities p 6,5hh (all R²). Finally, we have pe o me P: , E X, C "2 As a result, x(t) includes 36 states in total and we use
(t) = [p] T 100.
In this work, we only included the joint positions as the robot states (with the only exception of NGF's second-order policy) for the following reasons: 1) Given that these tasks are not repetitive, we found that joint position information was sufficient to disambiguate the robot's next action, 2) even when ambiguity arises for a given joint position, object state information can help with disam- biguation. Further, the impressive performance achieved by KODex in our experiments support this design choice. Indeed, KODex is agnostic to this specific state design. One can incorporate velocity information into the robot state space without the need of any changes to the training procedure.
C Task Success Criteria
The task success criteria are listed below. The settings were the same as proposed [9]. Tool Use: The task is considered successful if at last time step T, the Euclidean distance between the final nail position and the goal nail position is smaller than 0.01.
Door Opening: The task is considered successful if at last time step T, the door opening angle is larger than 1.35 rad.
Object Relocation: At each time step t, if V Ip - p < 0.10, then we have p(t) = 1. The task is considered successful if E p(t) > 10.
In-hand Reorientation: At each time step t, if 080al o.Th > 0.90 (o oo measures orientation
similarity), then we have p(t) 1. The task is considered successful if E p(t) > 10.
ET=T(())
D Sampling Procedure
We describe the sampling procedure in this section. The sample distributions used for RL training and demo collection were identical, as suggested in [9]. The out-of-distribution data were generated to evaluate the zero-shot out-of-distribution generalizability
each policy.
Tool Use: We randomly sampled the nail heights (h) from uniform distributions. Within distri- bution: we used h E H ~ 0.20)) Out of distribution: we used E H. ~ U(0.05, 0.1) U
u(0.25,0.33 0.3).
Door Opening: We randomly sampled the door positions (zyz) from uniform distributions. Within distribution: we used X ((3033 0), y y 200.2, 0.35), and 2 L4(02552 0..02))) Out of distribution: we used E y ~ (/(.11,0000 0.2) (24(0.55, 0.4) (z, 2 remained unchanged).
Object Relocation: We randomly sampled the target positions (Izz2 from uniform distributions.
Within distribution: we used x E X ~ u(-0.25, 0.25), E y ~ u(-0.25, 05,5,,666 0.25), and E Z ((0115, 0.35); Out of distribution: we used z Z ((0.35, 0.40) (x, y remained unchanged).
In-hand Reorientation: We randomly sampled the pitch (a) and yaw (3) angles of the goal orientation from uniform distributions. Within distribution: we used a E A ~ u(-1,1) and B U(-1, 1); Out of distribution: we used {(a, 3) E (A, (U(-1, ((,
(1(1, 1.2), ((-12, 1) 14 -1
E
Policy Design
We show the detailed policy design in this section. All the baseline policies were trained to minimize the trajectory reproduction error.
KODex: The representation of the system is given as: x, [ x , xx1] and x, [nrd, 22 ... ..m) and superscript used to index states. The details of the state design for each task provided in Appendix B. In experiments, the vector- lifting functions v and so in (8)
13

---

were polynomial basis function defined as
1ト -x]], {( } for i.j = 1, ........ ,
(11)
v. ( U (xg)) for i,j 1,.... m
Note that a//.////f. only appears once in lifting functions (similar to xb and we ignore as the lifting functions are the same across the time horizon.
The choice of lifting functions can be viewed as the hyper-parameter of KODex. We make this choice as inspired from [25] and experimental results also indicate its effectiveness. Through all the experiments, we sticked with the same set of lifting functions, which helped to relieve us from extensive efforts of tuning the hyper-parametent e.g. network layer size, that were necessary for baseline policies as shown in Appendix F.
The first baseline is a feedforward network that ingests the states x(1) and iteratively produces the predictions x()),= 2, .... T via the rollout of a Multilayer Perceptron (MLP). The reference joint trajectories (t)) are then used to execute the robot with the learned controller C. The significance of this baseline is to evaluate policy that produces high-dimensional motion without any additional structure.
Full-connected Neural Network (NN):
Long Short-Term Memory (LSTM): We create an LSTM-based policy under the same input- output flow as the NN policy. We also apply two fully-connected layers between the task in- put/output and the input/hidden state of the LSTM network. Similarly, the same controller C is deployed to track the reference joint trajectory. LSTM networks are known to be beneficial to imi- tation learning [33] and suitable for sequential processing [42], e.g, motion generation. Therefore, we expect to evaluate the performance of the recurrent structures in these tasks.
Neural Dynamic Policy (NDP): The Neural Dynamic Policy [18] embeds desired dynamical struc- ture as a layer in neural networks. Specifically, the parameters of the second order Dynamics Motion Primitives (DMP) are predicted as outputs of the preceding layers (MLP in [18]). As a result, it al- lows the overall policy easily reason in the space of trajectories and can be utilized for learning from demonstration. We train an NDP policy following the imitation learning pipeline described in [18]. For each task, given x(1), the neural network components in NDP generate the parameters of DMPs (radial basis functions (RBFs) [18]), which are forward integrated to produce the reference joint trajectories for tracking.
Neural Geometric Fabrics policy (NGF): The Neural Geometric Fabrics [4), a structured pol- icy class, that enables efficient skill learning for dexterous manipulation from demonstrations by leveraging structures induced by Geometric Fabrics [43]. Geometric Fabrics is a stable class of the Riemannian Motion Policy (RMP) [44]. It has been demonstrated that NGF outperforms RMP in policy learning for dexterous manipulation task in [4]. The NGF policy is defined in the con- figuration space of the robot, which is composed of geometric policy, potential policy and damping term. More specifically, the NGF policy is constructed as follows: (1) define a geometric policy pair M,, T] and potential policy pair [M,, / in the configuration space q, (2) energize the geometric policy (project orthogonal to the direction of motion with p.) to create a collection of peerry-preeeviingg paths (the Geometric Fabric), and (3) force the Geometric Fabric with potential defined by [M,, و and damp via b applied along 4, which ensures convergence to the potential's minima. The potential policy y is the gradient of function of position only. Note that we param- eterize the geometric policy pair [M, r.. the potential policy pair [M,, Ty], and the damping scalar 6 with MLP networks and learn them from demonstration data.
F
Optimizing baseline model size
As described in Appendix E, we sticked with the same set of lifting functions for KODex and report the task success rate when we trained KODex on training set and tested it validation set in Table. I. However, for baselines, the for each task using the training set over three choices of model size, including small size, median size and large size. We generated five random seeds for parameter initialization per model size, per baseline, and per task, as all learning based baseline models are sensitive parameter initialization [24]. For each baseline policy, we report the mean and standard deviation of the task success rate on the validation set over five random seeds in Tables. 2-5.
were selected through a set of ablation experiments
Based on these results, we selectd the model size that offers the best performance in terms of task success rate. In addition, these results indicate that, unlike KODex, extensive parameet tun-
14

---

ing and various trials on parameter initialization for baseline models are necessary. Note that we use I denote dim(x(t)).
Table I: Task success rate validation set (KODex)
Tool
Door
Relocation
Reorientation
100.0%
96.0%
88.0%
62.0%
Table 2: Hyper-parameters on NN Network Sizes
Success Rate (%)
Task
Tool
Door
Relocation
Reorientation
Model Size
MIP: (32, 64, 32)
0.4(
0.0(=0.0)
0.4(100.8
6.8(13.9)
MLP: (64, 128, 64)
0.0(#0.0)
0.4(#0.8)
1.2(+2.4)
10.4(+:6.6)
MLP: (128, 256, 128)
0.0(+0.0)
0.0(
0.8(+1.6)
6.0(+11.5)
Table 3: Hyper-parameters on LSTM Network Sizes
Success Rate (%)
Task
Tool
Door
Relocation
Reorientation
Model Size
LSTM: 200
28.8(+25.0)
87.6(+10.3)
7.6(+)59)
56.4(:17.4)
(1, 100), (200, I)
LSTM: 250
60.8(+36.6)
80.8(+-24.5)
7.6(+77.5)
48.0(+17.00)
fc: (1, 175), (250, 1)
LSTM: 300
44.8(+31.8)
82.0(+13.9)
16.4(+14.5)
54.0(+11.0)
fc: (1, 250), (300, 1)
Table 4: Hyper-parameters on NDP Network Sizes
Success Rate (%)
Task
Tool
Door
Relocation
Reorientation
Model Size
MLP: (32, 64, 32)
0.0(++0.0)
8.0(+2.5)
30.0(#9.3)
57.2(+8.6)
10 RBFs
MLP: (64, 128, 64) 20 RBFs
16.8(++29.8)
40.8(+8.1)
74.0(+14.9)
59.2(+66.5)
MLP: (128, 256, 128)
18.4(+31.9)
66.0(+5.2)
79.2(+7..7)
62.4(++.8)
30 RBFs
Table 5: Hyper-parameters on NGF Network Sizes
Success Rate (%)
Task
Tool
Door
Relocation
Reorientation
Model Size
MLP: (64, 32)
99.2(+1.6)
87.2(+12.0)
87.6(+8.5)
77.6(+2.3)
MLP: (128, 64)
100.0C10.0)
90.0(+5.9)
94.4(+3.2)
72.4(14.5)
MLP: (256, 128)
83.6(+20.1)
90.8( :433)
95.2(11.66
78.4( 1:3.4)
G Hyper-parameters for controller learning
The hyper- pyper-parameters we used to learn the inverse dynamics controller for each task were the same as listed in Table. 6. Note that we use (, to denote (i))
15

---

Table 6: Hyper-parameteess controller learning
Hidden Layer
Activation Learning Rate
Iteration
(4, 41y, 2(,)
ReLU
0.0001
300
Table 7: One-step inference time (in milliscconds) with mean standard deviation over the task horizon.
Task
Policy
Tool
Door
Relocation
Reorieniation
NX
1.39
1604000000 39)
1.02(20.09)
1 12)
LSTM
1.710 0.228
1.32 0.33()))34)
1..577
142(10.133
NDP
188
LU80
1.2(+40211)
NGF
1.37==0110)
1,17( 26)
1.72(=0.36)
IX0=0 12)
KODXX
1.7((20.48)
L04(2:0:77)
1.12(20.24)
1.8(=0060)
H
Inference Time
report the inference time each method Table. 7. Our results indicate that KODex's infer- ence time is on par with the SOTA baselines. As such, it reveals that KODex can be translated to physical hardware and meet necessary control frequency.
I Zero-Shot Out-of-Distribution Generalization
We generated set of 10,000 out-of-distribution samples evaluate how the policies that were trained on 200 demonstrations generalize to unseen samples (see Appendix D for details on the sampling procedure). In Fig. 5, we report the task success rates of each method trained on the 200 demonstrations and tested on the 10,000 out-of-distribution samples. In addition, we also report the task success rate of the expert policy on the same 10,000 out-of-distribution samples establish a baseline. Perhaps unsurprisingly, none of the methods are able to consistently outperform the expert policy in most tasks. We observe that KODex able outperform the four baselines Tool Use task. In the other tasks, the highly-structured NGF performs the best, and KODex's 's performs comparably NDP and LSTM.
LSTM
NDP
NGF
KODex
Expert
Task Success Rate
100
Task Success Rate (%))
75
50
25
Tool
Door
Reloc
Reorient
Tasks
Figure 5: Zero-Shot Out-of-distributioo task success rates
Robustness changes in physical properties
We evaluate the robustness of the reference dynamics learned by each method to changes in hand mass or object mass for each task. This experiment is motivated by the fact that sim-to-real transfer often involves changes in physical properties. Further, consistent use of robotic hardware could result in changes to physical properties. Specifically, consider four variations per task:
16

---

Tool Use: i) Heavy Object (Hammer): 0.25 (default) + 0.85 (new), ii) Light Object (Hammer): 0.25 (default) - 0.10 (new), iii Light Hand (Palm): 4.0 (default) -> 1.0 (new), and iv) Heavy Hand (Palm): 4.0 (default) - 8.0 (new)
Door: i) Heavy Object (Latch): 3.54 (default) -+ 12.54 (new), ii) Light Object (Latch): 3.54 (default) - 0.54 (new), iii) Light Hand (Palm): 4.0 (default) -> 1.5 (new), and iv) Heavy Hand (Palm): 4.0 (default) - 7.0 (new)
Relocation: i) Heavy Object (Ball): 0.18 (default) + 1.88 (new), ii) Light Object (Ball): 0.18 (default) - 0.05 (new), iii) Light Hand (Palm): 4.0 (default) - 3.0 (new), and iv) Heavy Hand (Palm): 4.0 (default) + 5.0 (new);
Reorientation: i) Heavy Object (Pen): 1.5 (default) + 9.5 (new). ii) Light Object (Pen): 1.5 (default) - 0.2 (new), iii) Light Hand (Finger Knuckles): 0.008 (default) 0.0001 (new), and iv) Heavy Hand (Finger Knuckles): 0.008 (default) -> 0.20 (new)
It is important to note we held the reference dynamics learned by each method constant for this experiment, irrespective of the changes to the hand or the object. Instead, we relearned the tracking controller using 200 rollouts from the expert agent, following the procedure detailed in Section. 4.3.
In Tables. 8-11, we report the task success rate of KODex, and other baseline policies (all trained on 200 demonstrations) before and after relearning the controller. We also report the task success rates of the expert agents to establish baselines.
We find that the Light Hand variation results in the lowest drop in performance across all methods and all tasks, thus consequently relearnngg controllers does not offer any considerable improve- ments. In contrast, all methods benefit from relearning the controller in the Heavy Hand variations, evidenced by the increased task success rates. Overall, find that KODex outperforms all base- lines, with the exception of NGF which performs better than KODex under a few variations and tasks. Surprisingly, KODex (and some baselines) when used with the original controller outperform the expert policy under a few variations (e.g., Heavy Object in Relocation task, and Heavy Object in Door task). We believe this is due the fact that KODex and the baselines learn to generate and track desired trajectories separately, while the expert RL directly generates control inputs from state information. particular, the learned desired trajectories for a given tasks are likely invariant to slight changes in physical properties. On rare occasions where this is not the case, we indeed find that fine-tuning the tracking controllers worsens the performance.
These results demonstrate that changes to the robot/system dynamics can be handled by fine tuning the tracking controller without the need for relearning the reference dynamies. Once again, KODex is able to perform comparably to or outperform SOTA approaches despite its simplicity.
Table 8: Robustness to variations in the physical properties (Tool Use)
Success Rate
Variation
Heavy Object
Light Object
Light Hand
Heavy Hand
Controller
Expert agent
93.5
66.2
65.4
71.2
KODex Original controller
46.0
64.0
99.5
46.5
NN + Original controller
0.0(++0.0)
0.0(+
0.0(++000)
0.7(++1.4)
LSTM Original controller
32.7(++1.77)
35.0(+-22.1)
44.3(+23.1)
52.7(++2.55)
NDP + Original controller
0.0(++0.0)
68.0(+20.8)
45.4(=37.4)
0.0(+0.0)
NGF Original controller
33.4(+11.55)
62.9(-27.5)
83.2(+26.3)
40.3(+20.2)
KODex + Expert-tuned controller
53.5
44.0
89.0
92.5
NN Expert-tuned controller
0.0(+00.0)
0.0(+
0.2(+0.4)
0.0(+00.0
LSTM + Expert-tuned controller
42.4(+34.3)
33.7(+14.9)
52.2(+22.7)
69.9(+19.4)
NDP Expert-tuned controller
33.3(+20.00)
23.8(+24.4)
29.A(137.1)
39.8(+24.5)
NGF Expert-tuned controller
48.2(++1.00
48.7(+12.2)
94.6(-8.9)
82.1(+7.55
K
The impact of the choice of basis functions
We evaluate if KODex's performance is impacted by different sets of polynomial functions that are used as the lifting function. We trained all policies on 200 demos and tested them on 10,000 unseen initial conditions.
17

---

Table 9: Robustness variations in the physical properties (Door)
Success Rate
Variation
Heavy Object
Light Object
Light Hand
Heavy Hand
Controller
Expert agent
45.2
91.7
82.0
74.9
KODex Original controller
57.0
97.0
56.5
33.5
NN + Original controller
0.0(+0.0)
0.2(=00.4)
1.3(+2.1)
0.0(#0.0)
LSTM Original controller
34.4(+8.7)
75.8(+19.5)
38.1(=10.4)
33.5(#11.4)
NDP + Original controller
22.1(+1.9)
51.1(=4.9) 62.8(+5.2)
3.1(+2.3)
NGF Original controller
48.7(16.7)
95.0(42.1)
42.1(+11.0)
33.8(+10.0)
KODex + Expert-tunee controller
39.0
94.0
54.0
81.5
NN Expert-tuned controller
0.0(#0.00)
0.0(#0.0)
0.7(#0.9)
0.0(#0.0)
LSTM Expert-tuned controller
21.2(++5.3
75.4(+18.0)
49.2(+8.1)
56.9(+18.7)
NDP Expert-tuned controller
15.5(++3.0)
36.2(+
25.5(+4.4)
8.8(+3.0)
NGF Expert-tuned controller
36.6(+5.1)
95.5(+1.8)
57.7(#4.)
77.1(+6.7)
Table 10: Robustness to variations the physical properties (Relocation)
Success Rate
Variation
(%
Heavy Object
Light Object
Light Hand
Heavy Hand
Controller
Expert agent
77.0
100.0
100.0
100.0
KODex +
Original controller
19.5
89.5
82.5
21.5
NN Original controller
0.1(+:0.2)
1.6(#2.55
1.5(+2.1)
1.7(+2.2)
LSTM + Original controller
0.4(+10.4)
15.4(=10.7)
9.5(=8.1)
7.7(49.4)
NDP Original controller
13.5(4.5.0)
85.6(18.1)
72.1(19.))
31.6(+10.0)
NGF + Original controller
25.8(++4.)
96.4(+1.4)
96.6(+0.97)
19.3(+3.8)
KODex Expert-tuned controller
34.0
93.0
85.0
89.0
NN + Expert-tuned controller
0.2(+00.4
0.6(10.7)
1.4(1.8))
1.5(+2.3)
LSTM Expert-tuned controller
5.8(+4.7)
15.2(=12.5)
15.5(=10.7)
14.1(#9.3)
NDP Expert-tuned controller
19.9(+5.8)
84.5(48.9)
63.2(=15.0)
92.4(+1.2)
NGF Expert-tuned controller
52.6(13.6)
98.1
95.6(=2.2)
94.5(+00.9)
Table 11: Robustness to variations in the physical propertics (Reorientation)
Success Rate
Variation
(%)
Heavy Object
Light Object
Light Hand
Heavy Hand
Controller
Expert agent
46.8
69.0
95.2
89.7
KODex Original controller
53.5
55.0
66.5
61.5
NN + Original controller
4.7(1:2.6)
9.6(#8..1)
9.5(16.4)
7.9(=6.5)
LSTM + Original controller
34.3(+7.88
52.3(+10.6)
60.3(16.0)
55.6(+1.8)
NDP + Original controller
49.4(=3.6)
58.4(-1..4)
59.8(47.6)
55.7(19.7)
NGF + Original controller
39.9(=1.9)
57.1(+2.2)
81.6(#1.8)
73.4(+3.8)
KODex Expert-tuned controller
52.0
63.0
71.5
65.5
NN Expert-tuned controller
1.5(+...)
5.2(+4.2)
3.8(+1.7)
3.7(=2.6)
LSTM + Expert-tuned controller
43.5(17.9)
47.7(18.8)
61.4(14.2)
54.4(+..55
NDP + Expert-tuned controller
55.5(15.9)
59.0(15.5)
63.0(
57.0(
NGF + Expert-tuned controller
49.1(=2.6)
59.7(13.2)
79.4(11.9)
72.6(+1.22)
Design: Specifically, we define four sets of observables (one of which was used in the original and x. [=g. 22..... 2n" denote the robot
submission). Let robot state: X, = x2. , .x7) and the object state, respectively, with superseript indexing the states. We then define four vector- valued lifting functions v), and v. in (8) follows
Set 1
v, = [(x))) for 1......
v. (( for -.,...,,m
18

---

Set 2
4r = [ for i,j 1,...,
V/o = (m) for fori,j = 1,... m
Set 3 (used in this work)
1/- [,,,,] U ((x)))) for for = ,,...,,
4. = u (<x))) for i,j 1.. m
Set 4
3, = [, ((x)) for , v. { u (xx)) i.j = -
report the number of observables for each set and task combination in Table. 12.
Table 12: Number of observables
Task
Tool
Door
Relocation Reorientation
Set
n=26,m=15
m==
n=30,m=12
n=24,m=12
Set 1
82
70
84
72
Set 2
512
469
585
414
Set 3 (ours)
763
546
759
582
Sct 4
1413
1302
1629
1134
Relocation
Tool
Door
Reorientation
Truining Time Set of Lifting Function
Imitation Error Set of Lifting Function
Task Success Rate Set of Lifting Function
Triining Time (Sec)
200
0.100
Task Sucosss Rate (%)
75
150
0.075
50
100
Imitation
0.050
50
0.025
0.000
Set Liftnng Function
Lifting Function
of Litting Function
Figure 6: The effects of lifting function on training time (left), imitation error (center). and success rate (right).
Discussion: As shown in Fig.
clear that training time increases with the number of observ- ables since the Moore-Penrose inverse requires more computation for higher-dimension matrices. Importantly, KODex's success rate across all tasks remained roughly the same for Sets 2, 3, and 4. In general, as one would expect, increasing the number of observables tends to decrease imitation error and increase task success rate, The only exception to this trend is observed for the Object Relocation task, in which KODex performs marginally better when trained on Set 2 (585 observ- ables) compared with trained on Set 3 (759 observables). Taken together, these results suggest that KODex's 's performance is not highly sensitive to the specific choice of lifting function, as long as sufficient expressivity is ensured.
L Stability Analysis
Another unique advantage of utilizing Koopman Operators to model the reference dynamics for dexterous manipulation tasks that the learned policy is a linear dynamical system which can be readily inspected and analyzed, in stark contrast to SOTA methods built upon deep neural networks.
We analyzed the stability of the learned policy. For a linear dynamical system with complex conju- gate eigenvalues A; 0; + jws, i.e., KODex with Koopman matrix K, the system is asymptotically stable if all of the eigenvalues have magnitude (p.
0 + w/ less than one. From the standpoint
19

---

of control theory, it is beneficial to have asymptotically stable system because of the guarantee that all system states will converge. However, from the standpoint dexterous manipulation tasks considered in this work, strict stability might not be preferable because the final desired hand poses and object poses are identical for different initial conditions. This represents natural trade-off between safety and expressivity. As such, understanding how KODex addresses this trade-off can be illuminating.
Occurencce of Eigenvalue Magnitude
Occurrence of Eigenvalue Magnitude
100
Occurrence
75
Occurrence
40
50
20
25-
0
0.0
0.5
0.0
0.5
1.0
Eigenvalue Magnitude
Eigenvalue Magnitude
(a) Tool Use
(b) Door Opening
Occurrncce of Eigenvalue Magnitude
Occurrence of Eigenvalue Magnitude
150
60
Occurrencc
Occurrence
40
50
20
0
0.0
1.0
0.0
0.5
1.0
Eigenvalue Magnitude
Eigenvalue Magnitude
(c) Object Relocation
(d) In-hand Reorientation
Figure 7: Occurrence of Eigenvalue Magnitude
Table 13: Maximun Eigenvalue Magnitude
Tool Use Door Opening Object Relocation In-hand Reorientation
1.07888
1.00553
1.00859
1.00413
In Fig. 7, report histogram of the Koopman matrix's 's eigenvalue magnitudes in each task. In addition, we report the maximum eigenvalue magnitude in Table. 13. Based on these results, we can that most eigenvalues' magnitudes are less than one, suggesting that KODex tends learn nearly-stable policies that generate safe trajectories during execution, and ii) few eigenvalues have magnitude larger than one, suggesting KODex does not prioritize stability, the expense of expressivity required to achieve the reported performance.
M Comparisons against Behaviour Cloning with State-Action Mapping
We conducted an additional experiment involving new neural network based baseline policy that learns directly map states x(t) to actions T(t) instead of learning the reference dynamics and the tracking controller. The new policy (State-action BC) was built upon the MLP architecture with three hidden layers (164, 128, 64],, and was trained over three random seeds to minimize the state- action reproduction enror. For fair comparison, these policies were trained and tested on the same set of demonstrations and testing instances in Section. 5.3.
In Fig. 8, report the training time and the task success rate each task for KODex and State-
20

---

Thininn
Thinnng Datasct
Thiinng Datesci
Thinnng Dattact
Traning Time (5x2
4003
Traiiing Time (Se)
Traning (8e2
Tomtnn Time (Sc)
500
200
300
300
300
100
100
100
100
Nanther
Taot Strcens Rato vs Derastt Niee
Tack Succons Rate Docser Nize
Tast Surcems Rate vn Dovsutt Niro
Text Strons Kite Datver Sire
100
Teak Sacors Duac (5)
100
Rone (50
Taak Sactes Rlacc
Sacress Rat:
75
25
25
25 50 100
1.50
200
25
50
190
1.4
200
25
50
1000
150
200
25
50
100
1.50
200
Nunther Demumetttion stlore
Demunstratioos
Numiher
Demomsttaion
Numberr Dermonstratiinn
(c) Object Relocation (d) In-hand Reorientation Figure& 8: The effects of number of demonstrations training time (top row), and success rate (bottom row) for KODex and State-action BC each task. Solid lines indicate trends and shaded areas show + standard deviation over three random seeds.
(a) Tool Use (Hammer)
(b) Door Opening
Pen Throw-out Rate vs Dataset Size
10
State acton BC
Pen Throw-out Rate
25
50
100
150
200
Number of Demonstrations
Figure 9: The effects of number of demonstrations pen throw-out rate for KODex and State-action BC.
Action BC. The results reveal familiar trend: across all tasks, KODex is drastically more compu- tationally efficient than State-Action BC, while performing comparably (if not better) in terms of success rate.
Further, we would like to highlight two other advantages of KODex over State-Action BC. First, KODex could be potentially applied on state-only demonstrations, with a manually-tunae PD con- troller replacing the learned tracking controller (one could also learn the controller via reinforcement leamning [39]). In contrast, state-action imitation learning methods inevitably need action labels. Second, KODex is safer for online execution. Since KODex separates motion generation and track- ing, it tends to take less risky actions. But state-action policies may take unsafe actions when they encounter unseen states due to covariate shift. In Fig. 9, we report the pen throw-out rate from the In-hand Reorientation task. can be seen that the State-action BC policy is more likely to generate undesirable behaviours, resulting in complete task failures. This implies that KODex may be safer for hardware implementations, thanks to the separation reference motion and tracking. Although there few other state-actic action policies that better address covariate shift (e.g., GAIL [45]), such comparisons are outside of scope of this work.
21


Reviews:
[{"review_id": "_sdCGzgUjzS", "reviewer": "Reviewer_aS5H", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a method for efficiently learning high degree-of-freedom manipulation tasks by leveraging Koopman operators. Using expert demonstrations, they learn a high-dimensional linear model that can be used to generated reference trajectories that complete the task. This is combined with a learned tracking controller that tracks the generated reference trajectories. The proposed method is evaluated on four complex dexterous manipulation tasks and compared to four baseline reinforcement learning methods. The method performs comparably with state-of-the-art approaches while taking dramatically less time to train. ", "summary_of_recommendation": "I believe this paper should be accepted to CoRL. The results are compelling, the paper is well-written, and the method offers a different perspective than most of the papers typically seen at this conference. I believe the community will benefit from seeing the results of this paper, which offers a new avenue to explore as we seek to overcome the limitations of traditional neural-network and deep learning methods. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "# Primary Concerns\n1. Is the method actually learning the underlying dynamics of the system? My principal concern is about the use of \"open-loop\" reference trajectory generation. After learning the Koopman matrix from expert demonstrations, the reference trajectory is generated by simulating the linear system forward one time step. The assumption here is that the \"learned dynamics\" somehow encode the task. The true underlying dynamics (based on physics) should be task-agnostic, so the authors aren't truly trying to learning the underlying dynamics of the system, but rather the *dynamics of the task.* I think this is an important distinction that should be made. I'm not saying this is a bad idea, just that the authors should be more careful in making this distinction. How could this method be used to actually the learn underlying system dynamics and improve task generalization?\n2. Why did the authors not compare against classical model-based methods? The authors state several times that one of the advantages of Koopman-based methods it that it allows the use of linear-systems and control-theoretic approaches, but other than a stability analysis included in the Appendix, this isn't really leveraged. \n3. Did the authors try to use an LQR tracking controller instead of a learned one? \n4. In Eq 8, you don't include any non-linear coupling of the robot and the object. Why?\n5. Why did the authors choose polynomials? The types of polynomials chosen by the authors are usually poorly conditioned. Better choices include options like Chebyshev polynomials. Also, manipulation tasks include friction and contact, which are both non-smooth, so why not include non-smooth functions like ReLU? \n\n# Secondary Questions\n6. In the third paragraph of 4.1, the authors state the system dynamics are \"arbitrarily complex.\" What do you mean by this? What is arbitrarily complex? We have well-understood models of rigid-body dynamics, and decent models of friction and contact (agreed these, along with deformation, can get fairly complex). Where do the authors see the arbitrary amounts of complexity? Mostly curious here, and I'd like to see a sentence or two describing why the dynamics are complex. \n7. Mostly out of idle curiosity, what sparse linear system solver are the authors using to solve for K in eq (6)? At least include this as a footnote or in the Appendix. \n8. The authors leverage MuJoCo, which approximates contact dynamics by \"softening\" them. How \"soft\" did the authors make the dynamics in their training? How could this impact the ability of the proposed method to move onto real hardware. Do the authors believe their method will be less sensitive to this sim-to-real gap than other learning-based methods? Why?\n9. In Section 5.1, the authors state the controller ran at 500 Hz. This is pretty fast. What happens if you slow down the controller? Can you make use of lower-level controllers (like joint-space PID or LQR) to make it easier and run the controller at a lower rate?\n\n\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, a solid paper with meaningful contributions to the field of robotics and manipulation. The paper is well written, with a solid background and introduction, a clear and concise description of the method, and meaningful examples with easy-to-interpret results. \n\nThe results of the paper are noteworthy: they achieve performance on par with recent state-of-the-art methods with a method that takes significantly less training time. While there are many areas for improvement and future work, the community will benefit from the results of this paper, which clearly show that learning methods outside the realm of neural networks and deep reinforcement learning have exceptional merit and are worth exploring. \n\nStrengths:\n* Clearly written\n* Impressive results\n* Use of a lesser-known learning technique\n* Clear and honest evaluation of current learning-based methods for manipulation\n* Consideration and evaluation of training time as an important metric\n\nWeaknesses:\n* Limitations and future work section is lacking\n* Description of the overall training pipeline is somewhat unclear\n* Confusion over what dynamics are being learned by the method\n* A few areas where the theory is glossed over or not expressed correctly\n* Too many discussion points are deferred to Appendices\n* Lack of comparison with non-learning approaches\n* Lack of demonstration of physical hardware (minor, given it's noted in the limitations section and left for future work)", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "JrFm0KdBqK", "reviewer": "Reviewer_RjjK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors investigate the utility of Koopman operator theory in alleviating the limitations of existing learning-based approaches, with a particular focus on general efficacy, computational efficiency, sample efficiency, and scalability. They develop a novel imitation learning framework, dubbed Koopman Operator-based Dexterous Manipulation (KODex), to evaluate the utility of Koopman operator theory for dexterous manipulation. Their experimental evaluation is carried out on four dexterous manipulation skills on the simulated Adroit hand: Tool Use, Door Opening, Relocation, and Reorientation. They benchmarked their method against NN and LSTM (unstructured baselines) and NDP and NGF (highly-structured baselines). They conclude that their KODex can i) analytically learn dexterous manipulation skills and ii) match or outperform SOTA imitation learning approaches on various dexterous manipulation tasks, while being an order of magnitude faster.\n", "summary_of_recommendation": "I would recommend this paper to be accepted, following some further thoughts/maybe preliminary experiments on using robot hardware and addressing the generalisation of the methods across different tasks (e.g. training on a task, testing on another similar task). ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "(1) \"First, implementing existing algorithms requires significant technical expertise and modern machine learning infrastructure.\" -> could you elaborate more on \"modern machine learning infrastructure\"? This seems a bit vague.\n\n(2) \"Third, existing approaches demand painstaking effort in tuning hyperparameters to achieve acceptable performance\". What do you really mean by \"acceptable performance\" as there have been methods that have achieve SOTA performance and not only acceptable? This sentence seems to downplay the strengths of learning-based methods.\n\n(3) In Section 4.3: \"a PID controller could be used instead of learning a tracking controller.\"  - Why do the authors recommend a PID and not a PD controller as is common in manipulation tasks? \n\n(4) Following from (3), a major challenge in learning controllers is the inability to guarantee easily stability. Did the authors think about this for their methods and how do they ensure their controller remains stable?\n\n(5) Also in Section 4.3: \"one would have to painstakingly tune the control gains and frequency, and do so for each task independently\" - Would the authors not argue it's the same for learning-based methods with the hyperparameter optimisation?\n\n(6) The authors use training time (time taken to train a policy) as an evaluation metric - why do they use only the training time and not inference time?\n\n(7) I would like to know the authors' thoughts on the choice of the 4 tasks - why these particular 4 (are they challenging their method in different aspects?). Also, did they consider how their method would generalise from task to task (e.g. learn in the easiest task and then transfer the policy to harder tasks?)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality: The overall quality of the work is good as it addresses an important challenge in using learning-based approaches, by adopting theories from control such as the Koopman operator theory. \n\nClarity: The aims, methods and experiments are clear, while the appendix is also very detailed.\n\nOriginality: The originality of the work is good, although there has been several efforts towards combining control theories with learning-based methods for improving sample efficiency and scalability.\n\nSignificance of the work: Overall, the significance could be further improved if the experiments were deployed and the choice of task environments further motivated.\n\nMain Strengths: The main strengths of the paper are the clarity and logical way the paper is presented, the fact that they clearly outline what KODex aims to achieve in comparison to other related works and the experimental evaluation is very comprehensive.\n\nWeaknesses: The main weaknesses of the paper are the fact that the experiments were carried out in simulation and I am not sure these would be easily deployed in the real world (although this is addressed in the limitations section).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "A3Y1Wby6ePV", "reviewer": "Reviewer_k76a", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a technique for modeling the dynamics of dexterous manipulation problems as linear dynamics in a lifted high dimensional space, using koopman operator theory. They then show the ability to learn this koopman operator from data as a means to represent linear dynamics in this lifted space. They then use this learned linear dynamics to generate reference trajectories and learn an inverse dynamics model with an MLP to track the generated reference trajectories. \n", "summary_of_recommendation": "Overall i like the simplicity and effectiveness of the approach, but I have suggested some additional empiricism that may help strengthen the paper. If these points are addressed, I'm willing to raise my score. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Repeating weaknesses here: \n1. More comparisons would be helpful, especially to techniques like guided policy search or differentiable MPC. \n2. A deeper understanding of failure modes would be helpful in understanding the tradeoffs of using Koopman versus other policy classes for imitation. \n3. An analysis across some more domains (eg PDDM) would help add to the paper. \n4. A comparison with standard imitation methods like IRL/BC would also be helpful. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall the idea of using koopman operators for modeling linear dynamics is reasonable, but is not new in MBRL, as the many citations in this paper also represent. I think it’s still worthwhile to understand how well this can work in the context of dexterous manipulation though. \n\nThe part I find most confusing about this paper in the way it is written right now is - where is the Koopman operator K actually used? If it’s simply an imitation learning framework, why do we need the Koopman operator at all? There seems to be a missing section on how the K is used to generate the reference trajectory - via LQR or something else? If it’s imitation learning and the trajectories are given to you then why not just directly track those? And if it’s about generating reference trajectories in new initial states, then is it just rolling out the Koopman operator forward since the actions are implicit in the dynamics? I suspect it is this, but this should be made very clear in the paper. \n\nThe basic premise of this work seems to be the fact that learning Koopman operators give you an easy way to roll out the trajectories in new scenarios, and these can be tracked with inverse dynamics. I think this should be compared with standard BC / IRL type methods on this task, which may work quite well. \n\nThe claim is made that the other techniques need things like gradient descent etc, which is true for NN policies, but there are other structured policies that could be used like differentiable MPC - or like guided policy search which can be extremely data efficient. Those might be more reasonable baselines to compare to. \n\nAlso the choice of \\phi seems very important or at least I’d expect it to be. Can the authors provide intuition on why this doesn’t matter?\n\n“While counter-intuitive, this 241 observation follows the recent finding that imitation error and task success rate might not necessarily 242 be correlated” —> can more be said about this?\n\nI think the demonstration is reasonable - but given the somewhat limited novelty in directly learning koopman dynamics and using it for imitation learning, a deeper empirical analysis - more comparisons, more domains (eg from PDDM (nagabandi et al)), and understanding the failure modes more clearly would be very helpful. \n\n\nStrengths: \n1. Simple method, seems like it has strong empirical performance. \n2. Big improvement in sample efficiency and in robustness which is powerful for imitation learning settings. \n\nWeaknesses: \n1. More comparisons would be helpful, especially to techniques like guided policy search or differentiable MPC. \n2. A deeper understanding of failure modes would be helpful in understanding the tradeoffs of using Koopman versus other policy classes for imitation. \n3. An analysis across some more domains (eg PDDM) would help add to the paper. \n4. A comparison with standard imitation methods like IRL/BC would also be helpful. \n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for learning manipulation tasks using Koopman operators and leveraging expert demonstrations.

Reviewers appreciated the clarity of presentation, including Koopman operator theory, results showing competitive performance with lower training time,

Reviewers raised concerns about the limitations and future work sections, missing descriptions of the overall pipeline, missing details on learned dynamics, needing more discussion relative to RL/BC/IRL, and lack of real-world demonstration.

The reviewers appreciated the improvements made in the rebuttal but did not raise their assessment further. The paper remains an accept.
Decision: Accept (Oral)