KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipulation
Authors: Hongyi Chen
ABULIKEMU ABUDUWEILI
Aviral Agrawal
Yunhai Han
Harish Ravichandar
Changliu Liu
Jeffrey Ichnowski
Venue: CoRL 2024 (2024)
Keywords: Manipulation
Koopman Operator
Visual Representation Learning

Abstract: Learning dexterous manipulation skills presents significant challenges due to complex nonlinear dynamics that underlie the interactions between objects and multi-fingered hands. Koopman operators have emerged as a robust method for modeling such nonlinear dynamics within a linear framework.
However, current methods rely on runtime access to ground-truth (GT) object states, making them unsuitable for vision-based practical applications.
Unlike image-to-action policies that implicitly learn visual features for control, we use a dynamics model, specifically the Koopman operator, to learn visually interpretable object features critical for robotic manipulation within a scene.
We construct a Koopman operator using object features predicted by a feature extractor and utilize it to auto-regressively advance system states. We train the feature extractor to embed scene information into object features, thereby enabling the accurate propagation of robot trajectories.
We evaluate our approach on simulated and real-world robot tasks, with results showing that it outperformed the model-based imitation learning NDP by 1.08$\times$ and the image-to-action Diffusion Policy by 1.16$\times$. The results suggest that our method maintains task success rates with learned features and extends applicability to real-world manipulation without GT object states. Project video and code are available at: https://github.com/hychen-naza/KOROL.

KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipulation
Hongyi Chen', Abulikemu Abuduweili', Aviral Agrawall, Yunhai Han?*,
Harish Ravichandar Changliu Liu', Jeffrey Ichnowski! CCrmegie Mellon University, 2 Georgia Institute of Technology
Learning dexterous manipulation skills presents significant challenges due to complex nonlinear dynamics that underlie the interactions between ob- jects and multi-fingered hands. Koopman operators have emerged as a robust method for modeling such nonlinear dynamics within lincar framework. How- ever, current methods rely on runtime access to ground-truit truth (GT) object states, making them unsuitable for vision-based practical applications. Unlike image- dy-
A bstract:
to-action policies that implicitly learn visual features for control, namics model, specifically the Koopman operator, to learn visually interpretable object features critical for robotic manipulation within a scene. We construct a Koopman operator using object features predicted by feature extractor and utilize to auto-regressively advance system states. We train the feature ex- tractor to embed scene information into object features, thereby enabling the accurate propagation of robot trajectories. We evaluate our approach on simu- lated and real-world robot tasks, with results showing that it outperformed the model-based imitation learning NDP by 1.08x and the image-to-action Dif- fusion Policy by 1.16x. The results suggest that our method maintains task success rates with learned features and extends applicability to real-world ma- nipulation without GT object states. Project video and code are available at: https:/ /tte::///ittiee httgthu........g/ cow/hychen- naza/KOROL.
Keywords: Manipulation, Koopman Operator, Visual Representation Learning
Introduction
Humans possess an extraordinary ability to manip- ulate objects, discerning position, shape, and other properties with just a glance. How can robots be endowed with similar perceptual and dexterous manipulation capabilities? Traditional control and optimization approaches typically require detailed models of the system dynamics [1, 2]. However, these models can be difficult to derive and often lack the flexibility and generalizability needed to adapt to task or environment changes. End-to-end data-driven methods overcome these challenges by learning actions directly from observations [3, 4, 5]. While these methods can make minimal assump- tions, they often require large number of demon- strations to master basic skills due to the high di- mensionality of the inputs.
pestioo
Vanilla Koopman Operator
KOROL (Ours)
Figure 1: left: Vanilla Koopman operators rely on ground-truth state which may be difficult to obtain in real-world settings. right: In contrast, we pro- pose KOROL, which learns dynamics model and task-relevant object features without labels ject states. The visualization shows the localization of learned feature around the door handle.
To combine sample efficiency of traditional model-based approaches with high generalizability of deep learning methods, one branch of recent work has focused on learning dynamics models to plan
Conference on Robot Learning (CoRL. 2024), Munich, Germany.
Denotes equal contribution

---

trajectories. These methods embed learning into various models, such as Koopman operator [6, 71. Dynamic Movement Primitives (DMP) [8], Neural Geometric Fabrics [9, 10], and more [11], showcasing good performance in simulations. However, they often falter real-world applications due to their reliance on hard-to-obtain ground-truth (GT) state information like object poses and contact points. Moreover, the problems of using computer vision to estimate these object states the uncertainty in determining the number of objects consider and which specific states to estimate. Additionally, the learned dynamics models do not transfer across different tasks without universal state space design.
We propose an approach to remove the dependency on GT states in model-based manipulation learn- ing (Figure 1). Central to this approach, Koopman Operator Rollout for Object Feature Learning (KOROL), is learning visual features that predict robot states during dynamics model rollouts. Un- like learning methods that learn implicit visual features for image-to-actinn policies, KOROL ex- plicitly trains on visual object features, encoding essential scene information to enhance predictions of robot states during autoregressive model rollouts. Central to this rollout process is the Koopman operator, which utilizes current object features to advance robot states. This establishes a synergistic relationship between the learned object features and the Koopman operator. KOROL uses trained features to refine the Koopman operator, which in turn improves the feature learning process through a more accurate dynamies model.
In experiments, we demonstrate how KOROL outperforms prior methods in ADROIT Hand [12] and generates interpretable visualizations by learning object features from images. We also show how KOROL, enables the application of Koopman operator in vision-based real-world manipulation tasks. Finally, we demonstrate how learning object feature instead of designing object states for dynamics modeling enables KOROL to construct universal Koopman dynamics across multiple manipulation tasks. This paper makes the following contributions:
We introduce KOROL, an imitation learning method, that uses the Koopman operator to learn object features and show that the Koopman operator with learned object features can outperform the one with GT object states.
We extend the application of the Koopman operator to uision-based manipulation tasks in real- world by learning object features from images and demonstrate its effectiveness through com- parisons with prior methods.
We demonstrate that KOROL learns dimensionally-aliinn object features across tasks, enabling the development of multi-tasking Koopman operator.
2 Related Work
Imitation Learning and Visual Representations for Manipulation. Imitation learning serves as a primary method for teaching robots to manipulate objects by mapping observations or world states directly actions. Common approaches include Behavioral Cloning (BC) [13], Implicit Behavioral Cloning (IBC) [14], Long Short-Term Memory (LSTM) networks [3, 15]. Transformers [4, 5], and Diffusion Models [16]. A significant challenge in imitation learning is representing the visual in- formation of scene. Strategies include using pre-trained 2D [17] or 3D backbones [5] to output visual embeddings. Other works propose end-to-end learning approaches that simultaneously train the visual encoder and the learning policy [4, 16]. Other techniques focus on learning visual rep- resentations through correspondence models [15], self-supervised novel view reconstruction using Neural Radiance Fields (NeRF) [18, 19], and Gaussian Splatting [20].
Model-Based Learning and Planning. In robotics, traditional model-based approaches rely expert knowledge of physics to design system models [21, 22, 2]. Since traditional methods can miss complex nonlincarities, and end-to-end learning approaches can be data-intensive, middle ground of data-driven model learning shows promise as data-efficient way to derive complex models [6, 23]. Model learning includes a variety of dynamics models such as Koopman oper- ators [24, 25], Deep Neural Koopman operators [7], Dynamic Movement Primitives [8], Neural Geometric Fabrics [9, 10], and others [1 26]. Additionally, some studies focus on learning environ- mental responses actions plan future trajectory [27, 28, 29], integrate planning in generative modeling process [30, 16], and seamlessly blend the learning of models with planning [31, 32].

---

Koopman Operator Theory. In the early 1930s, Koopman and Von Neumann introduced the Koopman operator theory to transform complex, nonlinear dynamics systems into linear ones in an ceinite-dimensional vector space, using observables as lifted states [33, 34]. This transform allows the application of linear system tools for effective prediction, estimation, and control with hand- designed observables [6. 35, 36, 37]. Recent methods using neural networks learn observables have proven more expressive and effective, particularly in chaotic time-series prediction [7, 38, 39]. Furthermore, the integration of neural network-derived Koopman observables with Model Predictive Control has shown promise in enhancing control tasks [40, 41J. As significant benchmark, Han et al. [6] demonstrate the effectiivnness of Koopman operators in manipulation tasks using GT object states. Building on this foundation, we extend the application the Koopman operator to vision- based manipulation tasks in real-world settings by learning object features directly from images
3 Background: Koopman Operator Theory
In this section, we provide a brief background on the Koopman Operator Theory. Consider the evolution of nonlinear dynamics system x(t + 1) F(x(t)). Given the original state space X, the Koopman Operator K. introduces lifted space of observables O using lifting function g X O, to transform the nonlinear dynamics system into a linear system in infinitediimensional observables space as g(x(t + 1)) = Kg(x(t)).
In practice, we approximate the Koopman operator by restricting observables to be a finite- dimensional vector space. Let ((x(t)) R represent finite dimensional approximation of ob- servables (xx())) and matrix K E RRxx approximate the Koopman operator K.. Thus, we rewrite the relationship
oxx(1 + 1)) Ko(((().
Given dataset D, in which each trajectory can learn K by minimizing the state prediction error [38]
... x(T)) containing T time steps,
T=
J(K) = Σ Σ XED
||x(x(( 1)) K)(x(1))
In manipulation tasks, we define the state x(t) [t) xxx JTT to include the robot state x,, (t) and object state (t), as we care about how objects move as a result of robot's motion. Moreover, since our goal to minimize the imitation error of the robot state x, (L), we design observables o( (x(()) that include lifted robot and object states as
((x(t)) = (t) T &), (x), where e,, : R" - R" and vo RT + RT are vector-valued lifting functions that transform the robot and object state respectively. We can thus retrieve the desired robot state by selecting the corresponding elements in ((x(t)). Let o denote the unlifting function to reconstruct the robot state from observables, xy) = x- dbx((t)) (we can also reconstruct the object state (!) in the same way). Considering the lifting function Equation 3, the unlifting function can be represented as
the (x,( Vt,
X, (t) = o (()(t)) = [I Oxx (nrtmmmm))) ((x(t)),
(4)
where Inxn and 0nxn denote an identity matrix and zero matrix respectively. To streamline notation throughout this paper, we define x, (t + 1) = K'(x, (t),, where K' == ~- 1 o K d d.
4 Method
We propose KOROL, which formulates dynamics learning and object feature learning as an imitation (supervised) learning problem robot states. Given a dataset D, in which each trajectory x,, y(T)] containing robot states x, (t) and image observation y(t) of the object, instead of object state xo(t), our goal is to learn visual object feature extractor fa and Koopman operator K which can predict object features from images that minimize the robot
[xx, (1),
states imitation errors. In this formulation, (2) becomes
arg min 0.K
Σ XED
Ix. (L + 1) - K' ((xx(())
(5)
3

---

Training
toast Tpl
Every batch
Update
Every M epoches
RGBD Image
Koopman
(t(t)
Operator
K
Koopman
Feature Extractor
Operator
Rollout
transform |
Execution
Feature Extractor
Koopman Operater Rollout
Inverse Dynamies Controller
Actions
Open
Eny
Image Frequency Domain
Figure 2: Training and Execution Pipeline. During training. KOROL updates the feature extractor fo based on the loss between the predicted robot trajectory f, = operator rollouts and the ground-truth robot trajectory т, = (2)) , (T)]. KOROL updates the Koopman operator with the new object features (t) every M epochs enhance the training of fo. During execution, KOROL feeds the generated trajectory the inverse dynamics controller produce the actions.
(2), ,T)) obtained through Koopman
Learning object feature. While traditional Koopman operator construction requires GT object state information (t) (section 3), instead, we adopt a neural network fo for object feature encoding and extraction from RGBD images. We initialize fe and predict object features x,)() for all images in D to construct the initial Koopman operator K, proposed by Han et al. [6]. During training, randomly sample the beginning time step to in trajectory т, and select (to) fo (y(oo)). Then can advance the observables forward using K)(.,)). We train fo by minimizing the loss function
L =
Σ | + 1) K' (&+ + 1), 8, + +)))
(6)
where N is the prediction horizon and & (0) (0) indicates that the system provides the initial GT robot state. See Figure 2 for visualization. Integrating spatial domain RGBD images with their frequency domain counterparts has been shown to enhance image classification performance by centuating discriminative features [42, 43]. Therefore, apply the Discrete Cosine Transform [44] to convert RGBD images into the frequency domain. We then concatenate the spatial and frequency domain images as input, enabling fe to detect changes in successive, cighly-correlated images more effectively than using spatial images alone. Subsequently, KOROL generates the reference trajec- tory ([&, (t) by rolling out the dynamics K. We feed these trajectories into the pre-trained inverse dynamic controller [6), which computes the required action a(t) using &, and x, + 1).
Updating Koopman Operator K. Subsequent updates to the Koopman operator K are neces- sitated by changes in the predicted object features (t), because K optimized for these specific robot states and object features. See pseudocode in Alg. for details. During the training of fo from line 7 to line 15, the dynamics K initially computed at line may no longer be optimal for the new object features, prompting need for recalculation. However, recalculating the object features across the entire training dataset and updating K for every fa modification is computationally in- tensive. Therefore, in KOROL, we defer the updates and recalculate K every M epoches to balance accuracy with computational efficiency, as detailed from line 16 to line 20.
Multi-tasking Koopman Operator. While a robot's 's state space remains consistent when using the same robot platform, object state spaces typically vary across different tasks. For example, prior Koopman manipulation study [6] includes a 15-DoF tool use task and a 7-DoF door opening task. Due to the differences object state space definition and dimensions, Koopman operators trained for different tasks not be shared, limiting their scalability. In KOROL, propose training object features (t) to serve universal interface for representing length-varied object states

---

Algorithm Object Feature Learning and Koopman Operator Updating
1: Require training dataset D with robot states x, and images y, feature extractor fo, function for calculating Koopman operator fune(-)
2: R, <- fo(y) for y D Predict object features across dataset
3: K < Func(xr; 8o) // Calculate the initial dynamics K
4: for epoch ,...,,N,, do
5: t, ~ // Sample the trajectory and beginning time steps
6:
x-(fo), + T(o)
7:
loss +-
8:
for i i- -....., N.....No. do
9:
if - then
10:
(to) x, (to): (to))) fo(u(to)) // Predict object feature with feature extractor
11:
end
12:
(to + loss + loss+ x, (ta + i 1) x, (to + 1)ll Calculate and sum the loss using Equation 6
K (to i), (No + 6)) // Predict the next states with K
13:
14:
end for
15:
Update the feature extractor fo to minimize loss
16:
epoch % M = then
17:
No t foy) for in D
18:
K func(xe; & // Update the Koopman operator
19:
end if
20:
end for
across tasks, enabling the generalization of K to multiple tasks. Moreover, these object features act as latent conditional vectors that differentiate among tasks. Thus, as long as the feature extractor can identify useful object features, it is possible to use datasets from various tasks train a single multi-task Koopman operator Kmuutii
Experiments
In this section, we evaluate the performance of KOROL along with existing unstructured learning and model-based learning approaches in simulation and real-world tasks.
5.1 ADROIT Hand Simulation Experiment
Setup and Baselines. We conducted our simulation experiments on the ADROIT Hand [12] 30-DoF simulated system (24-DoF articulated hand 6-DoF floating wrist base). There are simu- lation tasks: Door opening, Tool use, Object Relocation, and In-hand Reorientation. We compared KOROL to the baselines: (1) Behavior Cloning (BC): Unstructured fully-comnede neural network policy; (2) Neural Dynamic policy (NDP): Neural network policy with embedded structure of dy- namics systems [8]; (3) Diffusion Policy: Learning policy using probabilistic generative model [16]. To allow equal comparison, all models use ResNet18 [45] as feature extractor. Appendix provides details about task state space design and baselines implementation.
Door opening
Tool use
Relocation
Reorientation
Model
10
200
10
200
10
200
10
200
BC w GT
0%
96.1% 1%
0%
49.5%
0%
48.1%
19.4%
67.8%
NDP w GT
5.2%
99.9%
30.2%
96.9%
1.9%
99.8%
21.6%
64.6%
Diffusion Policy w GT
97.5%
100%
99.4%
100%
59.6%
99.2%
83.8%
93.3%
Koopman Operator w GT
99.6%
100%
100%
100%
77.0%
95.6%
7.6%
83.6%
BC
0%
0%
0%
0%
0%
0%
0%
NDP
99.3%
0%
96.2%
0%
92.7%
25.3%
67.7%
Diffusion Policy
93.2%
99.9%
97.8%
99.7%
86.4%
100%
31.5%
33.0%
KOROL
98.6%
99.9%
94.3%
100 %
99.8%
100%
55.6%
86.4%
across 5 random seeds for all models, trained with either 10 and 200 demonstrations per task. We evaluated each model on 200 unseon cases per task. The upper half of the table displays results for models using GT object states, while the lower half displays results from models employing features extractor ResNet18.
Table 1:
Quantitative Performance in ADROIT Hand. The averaged task success
Numerical Results of KOROL and Baselines. From the Table 1, draw two conclusions.
5

---

RGBD
Step 0
Step 15
Step 30
Step 45
Figure 3:
Visualization of Object Features Using Class Activation Mapping (CAM) [46]. The sequenee from top to bottom illustrates the tasks of door opening, tool use, relocation, and reorientation, while from left right shows the exccution of each task.
(1) With sufficient data, KOROL with learned feature achieves similar or higher success rafe com- pare to Koopman operator with GT object state and other baselines. Across all tasks with 200 demonstrations, we notice the difference of KOROL and Koopman operator on casier tasks (Door opening and Tool use) are minimal and the margin magnified on harder tasks (Relocation and Re- orientation). KOROL with learned feature achieves 4.4 and 2.8 % higher success rate on harder tasks respectively. This enhanced performance is attributed to the capability of the learned features to undergo continuous updates during the training of the ResNet model and to adapt dynamically during task execution (see Figure 3). This approach contrasts with using fixed object state, en- hancing KOROL's generality and robustness. In comparison, KOROL with learned features exceeds the model-based NDP across four tasks with an average enhancement of 1.08x x and surpasses the learning-based Diffusion Policy by 1.16x when supplied with 200 demonstrations.
(2) While KOROL's performance diminishes under limited data (10) constraint, it still substantially outperforms other baselines, suggesting it has better sample efficiency. BC exhibit zero near-zero performance on most tasks, regardless of whether they use GT object states or learned object fea- tures. NDP yield results comparable to KOROL with 200 demonstrations but underperform when reduced to 10, underscoring its dependence on large training datasets. Overall, KOROL exceeds NDP with an average enhancement of 13.77x and surpasses Diffusion Policy by 1.13x when sup- plied with 10 demonstrations. Our experiments also show that KOROL with learned features exhibits smaller performance drop (9.5 % average across four tasks) compared to the Koopman operator with GT object state (23.75 %) when reducing demonstrations from 200 to 10. This sample effi- ciency in KOROL stems from employing the dynamics model Loddel-Koopmnn operator operator-and learning robust, generalizable object features.
Object Feature Visualization. The results in Fig reveal variable focus within the activation maps. Notably, during the door opening task, initial activation predominantly targets the robot's hand, aligning with our training objective to minimize prediction errors in robot state. As the hand approaches the door handle, the activation extends to encompass the hand and the handle. Ulti- mately, the activation map prominently highlights the handle and the door. In the tool-use task, activation primarily centers on the nail and hammer, whereas in the relocation task, focuses on the robot's hand. These activation mappings are derived from ResNet18 in training images, specifically

---

RGBD
1rial
Trial
Trial
Trial
Figure 5: Visualization of Object Features Using CAM in Three Real-World Tasks. From top to bottom, the sequence showcases training images from various trials of toy relocation, teapot pickup, and cube insertion tasks, demonstrating the feature extractor's 's generalization to positional variance.
from the output of the last convolutional layer [46]. These activation maps also serve as valuable indicators the model's training progress. A sufficiently trained KOROL typically exhibits task- relevant feature activation. Conversely, activation focused on irrelevant areas suggests inadequate learning of object features, potentially leading to task failure.
evaluate whether the training of fo depends on the Koopman operator by ablating the update of and plotting the training curves in Figure 4. The blue lines show the stan- dard training of KOROL and the orange lines show the ablation. The loss decreases significantly after recalculating K at epochs 50; otherwise, remains stagnant. Subsequent updates to K at epochs 100, 150, and 200 show minimal impact, likely due to the already diminished magnitude of the loss. Addi- tional ablation studies the performance improve- ment from using frequency domain images can be found in the Appendix.
Effeet of Model Update.
50
dynemics K update (crain)
dynemics K update (vall
40
dynamics update (train)
dynamics K update (vai)
30
20
10-
200
Epoch
Figure 4: Training and Validation Loss Curves in Door Task. The dashed line indicates the times of updating
5.2 Real World Experiment
Setup. In our real-world robot experiments, we employed a 7-DoF Kinova robot equipped with parallel gripper to perform three distinct tasks: (1) Toy relocation: Move the green toy on the gripper to randomized target location (blue bounding box) and release (2) Tea pot pickup: Grasp the handle of the teapot, which is placed a randomized position the table, and lift it up. (3) Cube insertion: Move the blue cube the gripper to randomized target location (shape sorter box) and drop it into the corresponding shape sorter. We provide 20 and 50 unique demonstrations to each task respectively, and compare KOROL NDP and Diffusion Policy.
Numerical Results and Feature Visualization. KOROL consistently outperforms the baselines, achieving superior average performance (see Table 2). The most frequent failure mode for KO- ROL involves the gripper moving to position, typically 1 to cm away from the target, before attempting to grasp the handle or drop the cube. This imprecision results in missing the handle or inaccurately aligning with the shape sorter. In Figure 5, the activation maps of object features delin- eates the bounding box and the teapot. However, it does not highlight the cube shape sorter, instead emphasizing surrounding areas. This may explain the lower success rate in the insertion task.

---

Relocation
Pickup
Insertion
For baseline models, NDP continues to struggle to accurately predict the correct po- sitions in Pickup and Insertion tasks and doesn't have much performance improve- ment even with more data. In the Relocation task, Diffusion Policy generally succeeds in positioning the gripper correctly but fails to learn the appropriate timing for opening the gripper with only 20 demonstrations. How-
Task
20
50
20
50
20
50
NDP
10
11
0
0
Diffusion Policy
13
5
KOROL
20
20
17
19
11
14
Table 2: Real-World Manipulation Quantitative Per- formance. The number of successful task executions all models trained with and demonstrations respectively, and evaluated on 20 unique cases per task in real world.
ever, it improves in opening the gripper with additional training data. In Pickup and Insertion, which require high precision in positional accuracy, Diffusion Policy typically cannot generate sufficiently accurate positions for picking or dropping.
5.3 Multi-tasking Experiment
To evaluate the multitasking capabilities of us- ing object features, we combined the train- ing datasets from four tasks into 800 demon- strations and trained single ResNet model fo alongside multitasking Koopman operator Kmiti. The results in Table reveal that the multitasking Koopman operator sustains robust performance across the Door opening, Tool use, and Reorientation tasks, but exhibits perfor- mance declines in the Relocation task com- pared to KOROL trained with 200 demonstrations per task (sce Table 1). Furthermore, the results highlights the need for a feature extractor with substantial capacity to ensure generalizability across tasks. Specifically, the multitasking Koopman operator Kmult with ResNer34 and ResNet50 im- proves performance the Relocation task over ResNet18. However, ResNet50 may be too large and thus prone to underfitting, leading to decline in performance.
Task
ResNet18
ResNet34
ResNet50
Door opening
99.9%
100%
100%
Tool use
100%
99.9%
100%
Relocation Reorientation
78.2%
93.8%
81.3%
85.9%
86.8%
85.9%
Table 3: Quantitative Performance of KOROL in Multi-Tasking, The averaged multi-tasking success rates across 5 random seeds of KOROL with ResNet18, ResNet34 or ResNet50 trained with 800 demonstrations and evaluated on 200 unseen cases per task.
6 Conclusion
This work introduces and evaluates KOROL, which leverages the Koopman operator rollouts to learn object features for manipulation tasks. KOROL iterative updates the Koopman operator along- side the trained object features to enhance performance. Experiments suggest that KOROL can: (i) improve performance across various simulated manipulation tasks compared to the Koopman oper- ator with GT object state and baseline models, (ii) extend Koopman-bask methods to vision-based real-world tasks, and (iii) facilitate multitasking Kmutti with dimensionally- object features.
7 Limitations and Future Work
KOROL has several limitations and directions for future research: (1) We currently compute the Koopman operator K by solving least-squares problem. Advancements in neural Koopman ap- proaches [7] could allow training the Koopman operator and object features in an end-to-end way. (2) KOROL underperforms in fine-grain manipulation tasks, such as cube insertion. Future work could focus on refining object feature accuracy and enhancing control precision using more ad- vanced feature extractors, such as vision transformers [47]. (3) The CAM visualization technique for object features is restricted to spatial domain RGBD images and is not applicable to frequency domain images. Currently, we verify object feature accuracy through CAM visualization and test model performance using RGB-D images before incorporating frequency domain images to enhance performance, albeit without visualization. Exploring visualization techniques for frequency domain images represents a promising avenue for future research.
8

---

Acknowledgments
The authors would like thank the anonymous reviewers for their insightful feedback, which has helped improve the quality of this paper. We are grateful to Abulikemu Abuduweili, Aviral Agrawal, and Yunhai Han for their valuable discussions and contributions throughout the project. Special thanks go to our advisors, Harish Ravichandar, Changliu Liu, and Jeffrey Ichnowski, for their con- tinuous guidance and mentorship. We also acknowledge the computing resources and robotic in- frastructure provided by Changliu Liu's Intelligent Control Lab at Carnegie Mellon University.
References
[1] R. Hogan, J. Ballester, S. Dong, and A. Rodriguez. Tactile dexterity: Manipulation primi- tives with tactile feedback. In 2020 IEEE international conference on robotics and automation (ICRA), pages 8863-8869. IEEE, 2020.
[2] I. Mordatch, Z. Popović, and Todorov. Contact-invariant optimization for hand manipula- tion. In Proceedings of the ACM SICGRAPH/Eunogaaphies symposium on computer anima- fion, pages 137-144, 2012.
[3] S. Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. Journal Machine Learning Research, 17(39):1-400 2016.
[4] Z. Zhao, Kumar, Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint 13X1v 2023.
[5] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.
[6] Y. Han, M. Xie, Zhao, and H. Ravichandar. On the utility of koopman operator theory in learning dexterous manipulation skills. In Conference on Robot Learning, pages 106-126. PMLR, 2023.
[7] B. Luseh, J. N. Kutz, and S. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics, Nature communications, 9(1):4950, 2018.
[8] S. Bahl, M. Mukadam, A. a. Gupta, and D. Pathak. Neural dynamic policies for end-to-end sensorimotor learning. Advances in Neural Information Processing Systems, 33:5058-555 5069, 2020.
[9] M. Xie, A. Handa, S. Tyree, D. Fox, H. Ravichandar, N. Ratliff, and K. Van Wyk. Neu- ral geometric fabrics: Efficiently learning high-dimensional policies from demonstration. In Conference Robot Learning, pages 1355-1367. PMLR, 2023.
[10] K. Van Wyk. A. Handa, V. Makoviychuk, Y. Guo, Allshire, and N. D. Ratliff. Geometric fabrics: safe guiding medium for policy learning. arXiv preprint a255:2240.002250, 2024.
[11] A. Nagabandi. G. Kahn, Fearing, and S. Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference robotics and automation (ICRA), pages 7559-7566. IEEE, 2018.
[12] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, Schulman, E. Todorov, and S. Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demon- strations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
[13] D. A. Pomerleau. Alvinm: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1. 1988.
[14] Florence, C. Lynch, A. Zeng, O. A. Ramirez, Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. In Conference Robot Learning, pages 158-168. PMLR, 2022.

---

[15] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492-499, 2019.
[16] C. Chi, Feng. Du, Z. Xu, E. Cousineau, Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[17] H. Liu, L. Lee, K. Lee, and P. Abbeel. Instruction-followinng agents with jointly pre-trained vision-language models. 2022.
[18] Y. Li, S. Li, V. Sitzmann, Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. In CoRL, pages 11-1123,022.
[19] Y. Ze, G. Yan, Y.-H. Wu, A. Macaluso, Ge. J. Ye, N. Hansen, L. E. Li, and X. Wang. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In CoRL, pages 284-301. PMLR, 2023.
[20] G. Lu, S. Zhang, Z. Wang, C. Liu, J. Lu, and Y. Tang. Manigaussian: Dynamic gaussian splatting for multi-task robotic manipulation. arXiv preprint arXiv:2403.08322. 2024.
[21] M. T. Mason and J. Salisbury Jr. Robot hands and the mechanics of manipulation. 1985.
[22] S. Collins, A. Ruina, R. Tedrake, and M. Wisse. Efficient bipedal robots based passive- dynamic walkers. Science, 307(5712):1082
2005.
[23] M. P. Deisenroth, D. Fox, and C. Rasmussen. Gaussian processes for data-efficient learning robotics and control. IEEE transactions on pattern analysis and machine intelligence, 37 (2):408-4233 2013.
[24] M. Korda and I. Mezić. Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control. Automatica, 93:149-160, 2018.
[25] Bevanda, S. Sosnowski, and S. Hirche. Koopman operator dynamical models: Learning, analysis and control. Annual Reviews in Control, 52:197-212, 2021.
[26] D. Nguyen-Tuong, M. Seeger, and J. Peters. Model learning with local gaussian process gression. Advanced Robotics, 23(15):2015-2035 2009.
[27] N. R. Ke. A. Singh, A. Touati, A. Goyal, Y. Bengio. D. Parikh, and D. Batra. Modeling the long term future in model-based reinforcement learning. In International Conference on Learning Representations, 2018
[28] Yang, O. Nachum, Y. Du, J. Wei, P. Abbeel, and Schuurmans. Foundation models for decision making: Problems, methods, and opportunities, 2023.
[29] J. Sun, D.-A. Huang, B. Lu, Y.HH. Liu, B. Zhou, and A. Garg. Plate: Visually-grounded planning with transformers in procedural tasks. IEEE Robotics and Automation Letters, 7(2); 4924 4930, 2022.
[30] M. Janner, Y. Du, J. B. Tenenbaum, and Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999) 2022.
[31] Y. Du, T. Lin, and I. Mordatch. Model based planning with energy based models. In Conference on Robot Learning, 2019.
[32] H. Chen, Y. Du, Y. Chen, Tenenbaum, and P. A. Vela. Planning with sequence models through iterative energy minimization. arXiv preprint arXiv:2303.16189, 2023.
[33] B. O. Koopman and J. Neumann. Dynamical systems of continuous spectra. Proceedings of the National Academy of Sciences, 18(3):55--23, 1932.
10

---

[34] B. O. Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences, 17(5)331--318, 1931.
[35] S. L. Brunton, B. W. Brunton, L.. Proctor, and J. N. Kutz. Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control. PloS one, 11(2): e0150171, 2016.
[36] M. Chang, T. Ullman, A. Torralba, and B. Tenenbaum. A compositional object-based approach learning physical dynamics. arXiv preprint arXiv:1612.00331, 2016.
[37] D. Bruder, C. D. Remy, and R. Vasudevan. Nonlinear system identification of soft robot dy- namics using koopman operator theory. In 2019 International Conference on Robotics and Automation (ICRA), pages 6244-6250, IEEE, 2019.
[38] S. L. Brunton, M. Budišić, E. Kaiser, and J. N. Kutz. Modern koopman theory for dynamical systems. arXiv preprint
2021.
[39] E. Yeung, S. Kundu, and N. Hodas. Learning deep neural network representations for koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC), pages IEEE, 2019.
[40] Y. Li, H. He, J. Wu, D. Katabi, and A. Torralba. Learning compositional koopman operators for model-based control. arXiv preprint arXiv:1910008264, 2019.
[41] M. Wang. X. Lou, W. Wu, and B. Cui. Koopman-bused mpe with learned dynamics: Hierarchi- cal neural network approach. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[42] K. Xu, M. Qin, F. Sun, Y. Wang, Y.-K. Chen, and F. Ren. Learning in the frequency domain. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1740-1749, 2020.
[43] J. A. Stuchi, L. Boccato, and R. Attux. Frequency learning for image classification, 2020.
[44] N. Ahmed, T. Natarajan, and R. Rao. Discrete cosine transform. IEEE transactions on Computers, 100(1):90-93, 1974.
[45] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[46] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.
[47] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale. arXiv preprint arXiv:2010. /9229
1929, 2020.
[48] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.

---

Appendix
A ADROIT Hand Experimental Details
A.1 Task State Space Design
Door opening. Given randomized door position, undo the latch and drag the door open. In this task, (t) E X C _ (24-DoF hand + 3-DoF wrist rotation + 1-Dof wrist motion) the floating wrist base can only move along the direction that is perpendicular to the door plane but rotate freely. Regarding the object states, (t) iphandle 2t, pdoor] E X. C R?, containing the door position pdoor, handle position phandle and the angular velocity of the door opening angle 12. In each test case, we randomly sampled door positions puoor (zyz) from uniform distributions: x ~ U(-0.33 0), y (0.2,, 0.35), and (0...5., 0.402).
Tool use. Pick up the hammer to drive the nail into the board placed at randomized height. In this task, (t) E X, C 26 (24-DoF hand 2-DoF wrist rotation) as the floating wrist base can only rotate along the x and y axis. (I) too of [ containing the nail goal position paall, hammer positions poool and orientations o!oll In each test case, we randomly sampled nail height (z) in p from uniform distribution:
2(0.1,10000 0.25).
Object relocation. Move the blue ball randomized target location (green sphere). In this task, (t) X² C R230 (24-DoF hand 6-DoF floating wrist base) the ADROIT hand is fully actuated. (t) = [p) hall of containing the target positions praret and current positions p! In each test case, we randomly sampled target positions ptaget (ryzz from uniform distributions: x ~ 0.25, 0.25), y ~ (.02225, 0.25), and z ~ (0.115 0.35).
In-hand reorientation. Reorient the blue pen to a randomized goal orientation (green pen). In this task, (t) E X, C 2 (24-DoF hand) as floating wrist base is fixed. (t) = [p, pen off pedd containing the goal orientations gBoal and current pen orientations of which are both unit direction vectors. In each test case, we randomly sampled the pitch (a) and yaw (8) angles of the goal orientation o from uniform distributions: ~ 1) and B U( 1,
The task success criteria is the same as defined in [6].
A.2 Policy Design and Training
Koopman Operator The lifting functions of Koopman Operator are taken from [6]. The represen- tation of the system is given as: X, +, ... ",,,"]] and x,, = [x] x2 xx and superscript used to index states. In experiments, the vector-valued lifting functions eh and v/o in (3) were defined as polynomial basis functions:
/ = (O for i.j -1...,, ,n v. [ (())))) ((r)) for i,j 1,..,,
Note that u.f./ and v/fog/rfa each appear only once the lifting functions. / ignored here as the lifting functions are the same across the time horizon. Thus, the dimension of the Koopman Operator K E RPX,, where = 3n + 2m + m² + m(n-1) + m(m-1)
KOROL Training In Door opening and Tool use tasks, the feature extractor is trained solely using RGBD images. While in Relocation and Reorientation tasks, the feature extractor is additionally provided with the desired goal locations p parget and goal orientations gamll The full list of training hyperparameters can found in Table
Effect of Model Update Frequency We analyze the impact of hyperparameter M, the frequency of updating K, on KOROL'S performance in Table 5. Without model updating, the success rate zero because the Koopman operator K becomes outdated for the trained object features. Conversely, excessively frequent updates to K destabilize object feature training and degrade performance, sim- ilar to the target policy update delay in Twin Delayed DDPG (TD3) [48]. Therefore, the update
12

---

Hyperparameter
Value
Feature Extractor
ResNet18
Input RGBD Image Dimension
256 x 256 x 4
Input Desired Poisition and Orientation Encoder
HarnonicEmbeedding
Input Desired Poisition and Orientation Dimension
Output Desired Poisition and Orientation Embedding Dimension
Output Object Feature Dimension
Batch Size
8
Prediction Horizon
40
Learning rate
1 1+10
Adam betas
(0.9, 0.999)
Learning rate decay
Linear decay (see code for details)
Max Training Epoch
300
Max Execution Step Num
100
Table 4: Hyperparameters of KOROL Training for ADROIT Hand Experiments.
Task Rate Demonstrations
Demonstrations
1.0
1.0
0.8
Task Success Rate (8)
0.8
- BC
-
BC
Suucess
NDP
NDP
Diffusion Policy
Diffusion Policy
0.4
KOROL
0.4
KOROL
0.2
0.2
10
50
100
200
10
100
200
Number of Demonstrations
Number Demorstrations
Door opening
Thol use
Task: Success Rate vs Number of Demanstrations
Tack Success Rate us Number of Demonstrations
Tadk Success Rate (95)
0.8
Tank Success Rate
0.6
0.6
NDP
Diffusion Policy
0.4
KOROL
0.2
0.2
0.0
0.0
50
100
200
10
50
100
200
Number Demorstrations
Number of Demarstrations
Relocation
Reorientation
Figure 6: The effects of number of demonstrations on success rate for all models on each task.
frequency of the Koopman operator K should be much lower than that of the object feature training. usually by at least an order of magnitude.
M (Number of epochs for K updating)
10
20
50
100
0O
Num of K Update
100
10
5
2
Success rate
80.4%
87.2%
96.2%
99.9%
99.9%
0%
Table 5: KOROL Performance in Door Opening Task After 100 Training Epochs with Varied K Update
Frequencies.
Effect of Number Demonstrations To investigate scalability and sample efficiency, we trained all models on varying numbers of demonstrations (10, 50, 100, 200) and plot the success rate on 200 unseen cases per task Figure 6. KOROL consistently achieves the highest task success rate in most scenarios, except for the Tool use task with 10 demonstrations.
13

---

NDP
Diffusion
Policy
Figure 7: Visualization Object Features from Baselines Using CAM in Door opening and Tool use Tasks. From top to bottom, the sequence displays visualization images from BC, NDP, and Diffusion Policy.
A.3 Baselines
We ran BC and NDP based on the implementation in [6]
For Diffusion Policy, we used the author's original implementation [16]
We present the object features visualizations of these baselines in Door Opening and Tool Use tasks Figure 7. These tasks involve several objects, including the robot hand, door, handle, hammer, and nails, which require special attention and are easier to interpret compared to Relocation and Reorientation. The results show that the object features learned from BC are more readable than those from NDP and Diffusion Policy, but not as good as KOROL's. This difference may be because KOROL and BC maintain relatively simple model structures that preserve the visual interpretability of object features, while the Diffusion Policy obscures this information due its complex denoising process. However, BC's model power it learns do not fully capture the essential information of the scenes for manipulation. In contrast, KOROL's features effecctivly highlight the door handle, nail, and hammer.
low (0% success rate tasks), and thus the features
14

---

A.4 Inverse Dynamic Controller
We employ a pre-trained inverse dynamics controller C, specific to each task, as detailed in [6]. Each controller C/ trained to output actions corresponding to the dimensionality of the robot state defined for its specific task.
B Real-World Experimental Details
B.1 Robot State Space and Task Definition
In the physical robot experiment, we employ a Kinova robotic arm. The configuration space of the robot (4) E X, C RR includes three degrees of freedom (DOF) for the end-effector's position, three DOF for its orientation (ranging from to 360 degrees), and one DOF for the gripper's position (ranging from 0 to 1). The task definition and success criteria are discussed in Section 5.2.
B.2 Experiment Details
The Koopman Operator design, KOROL and baselines training the in simulation. The only difference is that longer need to use inverse dynamic controller to compute torque for each joint. Instead, we publish the predicted end-effector position and gripper position through Kinova API to control robot.
Door opening
Tool use
Relocation
Reorientation
Model
50
200
10
200
10
50
200
50
200
w/o
0%
0%
0%
NDP w/o
0%
39.5%
99.3%
0%
43.4%
96.2%
0%
18.0%
92.7%
25.3%
35.6%
67.7%
Diffusion Policy w/o
93.2%
95.3%
99.9%
97.8%
99.6%
99.7%
86.4%
97.75
100%
31.5%
31.8%
33.0%
KOROL
95.7%
BC
0%
0%
0%
0%
0%
0%
0%
0%
NDP w
0%
87.6%
95.2%
0%
65.4%
89.2%
0%
27.8% 97.8%
100%
14.1%
26.3%
27.9%
Diffusion Policy
74.2%
72.1
66.1%
51.7%
100%
99.9%
90.9%
100%
30.99
30.5%
38.69
KOROL. 98.6%
99.9 %.
99.9%
94.3%
100%
100%
99.8%
100%
100%
55.6%
83.2%
86.4%
Table KOROL and Baselines Performance in ADROIT Hand with and w/o Frequency Domain Image, trained with 10, 50 and 200 demonstrations per task.
Task
Relocation
Pickup
Insertion
KOROL w/o
19/20
17/20
6/20
KOROL w
20/20
19/20
11/20
Table KOROL Performance in Real-World Manipulation with and w/o Frequency Domain Images.
KOROL w/o transformation
KOROL ResNet18
Task
ResNet18
ResNet34
ResNet50
ResNet34
ResNet50
Door opening
99.9%
96.0%
0%
99.9%
100%
100%
Tool use
75.3%
48.9%
0%
100%
99.9%
100%
Relocation
49.1% 19.1%
91.6%
0%
78.2% 85.9%
93.8%
81.3%
Reorientation
86.6%
85.3%
23.8%
86.8%
85.9%
Table 8: KOROL Performance in Multi-tasking Tasks with and w/o Frequency Domain Images.
C Multi-tasking Experimental Details
As discussed in Section A, the robot state space in the Mujoco environment varies slightly across different tasks. To standardize this, augment the state space to 3030 which includes 24-DoF
15

---

hand and a 6-DoF floating wrist base, by padding zeros to the missing robot states. For instance, in Door opening task, we pad zeros to the Tr and Ty motion directions.
For multi-tan multi-tasking controllers, is necessary to remove the padding from the robot state and select the appropriate elements compute the action accordingly. When evaluating the unified Koopman operator K and the feature extractor fe. continue to specific controller C' for each task due to time constraints. However, we believe it is entirely feasible to train a single, unified controller C for all tasks with dimensionally-aligen demonstrations.
D Ablation of Using Image Transformation
Because of the enhanced performance observed in prior works [42, 43] using frequency domain images, this section evaluates the impact of employing transformed images the frequency do- main across various settings: simulation, real-world manipulation, and multi-tasking. The model denoted as KOROL utilizes both spatial and frequency- w/o transformation uses only spatial images. The results in Table 6, Table 7 and Table 8 demon- strate significant of KOROL improvements achieved by incorporating transformed images in all tasks, corroborating the findings in [42, 43]. In detail, learning in the frequency domain helps neural network to learn richer features leading to higher distinguishing power between inputs with high similarity. For example, in the image observations of Door opening task, high-freyuency com- ponents in the transformed input capture the slight change of position of the door and the robot hand. the same time, the low-frequency components capture the general scene elements that remain largely unchanged, like the background. Table 6, observed similar performance im- provements in NDP with 50 demonstrations, but drop in Diffusion Policy across most cases (10, 50, 200 demonstrations). We noted that both NDP and KOROL maintain a relatively simple struc- ture by using object features as input to construct dynamical system, which is used for predicting robot trajectories. However, in Diffusion Policy, the features are used as inputs for further neural network computations (c.g., UNet denoising) that generate downstream features for their respective predicting modules. While DCT-based features help in creating richer features, subsequent neural feature extraction requires further experimentation to work effectively in tandem with DCT features.
images as inputs, whereas KOROL
16


Reviews:
[{"review_id": "pyBb4eIqtE", "reviewer": "Reviewer_ekAf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes to use Koopman operator with learned visual features directly from images, named KOROL. Koopman operator has been previously used with low-level object state, which makes it difficult to be applied in the real world. The proposed method uses a learned visual feature extracted from RGBD images. KORAL is evaluated on both sim and real data.", "summary_of_recommendation": "Interesting application but the experiments need to be strenghthened. I'm inclined to rejection for the current version.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- It wasn’t mentioned how many testing episodes are used. And I would love to see some numbers about variance since some numbers are really close.\n- Does the performance plateau at 200 demonstrations? It would be nice to have a plot so that the sample efficiency can be compared in a more straightforward way.\n- Why is Koopman having a big performance drop (compared to diffusion policy) on the reorientation task?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fa08mN2U2P", "reviewer": "Reviewer_aTms", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces KOROL, a behavior cloning method that learns visual features that observe the Koopman Operator in the feature space. KOROL learns a visual extractor and also the Koopman matrix at the same time. KOROL shows relatively superior performance than model-based and model-free baselines in ADROIT and three real-world tasks.", "summary_of_recommendation": "This is a well-executed paper with a novel idea, despite certain limitations. I recommend acceptance of the paper, and I hope the authors further address the limitations in the rebuttal.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "DCT is critical to the high success rate of KOROL as Table 5 in the appendix shows. Could you provide any explanation if you have any? I wonder if DCT brings some benefit that is specific to KOROL. Also for a fair comparison, DCT should also be applied to the baselines.\n\nCould you comment on how you can better handle more fine-grained manipulation tasks with some extension of KOROL?\n\nMulti-modality has been known to help BC generalize and be robust to perturbations — Does KOROL also handle multi-modality to some extent?\n\nAbout Figure 4: is it reasonable to consider validation loss at all with learning Koopman operator? It is possible the model is overfitting with the training loss drop. Or the magnitude is too large for any overfitting?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5JYQQNrhkp", "reviewer": "Reviewer_wypC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method for learning Koopman operators from high-dimensional observations in the context of dexterous manipulation tasks, enabling model-based control without the assumption of low-dim states. The central contributions are an algorithm and system architecture for jointly learning object feature encoders jointly with the Koopman operator from demonstrations. The model rollouts can then be tracked with a learned inverse dynamics controller from previous work. The experimental evaluation demonstrates the proposed approach outperforms state-of-the-art imitation learning algorithms on simulated Adroit tasks and real pick-and-place tasks.", "summary_of_recommendation": "Overall, the paper studies an important problem, offers an exciting solution, and provides thorough experimentation and analysis.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Please see weaknesses above; in particular, some discussion and/or experiments about additional sensing modalities, and perhaps additional real-world comparison with differing amounts of data could strengthen the argument.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper uses a koopman operator to learn interpretable visual feature for manipulation. In general, reviewers leaned towards positive on this work, though had some outstanding queries to be addressed in the rebuttal.

Strengths:
* Simple model, well explained
* Good multi-task transfer experimentation

Weaknesses:
* Some mismatch between claims and experiments, consider revising to align with experiments actually carried out 
* Relatively minor improvements over prior work (eg. KODex)
* Unclear where limitations are, given experimental settings
* Some questions remain about experimental settings

Post rebuttal:
Authors provided a strong rebuttal, including additional experimentation results and ablations. Reviewers are in agreement that this paper should be accepted.
Decision: Accept