KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipulation

Authors: hongyi  chen
abulikemu  abuduweili
aviral  agrawal
yunhai  han
harish  ravichandar
changliu  liu
jeffrey  ichnowski
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 05:06:20

Abstract: Learning dexterous manipulation skills presents significant challenges due to complex nonlinear dynamics that underlie the interactions between objects and multi-fingered hands. Koopman operators have emerged as a robust method for modeling such nonlinear dynamics within a linear framework.
However, current methods rely on runtime access to ground-truth (GT) object states, making them unsuitable for vision-based practical applications.
Unlike image-to-action policies that implicitly learn visual features for control, we use a dynamics model, specifically the Koopman operator, to learn visually interpretable object features critical for robotic manipulation within a scene.
We construct a Koopman operator using object features predicted by a feature extractor and utilize it to auto-regressively advance system states. We train the feature extractor to embed scene information into object features, thereby enabling the accurate propagation of robot trajectories.
We evaluate our approach on simulated and real-world robot tasks, with results showing that it outperformed the model-based imitation learning NDP by 1.08$\times$ and the image-to-action Diffusion Policy by 1.16$\times$. The results suggest that our method maintains task success rates with learned features and extends applicability to real-world manipulation without GT object states. Project video and code are available at: https://github.com/hychen-naza/KOROL.

Main paper: 
 related work: 
 Imitation Learning and Visual Representations for Manipulation. Imitation learning serves as a primary method for teaching robots to manipulate objects by mapping observations or world states directly to actions. Common approaches include Behavioral Cloning (BC) [13] , Implicit Behavioral Cloning (IBC) [14] , Long Short-Term Memory (LSTM) networks [3, 15] , Transformers [4, 5] , and Diffusion Models [16] . A significant challenge in imitation learning is representing the visual information of a scene. Strategies include using pre-trained 2D [17] or 3D backbones [5] to output visual embeddings. Other works propose end-to-end learning approaches that simultaneously train the visual encoder and the learning policy [4, 16] . Other techniques focus on learning visual representations through correspondence models [15] , self-supervised novel view reconstruction using Neural Radiance Fields (NeRF) [18, 19] , and Gaussian Splatting [20] .Model-Based Learning and Planning. In robotics, traditional model-based approaches rely on expert knowledge of physics to design system models [21, 22, 2] . Since traditional methods can miss complex nonlinearities, and end-to-end learning approaches can be data-intensive, a middle ground of data-driven model learning shows promise as a data-efficient way to derive complex models [6, 23] . Model learning includes a variety of dynamics models such as Koopman operators [24, 25] , Deep Neural Koopman operators [7] , Dynamic Movement Primitives [8] , Neural Geometric Fabrics [9, 10] , and others [11, 26] . Additionally, some studies focus on learning environmental responses to actions to plan a future trajectory [27, 28, 29] , integrate planning in a generative modeling process [30, 16] , and seamlessly blend the learning of models with planning [31, 32] .Koopman Operator Theory. In the early 1930s, Koopman and Von Neumann introduced the Koopman operator theory to transform complex, nonlinear dynamics systems into linear ones in an infinite-dimensional vector space, using observables as lifted states [33, 34] . This transform allows the application of linear system tools for effective prediction, estimation, and control with handdesigned observables [6, 35, 36, 37] . Recent methods using neural networks to learn observables have proven more expressive and effective, particularly in chaotic time-series prediction [7, 38, 39] . Furthermore, the integration of neural network-derived Koopman observables with Model Predictive Control has shown promise in enhancing control tasks [40, 41] . As a significant benchmark, Han et al. [6] demonstrate the effectiveness of Koopman operators in manipulation tasks using GT object states. Building on this foundation, we extend the application of the Koopman operator to visionbased manipulation tasks in real-world settings by learning object features directly from images 3 Background: Koopman Operator TheoryIn this section, we provide a brief background on the Koopman Operator Theory. Consider the evolution of nonlinear dynamics system x(t + 1) = F (x(t)). Given the original state space X , the Koopman Operator K introduces a lifted space of observables O using lifting function g : X → O, to transform the nonlinear dynamics system into a linear system in infinite-dimensional observables space as g(x(t + 1)) = Kg(x(t)).In practice, we approximate the Koopman operator by restricting observables to be a finitedimensional vector space. Let ϕ(x(t)) ∈ R p represent a finite dimensional approximation of observables g(x(t)), and a matrix K ∈ R p×p approximate the Koopman operator K. Thus, we rewrite the relationship asEQUATIONGiven a dataset D, in which each trajectory τ = [x(1), x(2), • • • , x(T )] containing T time steps, we can learn K by minimizing the state prediction error [38] EQUATIONIn manipulation tasks, we define the statex(t) = [x r (t) ⊤ , x o (t) ⊤ ] ⊤ to include the robot state x r (t)and object state x o (t), as we care about how objects move as a result of robot's motion. Moreover, since our goal is to minimize the imitation error of the robot state x r (t), we design observables ϕ(x(t)) that include lifted robot and object states asEQUATIONwhere ψ r : R n → R n′ and ψ o : R m → R m′ are vector-valued lifting functions that transform the robot and object state respectively. We can thus retrieve the desired robot state by selecting the corresponding elements in ϕ(x(t)). Let ϕ -1 denote the unlifting function to reconstruct the robot state from observables, x r (t) = ϕ -1 • ϕ(x(t)) (we can also reconstruct the object state x o (t) in the same way). Considering the lifting function Equation 3, the unlifting function can be represented asEQUATIONwhere I n×n and 0 n×n denote an identity matrix and zero matrix respectively. To streamline notation throughout this paper, we definexr (t + 1) = K ′ (x r (t), x o (t)), where K ′ := ϕ -1 • K • ϕ. 
method: 
 We propose KOROL, which formulates dynamics learning and object feature learning as an imitation (supervised) learning problem on robot states. Given a dataset D, in which each trajectory τ = [x r (1), y(1), x r (2), y(2), • • • , x r (T ), y(T )] containing robot states x r (t) and image observation y(t) of the object, instead of object state x o (t), our goal is to learn a visual object feature extractor f θ and a Koopman operator K which can predict object features from images that minimize the robot states imitation errors. In this formulation, (2) becomes Learning object feature. While traditional Koopman operator construction requires GT object state information x o (t) (section 3), instead, we adopt a neural network f θ for object feature encoding and extraction from RGBD images. We initialize f θ and predict object features xo (t) for all images in D to construct the initial Koopman operator K, as proposed by Han et al. [6] . During training, we randomly sample the beginning time step t 0 in trajectory τ , and select xo (t 0 ) = f θ (y(t 0 )). Then we can advance the observables forward using Kϕ(•, •). We train f θ by minimizing the loss functionEQUATION)EQUATION)where N is the prediction horizon and xr (0) = x r (0) indicates that the system provides the initial GT robot state. See Figure 2 for visualization. Integrating spatial domain RGBD images with their frequency domain counterparts has been shown to enhance image classification performance by accentuating discriminative features [42, 43] . Therefore, we apply the Discrete Cosine Transform [44] to convert RGBD images into the frequency domain. We then concatenate the spatial and frequency domain images as input, enabling f θ to detect changes in successive, highly-correlated images more effectively than using spatial images alone. Subsequently, KOROL generates the reference trajectory ({x r (t)} N t=1 ) by rolling out the dynamics K. We feed these trajectories into the pre-trained inverse dynamic controller [6] , which computes the required action a(t) using xr (t) and xr (t + 1).Updating of Koopman Operator K. Subsequent updates to the Koopman operator K are necessitated by changes in the predicted object features xo (t), because K is optimized for these specific robot states and object features. See pseudocode in Alg. 1 for details. During the training of f θ from line 7 to line 15, the dynamics K initially computed at line 3 may no longer be optimal for the new object features, prompting a need for recalculation. However, recalculating the object features across the entire training dataset and updating K for every f θ modification is computationally intensive. Therefore, in KOROL, we defer the updates and recalculate K every M epoches to balance accuracy with computational efficiency, as detailed from line 16 to line 20.Multi-tasking Koopman Operator. While a robot's state space remains consistent when using the same robot platform, object state spaces typically vary across different tasks. For example, a prior Koopman manipulation study [6] includes a 15-DoF tool use task and a 7-DoF door opening task. Due to the differences in object state space definition and dimensions, Koopman operators trained for different tasks can not be shared, limiting their scalability. In KOROL, we propose training object features xo (t) to serve as a universal interface for representing length-varied object states end if 20: end for across tasks, enabling the generalization of K to multiple tasks. Moreover, these object features act as latent conditional vectors that differentiate among tasks. Thus, as long as the feature extractor can identify useful object features, it is possible to use datasets from various tasks to train a single multi-task Koopman operator K multi . 
experiments: 
 In this section, we evaluate the performance of KOROL along with existing unstructured learning and model-based learning approaches in simulation and real-world tasks.Setup and Baselines. We conducted our simulation experiments on the ADROIT Hand [12] -a 30-DoF simulated system (24-DoF articulated hand + 6-DoF floating wrist base). There are 4 simulation tasks: Door opening, Tool use, Object Relocation, and In-hand Reorientation. We compared KOROL to the baselines: (1) Behavior Cloning (BC): Unstructured fully-connected neural network policy; (2) Neural Dynamic policy (NDP): Neural network policy with embedded structure of dynamics systems [8] ; (3) Diffusion Policy: Learning policy using probabilistic generative model [16] . To allow equal comparison, all models use ResNet18 [45] (1) With sufficient data, KOROL with learned feature achieves similar or higher success rate compare to Koopman operator with GT object state and other baselines. Across all tasks with 200 demonstrations, we notice the difference of KOROL and Koopman operator on easier tasks (Door opening and Tool use) are minimal and the margin magnified on harder tasks (Relocation and Reorientation). KOROL with learned feature achieves 4.4 % and 2.8 % higher success rate on harder tasks respectively. This enhanced performance is attributed to the capability of the learned features to undergo continuous updates during the training of the ResNet model and to adapt dynamically during task execution (see Figure 3 ). This approach contrasts with using a fixed object state, enhancing KOROL's generality and robustness. In comparison, KOROL with learned features exceeds the model-based NDP across four tasks with an average enhancement of 1.08× and surpasses the learning-based Diffusion Policy by 1.16× when supplied with 200 demonstrations.(2) While KOROL's performance diminishes under limited data (10) constraint, it still substantially outperforms other baselines, suggesting it has better sample efficiency. BC exhibit zero or near-zero performance on most tasks, regardless of whether they use GT object states or learned object features. NDP yield results comparable to KOROL with 200 demonstrations but underperform when reduced to 10, underscoring its dependence on large training datasets. Overall, KOROL exceeds NDP with an average enhancement of 13.77× and surpasses Diffusion Policy by 1.13× when supplied with 10 demonstrations. Our experiments also show that KOROL with learned features exhibits a smaller performance drop (9.5 % average across four tasks) compared to the Koopman operator with GT object state (23.75 %) when reducing demonstrations from 200 to 10. This sample efficiency in KOROL stems from employing the dynamics model-Koopman operator-and learning robust, generalizable object features.Object Feature Visualization. The results in Fig 3 reveal variable focus within the activation maps. Notably, during the door opening task, initial activation predominantly targets the robot's hand, aligning with our training objective to minimize prediction errors in robot state. As the hand approaches the door handle, the activation extends to encompass the hand and the handle. Ultimately, the activation map prominently highlights the handle and the door. In the tool-use task, activation primarily centers on the nail and hammer, whereas in the relocation task, it focuses on the robot's hand. These activation mappings are derived from ResNet18 in training images, specifically Effect of Model Update. We evaluate whether the training of f θ depends on the Koopman operator by ablating the update of K and plotting the training curves in Figure 4 . The blue lines show the standard training of KOROL and the orange lines show the ablation. The loss decreases significantly after recalculating K at epochs 50; otherwise, it remains stagnant. Subsequent updates to K at epochs 100, 150, and 200 show minimal impact, likely due to the already diminished magnitude of the loss. Additional ablation studies on the performance improvement from using frequency domain images can be found in the Appendix.Setup. In our real-world robot experiments, we employed a 7-DoF Kinova robot equipped with a parallel gripper to perform three distinct tasks: (1) Toy relocation: Move the green toy on the gripper to a randomized target location (blue bounding box) and release it. (2) Tea pot pickup: Grasp the handle of the teapot, which is placed at a randomized position on the table, and lift it up. (3) Cube insertion: Move the blue cube on the gripper to a randomized target location (shape sorter box) and drop it into the corresponding shape sorter. We provide 20 and 50 unique demonstrations to each task respectively, and compare KOROL to NDP and Diffusion Policy.Numerical Results and Feature Visualization. KOROL consistently outperforms the baselines, achieving superior average performance (see Table 2 ). The most frequent failure mode for KO-ROL involves the gripper moving to a position, typically 1 to 2 cm away from the target, before attempting to grasp the handle or drop the cube. This imprecision results in missing the handle or inaccurately aligning with the shape sorter. In Figure 5 , the activation maps of object features delineates the bounding box and the teapot. However, it does not highlight the cube shape sorter, instead emphasizing surrounding areas. This may explain to the lower success rate in the insertion task. 1 ). Furthermore, the results highlights the need for a feature extractor with substantial capacity to ensure generalizability across tasks. Specifically, the multitasking Koopman operator K multi with ResNet34 and ResNet50 improves performance in the Relocation task over ResNet18. However, ResNet50 may be too large and thus prone to underfitting, leading to a decline in performance. 
conclusion: 
 This work introduces and evaluates KOROL, which leverages the Koopman operator rollouts to learn object features for manipulation tasks. KOROL iterative updates the Koopman operator alongside the trained object features to enhance performance. Experiments suggest that KOROL can: (i) improve performance across various simulated manipulation tasks compared to the Koopman operator with GT object state and baseline models, (ii) extend Koopman-based methods to vision-based real-world tasks, and (iii) facilitate multitasking K multi with dimensionally-aligned object features. 
limitations and future work: 
 KOROL has several limitations and directions for future research: (1) We currently compute the Koopman operator K by solving a least-squares problem. Advancements in neural Koopman approaches [7] could allow training the Koopman operator and object features in an end-to-end way.(2) KOROL underperforms in fine-grain manipulation tasks, such as cube insertion. Future work could focus on refining object feature accuracy and enhancing control precision using more advanced feature extractors, such as vision transformers [47] . (3) The CAM visualization technique for object features is restricted to spatial domain RGBD images and is not applicable to frequency domain images. Currently, we verify object feature accuracy through CAM visualization and test model performance using RGB-D images before incorporating frequency domain images to enhance performance, albeit without visualization. Exploring visualization techniques for frequency domain images represents a promising avenue for future research. 
: 
 Humans possess an extraordinary ability to manipulate objects, discerning position, shape, and other properties with just a glance. How can robots be endowed with similar perceptual and dexterous manipulation capabilities? Traditional control and optimization approaches typically require detailed models of the system dynamics [1, 2] . However, these models can be difficult to derive and often lack the flexibility and generalizability needed to adapt to task or environment changes. End-to-end data-driven methods overcome these challenges by learning actions directly from observations [3, 4, 5] . While these methods can make minimal assumptions, they often require a large number of demonstrations to master basic skills due to the high dimensionality of the inputs.To combine sample efficiency of traditional model-based approaches with high generalizability of deep learning methods, one branch of recent work has focused on learning dynamics models to plan 8th Conference on Robot Learning (CoRL 2024), Munich, Germany.trajectories. These methods embed learning into various models, such as Koopman operator [6, 7] , Dynamic Movement Primitives (DMP) [8] , Neural Geometric Fabrics [9, 10] , and more [11] ,showcasing good performance in simulations. However, they often falter in real-world applications due to their reliance on hard-to-obtain ground-truth (GT) state information like object poses and contact points. Moreover, the problems of using computer vision to estimate these object states are the uncertainty in determining the number of objects to consider and which specific states to estimate. Additionally, the learned dynamics models do not transfer across different tasks without a universal state space design.We propose an approach to remove the dependency on GT states in model-based manipulation learning (Figure 1 ). Central to this approach, Koopman Operator Rollout for Object Feature Learning (KOROL), is learning visual features that predict robot states during dynamics model rollouts. Unlike learning methods that learn implicit visual features for image-to-action policies, KOROL explicitly trains on visual object features, encoding essential scene information to enhance predictions of robot states during autoregressive model rollouts. Central to this rollout process is the Koopman operator, which utilizes current object features to advance robot states. This establishes a synergistic relationship between the learned object features and the Koopman operator. KOROL uses trained features to refine the Koopman operator, which in turn improves the feature learning process through a more accurate dynamics model.In experiments, we demonstrate how KOROL outperforms prior methods in ADROIT Hand [12] and generates interpretable visualizations by learning object features from images. We also show how KOROL enables the application of Koopman operator in vision-based real-world manipulation tasks. Finally, we demonstrate how learning object feature instead of designing object states for dynamics modeling enables KOROL to construct universal Koopman dynamics across multiple manipulation tasks. This paper makes the following contributions:• We introduce KOROL, an imitation learning method, that uses the Koopman operator to learn object features and show that the Koopman operator with learned object features can outperform the one with GT object states. • We extend the application of the Koopman operator to vision-based manipulation tasks in realworld by learning object features from images and demonstrate its effectiveness through comparisons with prior methods. • We demonstrate that KOROL learns dimensionally-aligned object features across tasks, enabling the development of a multi-tasking Koopman operator. 
 
 Appendix: We employ a pre-trained inverse dynamics controller C, specific to each task, as detailed in [6] . Each controller C is trained to output actions corresponding to the dimensionality of the robot state defined for its specific task.B Real-World Experimental Details B.1 Robot State Space and Task DefinitionIn the physical robot experiment, we employ a Kinova robotic arm. The configuration space of the robot x r (t) ∈ X r ⊂ R 7 includes three degrees of freedom (DOF) for the end-effector's position, three DOF for its orientation (ranging from 0 to 360 degrees), and one DOF for the gripper's position (ranging from 0 to 1). The task definition and success criteria are discussed in Section 5.2.The Koopman Operator design, KOROL and baselines training are the same as in our simulation.The only difference is that we no longer need to use an inverse dynamic controller to compute torque for each joint. Instead, we publish the predicted end-effector position and gripper position through Kinova API to control robot. As discussed in Section A, the robot state space in the Mujoco environment varies slightly across different tasks. To standardize this, we augment the state space to R 30 , which includes a 24-DoF 


Decision: This paper uses a koopman operator to learn interpretable visual feature for manipulation. In general, reviewers leaned towards positive on this work, though had some outstanding queries to be addressed in the rebuttal.

Strengths:
* Simple model, well explained
* Good multi-task transfer experimentation

Weaknesses:
* Some mismatch between claims and experiments, consider revising to align with experiments actually carried out 
* Relatively minor improvements over prior work (eg. KODex)
* Unclear where limitations are, given experimental settings
* Some questions remain about experimental settings

Post rebuttal:
Authors provided a strong rebuttal, including additional experimentation results and ablations. Reviewers are in agreement that this paper should be accepted.

Reviews: review: title: title: Simple yet interesting method but experiments need to be improved
paper_summary: paper_summary: This paper proposes to use Koopman operator with learned visual features directly from images, named KOROL. Koopman operator has been previously used with low-level object state, which makes it difficult to be applied in the real world. The proposed method uses a learned visual feature extracted from RGBD images. KORAL is evaluated on both sim and real data.
main_review: main_review: ## Comments
+ The method is simple. The core contribution is to use a learned model in visual encoding. It’s nice having a learned visual feature extractor to replace the low-level object states. Together with the proposed training scheme, KOROL makes the Koopman operator more applicable in real-world settings.
+ Section 3 is well-written and provides a nice walk-through for readers not familiar with the Koopman operator. The equations are clear and easy to follow.
- I do think the experiments need to be improved, or the claims should be revised. Firstly, the experiments focus a lot on ‘sample efficiency’ of the Koopman operator from my view. It was compared to baselines using different numbers of training data. However, sample efficiency is not emphasized in previous sections, but only shows up as an analysis of the experimental result. It would be better to include discussions in previous sections on this point. Secondly, if the core contribution of KOROL is removing the need for GT low-level object state,  an important baseline will be trying to use perception modules to extract object state from images, and show that KOROL does better than that.
- I don’t understand why learned visual encoding will be better than GT object state. The GT state should always contain more information than a learned visual feature if all relevant low-level states are included in the GT state. Does the designed `GT object state' include all relevant information? e.g. according to the appendix, in tool use, only the goal position of the nail is in the object state, but not the current state of the nail.


## Misc
- I didn’t realize that K is computed by solving the least-squares problem, and I was confused about why not updating K more frequently. It could be helpful to clarify that earlier.
- For clarity, it’s better to swap 5.2 and 5.3 in my opinion, and somehow shows the original performance in Table 3. Right now it’s difficult to compare the success rate of 800 demonstrations to 200 demonstrations based on Table 3.
- Varying M (epoch num to update K) could be an interesting ablation.
questions: questions: - It wasn’t mentioned how many testing episodes are used. And I would love to see some numbers about variance since some numbers are really close.
- Does the performance plateau at 200 demonstrations? It would be nice to have a plot so that the sample efficiency can be compared in a more straightforward way.
- Why is Koopman having a big performance drop (compared to diffusion policy) on the reorientation task?
limitations: limitations: 3
review_summary: review_summary: Interesting application but the experiments need to be strenghthened. I'm inclined to rejection for the current version.
title: title: Well-executed work with certain limitation
paper_summary: paper_summary: This paper introduces KOROL, a behavior cloning method that learns visual features that observe the Koopman Operator in the feature space. KOROL learns a visual extractor and also the Koopman matrix at the same time. KOROL shows relatively superior performance than model-based and model-free baselines in ADROIT and three real-world tasks.
main_review: main_review: Strength:

1. I think the overall idea of learning visual features and applying Koopman operation is intuitive and well executed. The experiment results show significant improvement in BC success rate compared to strong baselines such as NDP and Diffusion Policy. (To be honest and no discredit to the authors, as someone who has not followed Koopman literature in robotics, I am surprised that this idea has not been explored before.)

2. Multi-task positive transfer shown in 5.3 is well appreciated — I think this is a missing piece of many recent BC literature. I also think more multi-task experiments with baselines would be great. I wonder if Koopman Operator encourages more positive transfer than baselines.

Limitation:

1. As the authors mention, KOROL underperforms in fine-grained manipulation tasks. To me this is an inherent limitation of using linear Koopman Operator, which limits the expressivity of the policy. My intuition is that on more challenging real-world tasks KOROL might not be scalable.

2. I am not sure about the object feature visualizations — since there is no comparison to features learned from the baselines, I don’t feel Fig. 4 and 5 help illustrate the benefit of KOROL much.
questions: questions: DCT is critical to the high success rate of KOROL as Table 5 in the appendix shows. Could you provide any explanation if you have any? I wonder if DCT brings some benefit that is specific to KOROL. Also for a fair comparison, DCT should also be applied to the baselines.

Could you comment on how you can better handle more fine-grained manipulation tasks with some extension of KOROL?

Multi-modality has been known to help BC generalize and be robust to perturbations — Does KOROL also handle multi-modality to some extent?

About Figure 4: is it reasonable to consider validation loss at all with learning Koopman operator? It is possible the model is overfitting with the training loss drop. Or the magnitude is too large for any overfitting?
limitations: limitations: 2
review_summary: review_summary: This is a well-executed paper with a novel idea, despite certain limitations. I recommend acceptance of the paper, and I hope the authors further address the limitations in the rebuttal.
title: title: KOROL
paper_summary: paper_summary: The paper proposes a method for learning Koopman operators from high-dimensional observations in the context of dexterous manipulation tasks, enabling model-based control without the assumption of low-dim states. The central contributions are an algorithm and system architecture for jointly learning object feature encoders jointly with the Koopman operator from demonstrations. The model rollouts can then be tracked with a learned inverse dynamics controller from previous work. The experimental evaluation demonstrates the proposed approach outperforms state-of-the-art imitation learning algorithms on simulated Adroit tasks and real pick-and-place tasks.
main_review: main_review: Strengths: 
- The paper proposes an exciting method for scaling Koopman operator learning  to high-dimensional observations for dexterous manipulation tasks, a critical challenge in enabling model-based control in realistic real-world conditions. 
- The move to visual observations allows for multi-task learning in a unified observation space. 
- The paper is clearly written and provides a concise, self-contained introduction to Koopman theory. 
- The method is technically sound and design decisions are justified with ablations. 
- The evaluation is comprehensive and demonstrates improvement over state-of-the-art imitation learning algorithms in sim and real tasks. 

Weaknesses:
- The scientific contribution is relatively minor over the previous work in KODEX; that said, I think there is enough technical detail and experimental evaluation to support this as a standalone paper. 
- The imitation objective is defined over robot state only. Is this always sufficient? It would be interesting to see more complex observation spaces like force/torque and tactile data which may be necessary for more dexterous tasks. Would the approach scale to these observations? Should they also be encoded into a learned space rather than regressed directly? Some additional simulation experiments in this direction could be interesting. 
- The number of demonstration trajectories in the real experiments seems arbitrary. Using a more standard quantity (i.e., 50-100) and performing the same evaluation scheme as in sim (i.e., to quantify the effect of more data) would strengthen the argument.
questions: questions: Please see weaknesses above; in particular, some discussion and/or experiments about additional sensing modalities, and perhaps additional real-world comparison with differing amounts of data could strengthen the argument.
limitations: limitations: 3
review_summary: review_summary: Overall, the paper studies an important problem, offers an exciting solution, and provides thorough experimentation and analysis.