Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The paper presents a method for efficiently learning high degree-of-freedom manipulation tasks by leveraging Koopman operators. Using expert demonstrations, they learn a high-dimensional linear model that can be used to generated reference trajectories that complete the task. This is combined with a learned tracking controller that tracks the generated reference trajectories. The proposed method is evaluated on four complex dexterous manipulation tasks and compared to four baseline reinforcement learning methods. The method performs comparably with state-of-the-art approaches while taking dramatically less time to train. 
review: Overall, a solid paper with meaningful contributions to the field of robotics and manipulation. The paper is well written, with a solid background and introduction, a clear and concise description of the method, and meaningful examples with easy-to-interpret results. 

The results of the paper are noteworthy: they achieve performance on par with recent state-of-the-art methods with a method that takes significantly less training time. While there are many areas for improvement and future work, the community will benefit from the results of this paper, which clearly show that learning methods outside the realm of neural networks and deep reinforcement learning have exceptional merit and are worth exploring. 

Strengths:
* Clearly written
* Impressive results
* Use of a lesser-known learning technique
* Clear and honest evaluation of current learning-based methods for manipulation
* Consideration and evaluation of training time as an important metric

Weaknesses:
* Limitations and future work section is lacking
* Description of the overall training pipeline is somewhat unclear
* Confusion over what dynamics are being learned by the method
* A few areas where the theory is glossed over or not expressed correctly
* Too many discussion points are deferred to Appendices
* Lack of comparison with non-learning approaches
* Lack of demonstration of physical hardware (minor, given it's noted in the limitations section and left for future work)
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I believe this paper should be accepted to CoRL. The results are compelling, the paper is well-written, and the method offers a different perspective than most of the papers typically seen at this conference. I believe the community will benefit from seeing the results of this paper, which offers a new avenue to explore as we seek to overcome the limitations of traditional neural-network and deep learning methods.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors investigate the utility of Koopman operator theory in alleviating the limitations of existing learning-based approaches, with a particular focus on general efficacy, computational efficiency, sample efficiency, and scalability. They develop a novel imitation learning framework, dubbed Koopman Operator-based Dexterous Manipulation (KODex), to evaluate the utility of Koopman operator theory for dexterous manipulation. Their experimental evaluation is carried out on four dexterous manipulation skills on the simulated Adroit hand: Tool Use, Door Opening, Relocation, and Reorientation. They benchmarked their method against NN and LSTM (unstructured baselines) and NDP and NGF (highly-structured baselines). They conclude that their KODex can i) analytically learn dexterous manipulation skills and ii) match or outperform SOTA imitation learning approaches on various dexterous manipulation tasks, while being an order of magnitude faster.

review: Quality: The overall quality of the work is good as it addresses an important challenge in using learning-based approaches, by adopting theories from control such as the Koopman operator theory. 

Clarity: The aims, methods and experiments are clear, while the appendix is also very detailed.

Originality: The originality of the work is good, although there has been several efforts towards combining control theories with learning-based methods for improving sample efficiency and scalability.

Significance of the work: Overall, the significance could be further improved if the experiments were deployed and the choice of task environments further motivated.

Main Strengths: The main strengths of the paper are the clarity and logical way the paper is presented, the fact that they clearly outline what KODex aims to achieve in comparison to other related works and the experimental evaluation is very comprehensive.

Weaknesses: The main weaknesses of the paper are the fact that the experiments were carried out in simulation and I am not sure these would be easily deployed in the real world (although this is addressed in the limitations section).
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I would recommend this paper to be accepted, following some further thoughts/maybe preliminary experiments on using robot hardware and addressing the generalisation of the methods across different tasks (e.g. training on a task, testing on another similar task).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a technique for modeling the dynamics of dexterous manipulation problems as linear dynamics in a lifted high dimensional space, using koopman operator theory. They then show the ability to learn this koopman operator from data as a means to represent linear dynamics in this lifted space. They then use this learned linear dynamics to generate reference trajectories and learn an inverse dynamics model with an MLP to track the generated reference trajectories. 

review: Overall the idea of using koopman operators for modeling linear dynamics is reasonable, but is not new in MBRL, as the many citations in this paper also represent. I think it’s still worthwhile to understand how well this can work in the context of dexterous manipulation though. 

The part I find most confusing about this paper in the way it is written right now is - where is the Koopman operator K actually used? If it’s simply an imitation learning framework, why do we need the Koopman operator at all? There seems to be a missing section on how the K is used to generate the reference trajectory - via LQR or something else? If it’s imitation learning and the trajectories are given to you then why not just directly track those? And if it’s about generating reference trajectories in new initial states, then is it just rolling out the Koopman operator forward since the actions are implicit in the dynamics? I suspect it is this, but this should be made very clear in the paper. 

The basic premise of this work seems to be the fact that learning Koopman operators give you an easy way to roll out the trajectories in new scenarios, and these can be tracked with inverse dynamics. I think this should be compared with standard BC / IRL type methods on this task, which may work quite well. 

The claim is made that the other techniques need things like gradient descent etc, which is true for NN policies, but there are other structured policies that could be used like differentiable MPC - or like guided policy search which can be extremely data efficient. Those might be more reasonable baselines to compare to. 

Also the choice of \phi seems very important or at least I’d expect it to be. Can the authors provide intuition on why this doesn’t matter?

“While counter-intuitive, this 241 observation follows the recent finding that imitation error and task success rate might not necessarily 242 be correlated” —> can more be said about this?

I think the demonstration is reasonable - but given the somewhat limited novelty in directly learning koopman dynamics and using it for imitation learning, a deeper empirical analysis - more comparisons, more domains (eg from PDDM (nagabandi et al)), and understanding the failure modes more clearly would be very helpful. 


Strengths: 
1. Simple method, seems like it has strong empirical performance. 
2. Big improvement in sample efficiency and in robustness which is powerful for imitation learning settings. 

Weaknesses: 
1. More comparisons would be helpful, especially to techniques like guided policy search or differentiable MPC. 
2. A deeper understanding of failure modes would be helpful in understanding the tradeoffs of using Koopman versus other policy classes for imitation. 
3. An analysis across some more domains (eg PDDM) would help add to the paper. 
4. A comparison with standard imitation methods like IRL/BC would also be helpful. 

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall i like the simplicity and effectiveness of the approach, but I have suggested some additional empiricism that may help strengthen the paper. If these points are addressed, I'm willing to raise my score.
review_title: null

Meta-review: The paper presents a method for learning manipulation tasks using Koopman operators and leveraging expert demonstrations.

Reviewers appreciated the clarity of presentation, including Koopman operator theory, results showing competitive performance with lower training time,

Reviewers raised concerns about the limitations and future work sections, missing descriptions of the overall pipeline, missing details on learned dynamics, needing more discussion relative to RL/BC/IRL, and lack of real-world demonstration.

The reviewers appreciated the improvements made in the rebuttal but did not raise their assessment further. The paper remains an accept.
Decision: Accept (Oral)