Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a novel task-oriented approach using a contrastive encoder for Koopman-based control of robotic systems.  

They employ an end-to-end reinforcement learning (RL) framework, simultaneously learning the Koopman model and its associated linear controller over latent space within a single-stage loop. The primary objective is to minimize the RL task cost, with model prediction error minimization as an auxiliary objective. 

It use a contrastive encoder to learn the linear latent representation of the nonlinear system. They design the Koopman controller with a differentiable LQR solution process, allowing gradient optimization within the end-to-end RL framework for controller parameter optimization. 

The paper presents empirical evaluations through simulations on various tasks and is compared with two-stage Koopman-based control and pure RL methods, offering a comprehensive assessment of its effectiveness.
review: The idea of applying koopman control for RL is not original, but the author propose to use RL simultaneously learn the Koopman model and its associated linear controller  instead  of only focusing on the accuracy of the dynamic model. The idea which is simple and effective that makes the model-based controller robust to the dynamic model's ion.

advantages:
- Improves the model-oriented koopman control method's performance. The controller is robust to the learned dynamic model's prediction error. 

weakeness:
- The experiment are evaluated on a limited selections of tasks, only 2 simple deepmind control suite tasks and one imaged-based cartpole control task. The dynamics and action dimensions are relatively simple in these tasks.

- Lacks detailed analysis and comparison experiments. For example, this work does not compare the method with general model-based RLs (e.g. dreamver, etc). It neither presents the sample efficiency as it is important for model-based RLs.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The idea of implementing task performance oriented koopman control is intersting and has certain novelty. However, the evaluation tasks are relatively simple and lacks comparison experiments and analysis. I would recommend a weak accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents a method for learning control policies based on task-oriented Koopman operators. The main idea being that prioritizing Koopman operator model fidelity around the task has higher performance than simply emphasizing prediction loss of the Koopman operator. The main result would be the ability to construct linear control based on the Koopman operator around the task rather than over the whole state-space. The approach uses a contrastive encoder to construct the Koopman function embedding and integrates an LQR controller learning into the learning loop.
review: The paper presents a novel approach to the problem of model-based control using Koopman operators. The concept of the contrastive encoder to model the Koopman embedding function is quite interesting and the interpretability of Koopman operator oriented around tasks rather than prediction error is an approach that has valuable merit in this area. 

A major strength of this work is enabling control-based interpretability of learned RL systems. This is often not a property of existing methods and should be highlighted. The ability to further adapt the LQR control problem in the loop is quite novel as a big issue with Koopman control is the ability to appropriately tune controllers based in the embedded space. 

However, one key weakness of the work is motivating analysis and experimental validation as to why the contrastive embedding is preferred over other canonical choices (e.g., normal MLP prediction loss). In fact one can perhaps claim that the true benefit is the task-oriented learning achieved with SAC that is key to ensuring the algorithm works well. As an improvement it is key that these kinds of choices have obvious empirical and theoretical backing, otherwise the choice of this approach seems arbitrary. It would benefit the paper to have such evidence that this is the right approach and what the expected improvements are so that one can validate the hypothesis experimentally. 

Another minor weakness that can be readily addressed is the lack of an ablation study on the parameters e.g., prediction time T, Koopman embedding dimension, etc. These are valuable as it strengthens the reproducibility of the work and highlights limitations of the method and future directions.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The work demonstrates a promising direction within robot learning that should be highlighted. However, the current work lacks sufficient robotics implementation. As such a weak accept is recommended as the work has potential, but requires more thorough robotics results and analysis to make a proper impact.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This work proposes a novel method for task-oriented koopman based control. The proposed method utilises end-to-end reinforcement learning and contrastive encoder to simultaneously learn the Koopman latent embedding operator and associated linear controller. Overall, the paper is written very well but it is evaluated purely in simulation. Having said that, the provided evaluation and analysis is very comprehensive and overall pleasant to read. The proposed method has the potential to be scaled to physical systems and therefore I recommend it for acceptance. A disclaimer here is that I am not an expert on Koopman based control.
review: ## Strengths and Weaknesses
### Strengths 
- well written and evaluated paper
- an interesting idea
- good results including on pixel based tasks

### Weaknesses
- no real world evaluation
- no clear indication for such attempts
- potentially very difficult to scale to complex physical tasks, e.g. like soft object manipulation
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall I enjoyed reading this paper. I did not like that it does not have any physical world experiments but regardless I see the proposed approach to have potential for being impactful.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a Model Based RL approach based on Koopman control.The paper proposes a contrastive learning approach to learn the koopman embedding as well as the linear dynamical system within the latent space. this setup can then allow to use classical control techniques to solve the control problem. The paper is clearly written and easy to understand the different parts of the paper and the algorithms. The experimental evaluation is fully in simulation on fairly simple control suite tasks. The results show that the proposed approach reaches near optimum performance on the tasks as well as decent predicting of the state space.
review: the paper is clearly written and easy to follow and understand the algorithmic setup. I also think the approach is interesting and to a certain degree novel in the sense that it stitches together different components coming from optimal control and ML. The end to end approach is interesting especially also the structure around learning the contrastive embedding and linearising the system this way. However the evolution of the method is not mature enough yet for publication. besides of the evaluation being purely in simulation and on fairly easy tasks the paper also lacks comparison to more baselines. Other MBRL baselines, especially ones that combine learning approaches with optimal control should be compared against, it would be interesting to compare in terms of actual task performance to see whether there is improved performance and also whether there is a benefit in sample efficiency.  The paper only compares to total control cost which is a bit harder to interpret in terms of how useful the approach is.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: given the lack of more realistic experimental evaluation I believe this paper is not ready for publication at CoRL yet.
review_title: null

Meta-review: This paper presents a task-oriented koopman based control that features end-to-end RL and conrastive encoder to learn the Koopman latent embedding oeperator. 

The reviewer found that that the paper presents an interesting and novel ideas in a well-presented manner. The evaluation part is pointed out relatively weak, but the rebuttal better clarifies on this aspect.
Decision: Accept (Oral)