Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes to separately develop a world model for the robot physics state and the environmental state in the context of model-based RL. The method is particularly developed on the backbone of recurrent state-space model, and case studies robot navigation tasks demonstrate the efficacy of the proposed scheme. 

The proposed idea looks interesting and technically sound, and the paper is relatively well-written. However, the contribution of the paper needs to be better clarified, as the baseline idea of separating the physics state and the environment state is not new. Traditional model-based approach actually all rely on this type of separation and it may be the artifact of the effort to try to learn everything in a data-driven way. In this sense, a more fundamental motivation of this work needs to be provided other than the state-of-the-art RL is not working very well. 
strengths_and_weaknesses: Strengths
- The idea is convincing and makes physical senses. 
- The paper is relatively well-written and well-organized. 

Weaknesses:
- The fundamental motivation is not very clear. 
- The contribution needs to be better clarified. 
summary_of_recommendation: This paper presents a technically sound idea that is tested on a reasonable case study. 
issues: - There are some typos. 
- Motivation and contributions need to be better clarified.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work performs model based reinforcement learning with a factored latent space, where the environment and robot dynamics are factored into separate components. For the environment component, the forward model uses proprioceptive values instead of agent actions for transitions and representation learning. For the robot dynamics, the robot dynamics ignore environment information for representation and transition values, and is able to reconstruct the proprioceptive values used by the environment model. Model based reinforcement learning follows the pattern of Dreamer, which uses imagined trajectories from rolled out latent states to predict performance. 

strengths_and_weaknesses: Strengths:
This work provides a clear, easy to implement system for incorporating robot information without needing too much explicit encoding of information.

The proposal of transfer is very interesting, and the assessment is describes fairly clearly. However, transfer on real robots would have been nice.

The experiments are interesting because they propose robot learning in the middle of a robotics pipeline, (requiring an EKF, noisy GPS), with real world evaluation. This is a much more realistic scenario than many of the other works based on Dreamer have, and demonstration in this way is encouraging.

Weaknesses:

By not including environment information in the robot dynamics, this means that any effect of the environment on the movement of the robot will be encoded as stochasticity (at best). However, the environment can often have a significant effect on the dynamics of the robot. For example, if the robot arm picks up a weight, and this changes the dynamics of the robot, or there is a moving barrier. 

Model based reinforcement learning methods, and other reinforcement learning methods, often explicitly encode a separate component of the model for the proprioceptive components of the state. This method is most likely unique in incorporating proprioceptive state into the Dreamer-like framework, but this is a relatively marginal difference from a normal network, hard-coding in robot dynamics, or other methods already being used to include this information. 

A component that is often under-appreciated in model based reinforcement learning methods is how to control the distribution of rollout data used to train the model, while the policy might be performing undesirable behavior. This is not particularly well addressed in this work as well, as it describes only a training schedule where the dynamics are updated less. By the nature of the experimental domain, this is probably not stressed, since dynamics in a boat domain do not often significantly change as long as the vehicle does not run aground.

A comparison with a dynamics method that also incorporates the robot information, though in a different way (including it in a larger neural network but a separate channel, including an explicit hand-designed model), would have probably provided a better comparison than Dreamer, since by separating out the robot this work adds additional information that Dreamer does not have, so comparison is limited. 

While the boat domain is realistic and interesting, as the only experimental domain (though with simulated versions), it does lack some desirable assessment capabilities. In particular, in this domain the effect of the environment on the agent is minimized, since the effect of the environment on movement of the boat can often be overcome relatively easily. In addition, the dynamics of the agent are intuitive, making it easier to set hyperparameters and balance exploration. It would be desirable to have a robotics domain involving a more complex proprioceptive system. 

summary_of_recommendation: I propose a weak accept, mostly because the work only provides marginal originality, and does not assess on a robotics domain which would have provided more dynamics learning challenge (such as robot manipulation). However, the work provides some interesting results relatively clearly along with the proposed algorithm

issues: Experiments on some domain that stresses a variety of skills would be preferable, such as a robot manipulation domain.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper introduces a new extension of a Dreaming, model-based neural network, in which policies are trained by creating synthetic, ie dreamed or imagined, trajectories in a latent state embedding for additional training data.
The contribution of the paper is to split the latent embeddings, and latent dynamics functions into two: One models the environmental transition function while the second models the dynamics of the intrinsic states of an agent.
Both models are then used to unroll two latent trajectories into the future: One predicting changes in the environment, the second changes to the robot or agents own state.
The benefit of this approach is that in principle, either the networks modelling the environmental dynamics or the networks modelling the agent dynamics can be replaced or re-trained. This would allow in principle to speed up the training process of a new agent, or in a new environment, or allow the possibility to transfer to new combinations.
For example, if agent A is trained in environment 1, and agent B in environment 2, one would be able to exchange the learned networks and transfer zero-shot to a combination of, for example, agent A and environment 2.

The paper provides some evidence that this is possible by using the proposed approach to train the environmental and agent dynamics of an autonomous boat and a vehicle. In evaluations, the paper is able to demonstrate the ability to transfer knowledge in a limited experiment in which the environmental dynamics experienced by the boat and the agent dynamics learned from the vehicle.
strengths_and_weaknesses: Strengths:
- The general idea of the learning algorithm presented in the paper is interesting and well motivated.
- Although not all presented results indicate that the proposed approach is able to outperform the baseline, ie Dreamer, there is some empirical evidence that the algorithm is working and would be applicable to a range of different scenarios when finetuned (see below)
- The experimental setup is well chosen, and a real-world experiment is conducted in which a real unmanned ship is steered along the shore of a lake.
- The authors promise to publish the code on Github and make it available to the wider research community under an open-source license.


Weaknesses:
- The paper was hard to follow at times. For example, for the dynamics models, the states are first introduced as s^dyn and s^env, later they are changed to greek letters, as well as x and o. Figure 1 uses an alternative notation, with phi, p and h. The notion should be harmonized in the final version.
- The appendix has empty sections.
- The proposed approach works only as intended if the internal robot variables are (a) are of the same type and order, (b) include exclusively state variables that are agnostic to the robot type and model, for example, its world position and velocity, and (c) the state dimensions are the same between different types of robots. The approach works well in the paper because the internal state of the agent contains only the GPS coordinates (position) and task-space velocities and accelerations produced by the onboard IMU. I  believe the proposed approach would have difficulties if the agent state, ie x_t, would also contain actuator states, for example. In such a case, the environmental transition function would also depend on the concrete type of agent - eg in the case of turbines versus wheels. This can be solved in principle by using only a suitable subset of the agent's internal state. This problem should be discussed in the paper to broaden its applicability.
- Environmental and dynamic models are pretrained in simulation. It would have been more interesting if the models were trained online during the learning process.

Others:
- A similar approach in which agent dynamics and environment dynamics are separated from each other, based on graph neural networks, is presented in [1]. This work could be discussed in the related works section 

[1] Sanchez-Gonzalez, Alvaro, et al. "Graph networks as learnable physics engines for inference and control." International Conference on Machine Learning. PMLR, 2018.
summary_of_recommendation: Although the paper has some weaknesses in its writeup, which could be improved, and not all experiments indicate an outstanding performance of the proposed approach, the presented method and its demonstration on a real-world USV should be interesting for the wider robot learning community. The idea of separating the environmental and agent dynamics in a Dreamer-like architecture seems interesting, of value, and well-motivated, especially to allow zero-shot transfer between different agent and environmental combinations.
I am also confident that the authors can address the outstanding issues for their final draft, especially regarding clarity and notation. 
It might have been beneficial to apply the proposed method on a few more toy problems in simulation, such as mujoco or pybullet, to demonstrate further its applicability and performance.
issues: - The notation in the introduction, figures and the methods section should be the same.
- The mentioned problem of having different dimensionalities between agents or having additional internal states, such as motor states, should be discussed.
- Ideally, the appendix should give a more detailed explanation of the learning setup, especially regarding the number of states for each task, their dimensionality and how they are defined.
- Appendix D does not exist - this also means that the details reg. the curriculum training process is missing. These details need to be added for the updated version.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a novel algorithm for robot-environment decomposition in MBRL, decomposing the prioprioceptive and exteroceptive modeling. Specifically, the method is based on Dreamer, and learns two RSSM models, one to predict next robot state given action, and another one to predict next visual (or in this case LIDAR) observation given the robot state. It is shown that this decomposition improves robustness to distributional shift, including the sim-to-real shift, with respect to a monolithic Dreamer model. Further, it can be used to adapt to new robots by composing an environment model of target environment with a robot model of a source robot. Experiments on ground and water surface navigation are provided in simulation and real world. 

strengths_and_weaknesses: Strengths:
- The algorithm is novel, sound, and likely will be used by robotics researchers
- The paper is well-written
- The experiments are extensive and clearly demonstrate the benefits of the proposed method

Weaknesses:
- The experiments are perhaps a little unsatisfying. First, the paper does not analyze how exactly the proposed method improves robustness. There are no provided ablations.
- Second, it is unclear why it is necessary to use Dreamer for the proposed use-case. Since the method assumes access to a simulator, a policy can be trained on the simulator rollouts directly using an on-policy algorithm. Similarly, since the simulator is available for all robots, it is easy to just train on the target robot and environment instead of performing transfer.

summary_of_recommendation: Overall, the strengths of the paper outweigh the weaknesses. The paper proposes a promising algorithm and provides sufficient experiments to validate the benefits of the proposed idea. The algorithm is relevant and is likely to be used by other researchers.
issues: See comments on Weaknesses. Further, there are the following writing issues:
- l30: Finn’17 should be cited together with [9]
- Decomposition on l41 is quite intriguing. However, as far as I can see it is not used anywhere else in the paper. If the proposed method indeed decomposes the dynamics as on l41, perhaps this can be derived explicitly.
- l196 the reference is broken

Finn’17, Deep visual foresight for planning robot motion
review_title: null

Meta-review: This work claims an idea by decomposing dynamic model into physical agent model and environmental model, which is straightforward but also promising. Authors also demonstrate this kind of decomposition can contribute to transfer efficiently among different agent sharing the same environments. Reviewers have consistent comments that the idea claimed in this work is convincing and also conducted with sufficient experiments overall. However, there are still some concerns raised by reviewer as follows.  
1. The contribution and novelty compared to the existing dynamic decomposing work should be more clearly presented. 
2. The representation of physical dynamic model sounds like too simple, which would be limited to other more complicated agents. 
3.  Experimental setting compared to Dreamer is a little bit unfair. 
4. Some notions are not unified. 

Although there are still some limitations existed in this work, most of reviewers thought the majority of the contributions have been clearly clarified with consistent opinion to recommend to accept it as CoRL paper. 
Decision: Accept (Oral)