Learning Behaviors through Physics-driven Latent Imagination
Authors: Antoine Richard
Stephanie ARAVECCHIA
Matthieu Geist
Cédric Pradalier
Venue: CoRL2021 Oral (2021)
Keywords: Model-Based Reinforcement Learning
Field Robotics
Latent Models

Abstract: Model-based reinforcement learning (MBRL) consists in learning a so-called world model, a representation of the environment through interactions with it, then use it to train an agent. This approach is particularly interesting in the con-text of field robotics, as it alleviates the need to train online, and reduces the risks inherent to directly training agents on real robots. Generally, in such approaches, the world encompasses both the part related to the robot itself and the rest of the environment. We argue that decoupling the environment representation (for example, images or laser scans) from the dynamics of the physical system (that is, the robot and its physical state) can increase the flexibility of world models and open doors to greater robustness. In this paper, we apply this concept to a strong latent-agent, Dreamer. We then showcase the increased flexibility by transferring the environment part of the world model from one robot (a boat) to another (a rover), simply by adapting the physical model in the imagination. We additionally demonstrate the robustness of our method through real-world experiments on a boat.

Learning Behaviors through Physics-driven Latent Imagination
Antoine Richard
Stéphanie Aravecchia IRI. 2958 GT-CNRS, Metz, France
Georgia Institute of Technology GA, USA antoine. rwchard@gatecc..eett
Matthieu Geist Google Research, Brain Team
Cédric Pradalier IRL 2958 GT-CNRS, Metz, France
Model-based reinforcement learning (MBRL) consists in learning a so- called world model, a representation of the environment through interactions with it, then use it to train an agent. This approach is particularly interesting in the con- text of field robotics, as it alleviates the need to train online, and reduces the risks inherent to directly training agents on real robots. Generally, in such approaches, the world encompasses both the part related to the robot itself and the rest of the vironment. We argue that decoupling the environment representation (for example, images or laser scans) from the dynamics of the physical system (that is, the robot and its physical state) can increase the flexibility of world models and open doors greater robustness. In this paper, we apply this concept to a strong latent-agent, Dreamer. We then showcase the increased flexibility by transferring the environ- ment part of the world model from one robot (a boat) to another (a rover), simply by adapting the physical model in the imagination. We additionally demonstrate the robustness our method through real-works experiments on boat.
Abstract:
Keywords: Masel Reinforcement Learning, Field Robotics, Latent Models
Introduction
Early forms of controllers, such as classic optimal control or dynamic programming, make use of a supposedly known model of the environment. On the learning side, Reinforcement Learning (RL) can also learn model and use it to train an agent. It is only during the last decade that Model-Based RL (MBRL) algorithms capable of controlling systems using high dimension inputs, such as raw images, were developed. These advances allowed their use on robots for solving complex tasks, on which traditional methods failed [1, 2]. In robotics, it is of paramount importance to learn quickly, as acquiring samples is not only ludicrously time-consuming. but can also be dangerous for both the robot and the operator overseeing the agent. Hence, due to their high sample efficiency and their potential resort to offline learning [3, 4], world models make for compelling choice in robotics applications.
Yet, because most of the agents are meant be used on classic benchmarks such as Atari [5], OpenAI Gym [6] or the DecpMind control suite [7], they process either images or low proprioceptive variables (for example, position or speed), but rarely both. In robotics, most of the works that used MBRL for solving nigh-dimensional tasks discard the proprioceptive variables [1, 2, 8), even though they could have been acquired fairly easily. Other works, such as [9, 10]. make use of both, but embed them as single model. Unfortunately, with neural networks there is no way of accessing the dynamics of the system directly integrated inside the environment because they make for a single black-box model. All in all, this means that the world model learned by an agent will be unique to this system.
By contrast, we argue that a world-mode should be made of two components: an environment, which encompasses everything that the agent can sense using laser-scanners or cameras, and the dynamics, iec the set of physical equations and variables that rule the movement of the mobile robot in the world. To do so, within the paradigm of Markov Decision Processes (MDPs), we decompose the total state of the system in two parts, one related to the environment and one to the dynamics of the
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

robot, Stat = (St as a change of the dynamics in response to the applied action, and change of the environment in | sy A concrete example of why such representations are powerful is to think of car manufacturers, All manufacturers make different cars with different physical properties, yet those cars all share the same environment: the road. Hence, a method like ours could allow the cars to share their environment while having their own dynamics, speeding up the training process and potentially allowing the distributed training of heterogeneous agents. Furthermore, it makes sense in robotics applications use proprioceptive information, as most real robotic tasks involve low level physical constraints coupled to high level perception constraints. In the end, MBRL is very promising solution bring more autonomy to real-robots, yet we think it would be better suited to mobile robotics application if it was to natively account for the robots' dynamic.
S. ( ), and we assume that the transition from state to state can be decomposed
response to the new dynamics, 1[ ST at) = P( |S at
In this paper, we modify one of the strongest latent-model agent, Dreamer [1I], and integrate the dynamics an independent part of the world model. As such, our world model will feature two states: an environment state that uses low dimensions proprioceptive variables to transition from one state to another (P(S St (t-1)), and physical state, that uses the action sent by the agent to transition from one state another (P(s | ..)). While it might seem fairly similar to the original world model with an observation of the proprioceptive input in addition to the images, it not. Decoupling the dynamics from the environment allows for interesting manipulations within the imagination process used to learn the actor. Because the dynamics is no longer attached to the environment, we can now swap the dynamic model of a robot for the dynamic model of another robot as long as their proprioceptive states are of the same size.. Moreover, our method ensures that the imagination roll-out have environment-ssates that are consistent with the physical-states. Eventually, we can use an analytic approximation of the robot dynamic model to alleviate the need to rely on learned model.
Our contributions can be summarized as follows: (1) we propose a latent-state representation that decouples the environment from the dynamics to better fit mobile robots; (2) we test this method in simulation and on a real robot and show that it robustifies the agents compared to the baseline; and (3) we demonstrate that this method can also be used to transfer an environment from one robot to another in zero-shot setup. All the code to reproduce those experiments is open-source and available on github (Appx. D.1).
2 Related Work
Model-Based Reinforcement Learning. The earliest form of Model-Based Reinforcement Learn- ing (MBRL) is probably optimal control. In optimal control, the model of the system is typically known, and optimal controller is computed using methods such as LQR. Until recently MBRL al- gorithms such as Pilco [12], among others [13], were designed to process and model low dimensional inputs. These methods were applied to robotic applications [14, 15] with impressive results, but they were limited and struggled to control systems using high dimension sensory inputs such as images. However, with the advent of deep-neural-netweee inputs, as shown in the comprehensive survey of [16]. As of today, MBRL performances are nearing if not exceeding the performances of model-free agents [11, 17]. The main advantage of MBRL relies on its higher sample efficiency when compared to the model-free agents. Methods based on Variational Ento--Essders (VAE) [18], such as latent state methods [19, 11, 17], can learn robust policies thanks to the VAE stochastic states. Unfortunately, as these methods started processing higher dimension inputs, they also started to disregard proprioceptive information as made solving the task easier.
MBRL was applied successfully to high dimensions
Dreaming applied to real-world. Latent-state methods have shown impressive results when ap- plied on real robots. [8] applies Dreamer on a 1/10th racecar with a LIDAR and compares it to other model-free methods in simulation. They show that, on their task, Dreamer performs better than other model free RL algorithms, such as DDPG [20] or PPO [21]. They also demonstrate that Dreamer transfers well from simulation to real robots, in well controlled environments.
[2] also applies Dreamer on an Unmanned Surface Vessel (USV) with a LIDAR and trains it in simulation on a shore following task. They then deploy it on the real robot, without retraining it, and evaluate the robustness of the approach by both changing the dynamics of the real systems and evaluating it in different weather conditions. Both of these works also use Dreamer as their backbone, but unlike

---

(a) Learn dynamics
(b) Learn behavior in imagination
(c) Play
Figure I:
The three concepts of the proposed extension of Dreamer: (a) Using its replay buffer, the agent learns to encode environment high-dimension sensory observation and proprioceptive input into a compact latent environment state Additionally, using a similar process, the agent learns to encode proprioceptive inputs and actions into a latent physical state Both of these states are then used estimate the reward
(b) Using both learned latent spaces, the agent predicts state values (T) and actions ()) as in the original Dreamer. (c) The agent observes its environment and predicts the best action. More details about the algorithms used here can be found in Appx. C.
ours, they completely disregard the physical state of the system. [1] shows an application of latent models to a real world system using cameras. In [9], they rely latent state model and integrate both the physical state of the system and high level distance sensors to move their robot around. However, they still consider the systems dynamics
be part of the whole model. A work that is closer to ours is [22], in which they learn from images the Lagrangian dynamic of system, and then use reconstruct the observed images. In its current state, [22] is not applicable to real robots: the examples are simplistic, and make strong approximations about the dynamic model. [23] also shares similarities with the work presented here: they use Graph Neural Networks to perform system identification as in [24], and then apply a Model Predictive Controller and RL on the predicted physical states. The main difference with our work, is that we don't 't only have a dynamic module but also a perception module. In our method, the outputs of the dynamic module are used to predict the next perception state, and both of their latent states are then used by the actor to decide on which action to apply next.
3 Method
We now show how our model differs from the original implementation of Dreamer[11] []], which we started from. As in Dreamer, we solve Partially Observable Markov Decision Process (POMDP), with discrete time steps I, for which we have continuous actions at, high dimension sensory inputs o,, low dimension proprioceptive inputs 2t, and rewards T generated by the environment.
Similarly to Dreamer, our version articulated around three main concepts: the learning of the world model from previous experiences (Fig. 1a), the learning of the behavior inside the imagination process (Fig. lb), and finally the application of the policy using observations from the environment to collect new samples (Fig. Ic).
Unlike the original Dreamer, our agent does not rely on a single latent state. Instead, we consider that this single state can be decomposed into two states S!m" (St ( sdyn), where P(S 1 .x) = P( 1St aa, P(STT " In practice, we consider that we can learn function qa ( this means that can further decompose P(S |Stot .a)) P(S S (4p) ((Spav |S xt) as we will be able to estimate &; from S within the imagi- nation. In [11, 17], the latent dynamic model is composed of three modules: representation module (s, St-1, G1-1, 0r), a transition module (st St-1, at-1) and a reward module (r, st). With this setup, both the representation and transition modules embed the dynamics of the system and the understanding of the environment. This limits the capacities of the world model. Hence, we propose to separate the dynamics of the system from the environment. To do so, we change the latent dynamic model of Dreamer to include two states: Seav the environmental state, and S 5ohyn the physical state. We then add two extra modules and slightly modify the inputs of the original latent dynamic model.
3

---

The new latent dynamic model is
Dynamics representation module:
(o
Sdyn 04-1:
Dynamics transition module:
(
Sdhn 04-1)
Environment representation module:
(S
Sz1 (t
(1)
Environment transition module:
On (S
Sene T1.1
Reward module:
(r St S
Learning dynamics & Reconstructing Observations. To learn both the dynamics and the envi- ronment, we rely on Recurrent State Space Models (RSSM) [19]. RSSMS can be seen as non-linear Bayesian filter [25], with observation step and a prediction step. During the observation step, the RSSM is given its previous state and an observation, and outputs compact latent state. During the prediction step, the RSSM is given its previous state and command, and outputs compact latent state. This prediction step of the RSSM is the transition function, and the combination of a prediction and observation the representation module.
Tall WI"
Figure RSSM reconstruction of proprioceptive variables, 5 observations followed by 45 predictions.
We consider two RSSMs: one learns the dynamics of the system and the other learns the environment.
The RSSM that learns the dynamics takes as input both the raw proprioceptive variables and the actions. To teach the network to embed within its latent state relevant information, we also train decoder reconstruct the proprioceptive variables associated with that state. Fig. shows examples of reconstructed states.
Dynamics observation module:
2 (S
saam xz)
Dynamics transition module:
2 (S ym
Sddn ae.1
Dynamic reconstruction module:
( Sdym)
The RSSM that learns the environment takes as input high dimension sensory observation and the latent dynamic state. We then train another network to reconstruct from its latent state the inputs or projection of it.
Environment observation module:
4n (n
cenv 0t)
Environment transition module:
4n (Spm St11 .T-1)
Environment reconstruction module:
4n (Ōe Sere)).
Using these two latent states, Stnm and Stym we also learn predict the reward that will be given by the environment (r, Sume ..dy)). , ). Regarding the optimization these networks, we use the same method as Dreamer [11, Sec. 4] with a caveat: the dynamics is not trained jointly with the environment and the reward. To train the dynamics, we propose two options. In the first option, do not learn the dynamic model. For instance, one could use an already learned dynamic model and keep it fixed for the whole of the training. This is interesting as the physical model remains constant throughout the training. It helps learning the value and the reward model faster than if we were to learn it from scratch. Also, offers the possibility to completely remove the learning and neural network components from the dynamic estimation, by using an analytical model, for instance. In roboties, this makes sense, as good analytical/learned models are often already known. If we don't. know the dynamics, the second option is leamn discover environment. However, to ease

---

the optimization of the actor, the reward, and the value, we do not train the dynamics at the same rate as the rest of our agent's 's neural networks. We train the dynamics less often, but for more steps than the other components. While this technique does not offer as much stability as the first method, it still increases the overall stability of the learning process.
Learning behaviors. With the learning of the system dynamics and environment covered, we can now them in the imagination process. The imagination generates imagined trajectories from which the actor is learned. In the original Dreamer, imagined trajectories start from latent states obtained from actual observations, and follow the predictions of the transition model 8r q(-18--1, (,-1) and the policy a, ~ q(-1s7) In our case, we use the dynamic transition model as a physic engine for the imagination, and the environment transition model as the rendering engine. This means that to imagine a trajectory, we start from a pair of latent states (Stmu, S:im) obtained from actual observations, and iteratively apply the transitions models given in Eqs. 2 and 3, the policy being given by a- ~ q(-|Stmee SHum). As this process goes along, the proprioceptive variables 2, are reconstructed from the physical state Sdin using (2 | 5. They then used inside the environment transition model. An important note is that, since RSSMs are VAEs, if we sample from the stochastic state to reconstruct the physics, then the imagined proprioceptive variables will be noisy. To prevent that, we take the mode of the distribution which generates smooth physical predictions. Once have set of trajectories, them to estimate the reward and value associated with them. Using these, we train the policy and the value estimator using the same technique as the original Dreamer [11, Sec. 3].
can get each pair of states (5 (St and use
Environment Transfer. Unlike Dreamer, our method allows transferring environments from one robot to another. In this context, we consider that both robots share the same sensors and same spate-space. We also consider that the environments in which the robots are going to evolve are similar. Let us consider robot A, with already known dynamics modules p (S Sdlym ,4t-1, ,41,,,T)),
98 (S sdm a..1). Let us consider robot B, with already known environment modules / (St SET" 1. ot), p (Stav) StT" 1x,1), q (rx Stmv sddm)), and a pre-collected set of samples of robot B interacting with its environment. To learn the actor of robot A, using the environment of robot B, we learn entirely offline the actor, the reward, and the value, using the set of experiences collected on robot B. It should be noted that we would not need to retrain the reward if it was rewritten as o (r: S .zr). The main disadvantage of this method is that because we are using experiences collected on robot B, the observation of the dynamics will match the dynamics of robot B and not the one of robot A. However, as the imagination progresses the dynamics will match the one of robot A. Another issue can arise if the robots are radically different. In this case, the samples collected on robot B will most likely not explore the state-space of robot A enough to control it perfectly. Nonetheless, as we will demonstrate later, it allows learning good preliminary policies to further refine the agent from. The exact algorithm used to learn the agent policy is given in Appx. C. Besides, the state-space of both robots must be the same, which makes this method mostly aimed towards mobile robots. Examples of the state-space of our robots are given in Appx. D.2.
4 Robots, Task & Evaluation
Robots. To evaluate our approach we use two robots (Appx. A.I, Appx. A.2): an Unmanned Surface Vehicle (USV) and an Unmanned Ground Vehicle (UGV). The USV is a Clearpath Hcron, a small catamaran with a turbine in each hull. There are two action dimensions: the first element controls the left turbine, the second element controls the right one. This means that to drive in straight line, the agent must send the same value to both turbines. The UGV is a Clearpath Husky, a skid-steer ground robot controlled with twist input. The size of the action of the Husky is 2, but unlike the Heron, the first element controls the forward velocity, while the second controls the angular velocity. Both robots share similar sensors: 2D laser-scanner, an RTK GPS and 9DoF IMU. To prevent damaging our robots' actuators, we saturate their accelcration before applying the output of the policy. More information regarding the exact specifications of the USV and UGV can be found in Appx. A. and A.2. teach our agent to solve a sensor-based navigation task: the robots must follow the shore of lake fixed distance while maintaining set forward velocity.Hence the reward generated by the environment is made of two components: low-level constraint, the velocity. and high level constraint, the distance from the shore.
5

---

Vinbor
Task. The exact description of the reward is available in Appx. B. Fig. 3 illustrates the USV solving its task.
To fulfil their tasks, the robots rely on a 2D laser-scanner and on their own velocity, estimated from the fusion of the RTK GPS and the IMU through an Extended Kalman Filter (EKF). Using the EKF, we provide to the agent three proprioceptive variables: its forward- velocity, its lateral-velocity, and its angular-velocity. The action space of both robots is [-1.1]
Med eelyy
Figure 3: USV/task.
Evaluation. We test our method both in simulation and on real USV. In simulation, to evaluate the performance of our method, we used the different components of the reward as metrics. Each model ran for two hours and at the end of each run we collected the standard deviation, the mean, the 10% quantile and 90% quantile of the rewards. In the real world, we cannot compute the distance reward accurately: our estimation of the distance to the shore is uncertain because of the high level of noise of the laser data were carried out in late spring, where the luxurious vegetation makes laser-scans highly unreliable. For this reason, we will not evaluating the agent distance from the shore, but we will visually inspect the followed trajectories. The training settings are given in Appx. D
natural environment. This is particularly true because the experiments
5 Simulation Results
We compare our method to the original version of Dreamer, and illustrate the benefits of having a separated state for the dynamics. All models trained with approach have been trained with fixed dynamic model. This means that the dynamics was learned ahead of time and could not be refined during the training process. This was done to show that could have used analytical models instead.
Unmanned Surface Vehicle. To show the benefits of using a separated state for the dynamics, we a benchmark using the USV. To verify that the agents perform optimally, we first run a test where they are deployed in their original training environment. Then, to evaluate how the agents behave when the dynamics change, multiply the simulated robots damping factor by two. Increasing this factor makes the robots' dynamics softer: for the same inputs, the acceleration will be slower. To test the ro- bustness to perturbations the dynamics, we also run water current with speed of 0.4m/s Finally, to further stress the agents, we multiply the damping factor by two and add 0.4m/s of current. The results of this benchmark can be found in Fig. 4.
with the original dynamics and constant
Our method is more robust than Dreamer to both dynamics change and perturbations. Most notably, it maintains correct velocity reward across the whole benchmark, when Dreamer fails soon the dynamics changes. This suggests that images are not sufficient to approximate the velocity correctly, and that by using our separated physical state, the actor is able adapt its behavior match the desired velocity, despite having never seen such events in the training. Moreover, having access to a physical state seems to help solve the high-level constraints,
method outperforms the vanilla
Ideal
Damping x2
Current 0.4 mis
Current Damping
Velocity Reward
1.0
1.0
1.0
0.8
0.5
0.9
0.6
6.0
0.8
ours
dreamer
0075
dreamer
dreamer
O0TS
dreamer
Distance Reward
2.5
2.5
2.4
2.0
2.2
1.5
-2.5
-5
-7.5
-10
OuTS
drsamer
OUTS
dreamer
outs
dreamer
00T5
drcamer
Figure 4: Box plots illustrating the benefits of using separated state for the dynamics (ours) vs not using one (Dreamer). High value indicates better performances. Small spread indicates better consistency. Results obtained in simulation.

---

Dreamer on this part of the task too. The only exception is when the damping factor is multiplied by two. In this case, the lower velocity of the robot allows Dreamer follow the shore better. Overall, these experiments show that splitting the single latent state into two, one for the physics and one for the environment, is not only a viable concept but also increases the robustness of the agent.
USV to UGV transfer. To showcase the transfer capacities of our method, we learn an actor completely offline, using the environment the USV and the dynamics of the UGV. This is possible we separate the latent-state into those two parts, dynamics and environment. As reminder, the USV and the UGV do not react to actions in the same way. The USV is driven by sending commands to the left and right turbines, when the UGV is driven by sending forward velocity and angular velocity commands. Furthermore, the dynamics of both systems are radically different: the USV glides, has huge inertia and has significant amount of lateral velocity whereas the UGV has almost no slippage, no lateral velocity and no inertia. To train our actor, we first trained our USV to fulfill the shore following task, and then took its environment model (including the reward) along with the samples of its interaction with the simulation. We secondly learned the dynamics of the UGV from set of past interactions with the simulation. Finally, using the environment of the USV and the dynamics of the UGV, we trained the actor entirely offline. We then deployed it in simulation and compared it to an agent trained on the UGV and on the environment for an equivalent amount of training steps (300k).
Velocity Reward
Distance Reward
Agents Trajectories
1.0
P
2.5
T
online offline
moptou (m)
155
0.8
2.3
2.2
150
0.6
-15
-10
-5
offine transfer
online learning
offline transfer online learning
position (m)
(a) Velocity reward
(b) Distance reward
(c) Trajectories
Figure 5: Environment transfer results. Higher values indicate better performances, smaller spread
indicates better consistency. Results obtained in simulation.
Figs. 5a and .5b show the performance of the agent trained offline with environment transfer (offline transfer, ours) compared to the agent trained online with its own environment (Dreamer). It is worth noting that we did not include the results of the USV actor applied directly on the UGV: it completely failed to solve the task because of the complete difference in the commands mapping. From these results, we can see that the agent trained online achieves almost perfect results, with an average distance reward of 2.48. On the other hand, our agent achieves an average reward of 2.4, which amounts to about +33cm of error on the 10m distance it must keep from the shore. Fig. 5c shows the trajectory of the agents in the hardest spot of the simulation: a narrow hairpin-turn. We can that the trajectories of the agent learned online are closely packed together. On the contrary, the trajectories of the agent learned offline more loosely grouped, indicating that this policy is less reliable and most likely less overfitted to the environment. These approximate trajectories can be explained by the fact that the imagination is inaccurate at its beginning. Since the samples used to generate the starting states of the imagination are taken from the USV, these starting states contain dynamics that are not feasible on the UGV. Yet, despite these shortcomings, we demonstrated that we can casily transfer environments in a fully offline fashion even on robots with fundamentally different dynamics and action mappings.
6 Real World Results
Finally, we evaluate our method on real USV. To do so, we train two agents in simulation, one with Dreamer and one with our method, and deploy them directly on the real robot in a zero-shot setup. Thus, the dynamics learned by our agents is only approximately matching the one of the system. We evaluate the performances of both agents on a whole lap around the lake. These laps are shown in Fig. 6a, lap is about 1.6 km long and takes approximately 20 minutes. During their lap, neither of the agents collided with their environment. Regarding the trajectories, Fig. 6b shows that the trajectory followed by our method is much less winding than the one followed by Dreamer. The reason for
7

---

that seems to be that the agent using Dreamer tends to overshoot more than the agent trained using our method. This overshooting behavior can easily be spotted on the cropped image of the lake. On the middle right part of the image, we can clearly see the Dreamer agent overcompensating and navigating toward trees. This behavior cannot seen on the agent trained using our method, which follows the shoreline at much more constant distance. It further demonstrates that accessing the dynamics makes the policy naturally robust to changes in the dynamics.
As for the velocity, the right most picture of Fig. 6c shows that the agent with a physical state (ours) the closest from the desired forward velocity. Additionally, our method shows smaller spread when compared to the original Dreamer. It is interesting to note that this model can fulfil its task despite having noise on the physical inputs provided by the EKF, which something it had never encountered before. Furthermore, during the experiments, we observed that our agent was making less brutal accelerations and seemed have more fluid way of steering the boat. All in all, this confirms the simulation results shown in Fig. 4: our method is more robust and better matches the forward velocity target.
1.2
Ours
Forward Velecity (ml))
1.0
0.8
0.6
Dreamer
0.2
ours dreamer
(a) Trajectories (Full)
(b) Trajectorics (Zoom)
(c) Velocities
Figure 6: Real world experiment. First row, overhead imagery of the deployment site, with the full trajectory of the agents in yellow. Center trajectory of the agent can be seen in yellow. Last row, comparison of the forward velocities reached by the two agents. We invite the reader report Appx. E for higher resolution images. Overhead imagery from Google Earth, 2021, trajectories plotted using Google Earth KML API.
the bottom right corner of the lake, the
Conclusion
In this paper, we showed that splitting the latent-state of MBRL into two sub-latent-states, one for the environment and one for the dynamics, is a viable concept in both the simulated and real world. In all our experiments, our method was trained with a fixed dynamic model which seems confirm that similar results could be achieved with analytical models, or any other system identification method. Through our experiments, we showed that explicitly adding information about the dynamics of system make the agent more robust to both perturbations and changes of the dynamics. Finally, showed that our method could achieve environment swaps from one robot to another, allowing to train robots fully offline. The learned policy could then be deployed the target system and be refined.
Despite showing strong results, some limitations remain. Most notably, the transfer between systems will only be possible if they share the same state-space. Additionally, with the current setup, the dynamics does not explicitly depend on the environment. This could be addressed by providing either exteroceptive inputs or access to the environment state to the physics module. Future work will focus on integrating this component, along with how to apply domain randomization on the robots" dynamics within the imagination process. This is possible because we can manipulate the dynamics independently of the environment within the imagination while being sure that the imagined environment will be consistent with imagined physics. doing so, we hope to significantly speed up the learning process of robust actors. We will also investigate the integration of physical model that uses physics as model prior as [26, 22, 27].

---

Acknowledgments
This work was partially funded by the French Agence Nationale de la Recherche (ANR) under the reference number ANR-19-CE10-0N
References
[1] A. Piergiovanni, A. Wu, and M. S. Ryoo. Learning real-world robot policies by dreaming. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7680-7687. IEEE, 2019.
[2] A. Richard, S. Aravecchia, T. Schillaci, M. Geist, and C. Pradalier. How train your heron. IEEE Robotics and Automation Letters, 6(3):5247-5252, 2021. 10:10.1109/RR. 2021.3065278.
[3] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21810-21823. Curran Associates, Inc., 2020. URL
http://rrrodnnnnn..eue........ hww.ipp.cccppppr////0. pdf.
[4] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14129-14142. Curran Associates, Inc., 2020. URL http:: pttttp::/pproceednngg nwwriss. cc/ paper/2020/ wtt///////e///////////e//eeeeeee httee//2000/11eaa222222eeedddd1 a222552e00073332277767762bbbb
Paper pdf.
[5] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[6] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: 2016.
[7] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Li, D. d. L. Casas, D. Budden, A. Abdolmaleki,
J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint 1801.00000000600 2018.
[8] A. Brunnbauer, L. Berducci, A. Brandstätter, M. Lechner, R. Hasani, D. Rus, and R. Grosu. Model-based versus model-free deep reinforcement learning for autonomous racing cars. arXiv preprint arXiv:2 /03.049903 2021.
[9] P. Becker-Ehmed M. Karl, J. Peters, and P. van der Smagt. Learning to fly via deep model-based reinforcement learning. arXiv preprint arXiv:2003.08876, 2020.
[10] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
[11] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
[12] M. Deisenroth and C. Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference muchine learning (ICML-11), pages 465-472. Citeseer, 2011.
[13] M. Deisenroth, G. Neumann, J. Peters, et al. A survey on policy search for robotics. Foundations and trends in Robotics, 2(1-2):388-4)) 2013.
[14] J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32 32(11):223--22) 2013.
[15] Abbeel, A. Coates, M. Quigley, and A. Ng. An application of reinforcement learning to aerobatic helicopter flight. In B. Schölkopf, J. Platt, and Hoffman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2007. https: /tpp//tettp//prooeeeddnng Paper, pdd..
neurips.

---

[16] A. Plaat, W. Kosters, and M. Preuss. Model-based deep reinforcement learning for high- dimensional problems, survey. arXiv preprint arXiv:2008.05598, 2020.
[17] D. Hafner, Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. arXiv preprini arXiv:2010.02219, 2020. 2020.
[18] D. P. Kingma and M. Welling. Auto-encoding variational bayes. International Conference Learning Representations, 2014.
[19] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and Davidson. Learning latent dynamics for planning from pixels. In JCML, 2019.
[20] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Tassa, D. Silver, and Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
[21] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.063777 2017.
[22] Y. D. Zhong and N. Leonard. Unsupervised learning of lagrangian dynamics from images for prediction and control. Advances in Neural Information Processing Systems, 33, 2020.
[23] A. Sanzaez- N. Heess, T. Springenherg. Merel, M. Riedmiller, R. Hadsell, and P. Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pages 4470-4479, PMLR, 2018.
[24] Battaglia, R. Pascanu, M. Lai, D. J. Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems, pages 4502-4510, 2016.
[25] D. Fox, Thrun, and W. Burgard. Probabilistic Robotics. 9780262201629.
Kybernetes, 2006. ISBN
[26] M. Lutter, C. Ritter, and J. Peters. Deep lagrangian networks: Using physies model prior for deep learning. In International Conference on Learning Representations (ICLR 2019). OpenReview. net, 2019.
[27] Toth, D. J. Rezende, A. Jaegle, Racanière, A. Botev, and Higgins. Hamiltonian generative networks. In International Conference on Learning Representations, 2020.
[28] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, al. Tensorflow: system for large-scale machine learning. In 12th {USENIX) symposium operating systems design and implementation ({OSDI) 16), pages 265-283, 2016.
10


Reviews:
[{"review_id": "in3Y8RS3Jw8", "reviewer": "Reviewer_TKXJ", "summary": "This work performs model based reinforcement learning with a factored latent space, where the environment and robot dynamics are factored into separate components. For the environment component, the forward model uses proprioceptive values instead of agent actions for transitions and representation learning. For the robot dynamics, the robot dynamics ignore environment information for representation and transition values, and is able to reconstruct the proprioceptive values used by the environment model. Model based reinforcement learning follows the pattern of Dreamer, which uses imagined trajectories from rolled out latent states to predict performance. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis work provides a clear, easy to implement system for incorporating robot information without needing too much explicit encoding of information.\n\nThe proposal of transfer is very interesting, and the assessment is describes fairly clearly. However, transfer on real robots would have been nice.\n\nThe experiments are interesting because they propose robot learning in the middle of a robotics pipeline, (requiring an EKF, noisy GPS), with real world evaluation. This is a much more realistic scenario than many of the other works based on Dreamer have, and demonstration in this way is encouraging.\n\nWeaknesses:\n\nBy not including environment information in the robot dynamics, this means that any effect of the environment on the movement of the robot will be encoded as stochasticity (at best). However, the environment can often have a significant effect on the dynamics of the robot. For example, if the robot arm picks up a weight, and this changes the dynamics of the robot, or there is a moving barrier. \n\nModel based reinforcement learning methods, and other reinforcement learning methods, often explicitly encode a separate component of the model for the proprioceptive components of the state. This method is most likely unique in incorporating proprioceptive state into the Dreamer-like framework, but this is a relatively marginal difference from a normal network, hard-coding in robot dynamics, or other methods already being used to include this information. \n\nA component that is often under-appreciated in model based reinforcement learning methods is how to control the distribution of rollout data used to train the model, while the policy might be performing undesirable behavior. This is not particularly well addressed in this work as well, as it describes only a training schedule where the dynamics are updated less. By the nature of the experimental domain, this is probably not stressed, since dynamics in a boat domain do not often significantly change as long as the vehicle does not run aground.\n\nA comparison with a dynamics method that also incorporates the robot information, though in a different way (including it in a larger neural network but a separate channel, including an explicit hand-designed model), would have probably provided a better comparison than Dreamer, since by separating out the robot this work adds additional information that Dreamer does not have, so comparison is limited. \n\nWhile the boat domain is realistic and interesting, as the only experimental domain (though with simulated versions), it does lack some desirable assessment capabilities. In particular, in this domain the effect of the environment on the agent is minimized, since the effect of the environment on movement of the boat can often be overcome relatively easily. In addition, the dynamics of the agent are intuitive, making it easier to set hyperparameters and balance exploration. It would be desirable to have a robotics domain involving a more complex proprioceptive system. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I propose a weak accept, mostly because the work only provides marginal originality, and does not assess on a robotics domain which would have provided more dynamics learning challenge (such as robot manipulation). However, the work provides some interesting results relatively clearly along with the proposed algorithm\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Experiments on some domain that stresses a variety of skills would be preferable, such as a robot manipulation domain.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "MAhfTbxGpT", "reviewer": "Reviewer_Q3uB", "summary": "The paper introduces a new extension of a Dreaming, model-based neural network, in which policies are trained by creating synthetic, ie dreamed or imagined, trajectories in a latent state embedding for additional training data.\nThe contribution of the paper is to split the latent embeddings, and latent dynamics functions into two: One models the environmental transition function while the second models the dynamics of the intrinsic states of an agent.\nBoth models are then used to unroll two latent trajectories into the future: One predicting changes in the environment, the second changes to the robot or agents own state.\nThe benefit of this approach is that in principle, either the networks modelling the environmental dynamics or the networks modelling the agent dynamics can be replaced or re-trained. This would allow in principle to speed up the training process of a new agent, or in a new environment, or allow the possibility to transfer to new combinations.\nFor example, if agent A is trained in environment 1, and agent B in environment 2, one would be able to exchange the learned networks and transfer zero-shot to a combination of, for example, agent A and environment 2.\n\nThe paper provides some evidence that this is possible by using the proposed approach to train the environmental and agent dynamics of an autonomous boat and a vehicle. In evaluations, the paper is able to demonstrate the ability to transfer knowledge in a limited experiment in which the environmental dynamics experienced by the boat and the agent dynamics learned from the vehicle.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The general idea of the learning algorithm presented in the paper is interesting and well motivated.\n- Although not all presented results indicate that the proposed approach is able to outperform the baseline, ie Dreamer, there is some empirical evidence that the algorithm is working and would be applicable to a range of different scenarios when finetuned (see below)\n- The experimental setup is well chosen, and a real-world experiment is conducted in which a real unmanned ship is steered along the shore of a lake.\n- The authors promise to publish the code on Github and make it available to the wider research community under an open-source license.\n\n\nWeaknesses:\n- The paper was hard to follow at times. For example, for the dynamics models, the states are first introduced as s^dyn and s^env, later they are changed to greek letters, as well as x and o. Figure 1 uses an alternative notation, with phi, p and h. The notion should be harmonized in the final version.\n- The appendix has empty sections.\n- The proposed approach works only as intended if the internal robot variables are (a) are of the same type and order, (b) include exclusively state variables that are agnostic to the robot type and model, for example, its world position and velocity, and (c) the state dimensions are the same between different types of robots. The approach works well in the paper because the internal state of the agent contains only the GPS coordinates (position) and task-space velocities and accelerations produced by the onboard IMU. I  believe the proposed approach would have difficulties if the agent state, ie x_t, would also contain actuator states, for example. In such a case, the environmental transition function would also depend on the concrete type of agent - eg in the case of turbines versus wheels. This can be solved in principle by using only a suitable subset of the agent's internal state. This problem should be discussed in the paper to broaden its applicability.\n- Environmental and dynamic models are pretrained in simulation. It would have been more interesting if the models were trained online during the learning process.\n\nOthers:\n- A similar approach in which agent dynamics and environment dynamics are separated from each other, based on graph neural networks, is presented in [1]. This work could be discussed in the related works section \n\n[1] Sanchez-Gonzalez, Alvaro, et al. \"Graph networks as learnable physics engines for inference and control.\" International Conference on Machine Learning. PMLR, 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the paper has some weaknesses in its writeup, which could be improved, and not all experiments indicate an outstanding performance of the proposed approach, the presented method and its demonstration on a real-world USV should be interesting for the wider robot learning community. The idea of separating the environmental and agent dynamics in a Dreamer-like architecture seems interesting, of value, and well-motivated, especially to allow zero-shot transfer between different agent and environmental combinations.\nI am also confident that the authors can address the outstanding issues for their final draft, especially regarding clarity and notation. \nIt might have been beneficial to apply the proposed method on a few more toy problems in simulation, such as mujoco or pybullet, to demonstrate further its applicability and performance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- The notation in the introduction, figures and the methods section should be the same.\n- The mentioned problem of having different dimensionalities between agents or having additional internal states, such as motor states, should be discussed.\n- Ideally, the appendix should give a more detailed explanation of the learning setup, especially regarding the number of states for each task, their dimensionality and how they are defined.\n- Appendix D does not exist - this also means that the details reg. the curriculum training process is missing. These details need to be added for the updated version."}}, {"review_id": "AC5BDVSy97l", "reviewer": "Reviewer_WcEr", "summary": "The paper proposes a novel algorithm for robot-environment decomposition in MBRL, decomposing the prioprioceptive and exteroceptive modeling. Specifically, the method is based on Dreamer, and learns two RSSM models, one to predict next robot state given action, and another one to predict next visual (or in this case LIDAR) observation given the robot state. It is shown that this decomposition improves robustness to distributional shift, including the sim-to-real shift, with respect to a monolithic Dreamer model. Further, it can be used to adapt to new robots by composing an environment model of target environment with a robot model of a source robot. Experiments on ground and water surface navigation are provided in simulation and real world. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The algorithm is novel, sound, and likely will be used by robotics researchers\n- The paper is well-written\n- The experiments are extensive and clearly demonstrate the benefits of the proposed method\n\nWeaknesses:\n- The experiments are perhaps a little unsatisfying. First, the paper does not analyze how exactly the proposed method improves robustness. There are no provided ablations.\n- Second, it is unclear why it is necessary to use Dreamer for the proposed use-case. Since the method assumes access to a simulator, a policy can be trained on the simulator rollouts directly using an on-policy algorithm. Similarly, since the simulator is available for all robots, it is easy to just train on the target robot and environment instead of performing transfer.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the strengths of the paper outweigh the weaknesses. The paper proposes a promising algorithm and provides sufficient experiments to validate the benefits of the proposed idea. The algorithm is relevant and is likely to be used by other researchers.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "See comments on Weaknesses. Further, there are the following writing issues:\n- l30: Finn’17 should be cited together with [9]\n- Decomposition on l41 is quite intriguing. However, as far as I can see it is not used anywhere else in the paper. If the proposed method indeed decomposes the dynamics as on l41, perhaps this can be derived explicitly.\n- l196 the reference is broken\n\nFinn’17, Deep visual foresight for planning robot motion\n"}}, {"review_id": "6OBv4EUzofp", "reviewer": "Reviewer_RFKR", "summary": "This paper proposes to separately develop a world model for the robot physics state and the environmental state in the context of model-based RL. The method is particularly developed on the backbone of recurrent state-space model, and case studies robot navigation tasks demonstrate the efficacy of the proposed scheme. \n\nThe proposed idea looks interesting and technically sound, and the paper is relatively well-written. However, the contribution of the paper needs to be better clarified, as the baseline idea of separating the physics state and the environment state is not new. Traditional model-based approach actually all rely on this type of separation and it may be the artifact of the effort to try to learn everything in a data-driven way. In this sense, a more fundamental motivation of this work needs to be provided other than the state-of-the-art RL is not working very well. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The idea is convincing and makes physical senses. \n- The paper is relatively well-written and well-organized. \n\nWeaknesses:\n- The fundamental motivation is not very clear. \n- The contribution needs to be better clarified. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a technically sound idea that is tested on a reasonable case study. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- There are some typos. \n- Motivation and contributions need to be better clarified. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work claims an idea by decomposing dynamic model into physical agent model and environmental model, which is straightforward but also promising. Authors also demonstrate this kind of decomposition can contribute to transfer efficiently among different agent sharing the same environments. Reviewers have consistent comments that the idea claimed in this work is convincing and also conducted with sufficient experiments overall. However, there are still some concerns raised by reviewer as follows.  
1. The contribution and novelty compared to the existing dynamic decomposing work should be more clearly presented. 
2. The representation of physical dynamic model sounds like too simple, which would be limited to other more complicated agents. 
3.  Experimental setting compared to Dreamer is a little bit unfair. 
4. Some notions are not unified. 

Although there are still some limitations existed in this work, most of reviewers thought the majority of the contributions have been clearly clarified with consistent opinion to recommend to accept it as CoRL paper. 
Decision: Accept (Oral)