Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots
Authors: Katie Kang
Gregory Kahn
Sergey Levine
Venue: CoRL2021 Poster (2021)
Keywords: deep reinforcement learning
multi-robot learning
mobile navigation

Abstract: Deep reinforcement learning algorithms require large and diverse datasets in order to learn successful policies for perception-based mobile navigation. However, gathering such datasets with a single robot can be prohibitively expensive. Collecting data with multiple different robotic platforms with possibly different dynamics is a more scalable approach to large-scale data collection. But how can deep reinforcement learning algorithms leverage such heterogeneous datasets? In this work, we propose a deep reinforcement learning algorithm with hierarchically integrated models (HInt). At training time, HInt learns separate perception and dynamics models, and at test time, HInt integrates the two models in a hierarchical manner and plans actions with the integrated model. This method of planning with hierarchically integrated models allows the algorithm to train on datasets gathered by a variety of different platforms, while respecting the physical capabilities of the deployment robot at test time. Our mobile navigation experiments show that HInt outperforms conventional hierarchical policies and single-source approaches.

Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots
Katie Kang
Gregory Kahn
Sergey Levine
UC Berkeley
UC Berkeley
UC Berkeley
Abstract: Deep reinforcement learning algorithms require large and diverse datasets in order to Iearn successful policies for perception- based mobile navi- gation. However, gathering such datasets with single robot can be prohibitively expensive. Collecting data with multiple different robotic platforms with possi- bly different dynamics is more scalable approach to large-scale data collection. But how can deep reinforcement learning algorithms leverage such heterogeneous datasets? In this work, propose deep reinforcement learning algorithm with hierarchically integrated models (HInt). At training time, Hlnt learns separate perception and dynamics models, and test time, HInt integrates the mod- els in a hierarchical manner and plans actions with the integrated model. This method of planning with hierarchically integrated models allows the algorithm train on datasets gathered by variety of different platforms, while respecting the physical capabilities of the deployment robot test time. Our mobile navigation experiments show that HInt outperforms conventional hierarchical policies and single-sous source approaches.
Keywords:
Deep reinforcement learning, Multi-robot learning, Autonomous nav-
igation
Train
Test
Integrated planner
observation
Model
b Perception Model
action
data
data
Perception Model
Dynamics Model
Figure 1: Overview of our hierarchically integrated models (HInt) algorithm. At training time, HInt separately trains perception model and a dynamics model, then test time. Inlt combines the perception and dynamics model into single model for integrated planning and execution. Our modular training procedure enables Hint train the perception model using data gathered by multiple platforms, recording video with a hand-hel camera, while our integrated model at test time ensures the perception model only considers trajectories which are dynamically feasible.
as ground robots and even people
Introduction
Machine learning provides a powerful tool for enabling robots acquire control policies by learning directly from experience. One of the key guiding principles behind recent advances in machine learning is to leverage large datasets. In previous works in deep reinforcement learning control, the most common approach is to collect data from single robot, train a policy in an end-to- end fashion, and deploy the policy on the same data-collection platform. This approach necessitates collecting a large and diverse dataset for every model of robot we wish deploy on, which can present a significant practical obstacle, since there exists a plethora real world. What if we could instead leverage datasets collected by a variety of different robots in our training procedure? An ideal method could use data from any platform that provides useful knowledge about the problem. As example, it is expensive and time-consuming to gather large dataset for visual navigation on drone, due to on-board battery constraints. In comparison, is
robotic
different platforms in the
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

much easier to collect data on a robot car. Because knowledge about the visual features of obstacles can be shared across vision-based mobile robots, making use of visual data collected by a car train a control policy for drone could significantly reduce the amount of data needed from the drone. Unfortunately, data from such heterogeneous platforms presents a challenge: different platforms have different physical capabilities and action spaces. In order to leverage such heterogeneous data, we must properly account for the underlying dynamics of the data collection platform.
To learn from multiple sources of data, previous works have utilized hierarchical policies [1. 2, 3]. In this type of method, high-level and a low-level policy are trained separately. At test time, the actions generated by the high-level policy are used as waypoints for the low-level policy. By separating the policy into two parts, these algorithm are able to utilize data from multiple sources in the high-level policy, while representing robot-specific information such as the dynamics in the low-level policy. One drawback of these methods, however, that the low-level policy is unable to communicate any robot-specific information to the high-level policy. This makes it possible for the high-level policy to command waypoints that are impossible for the robot to physically achieve, leading to poor performance and possibly dangerous outcomes.
The key idea in our approach, illustrated in Fig. 1, to instead learn hierarchical models, and to integrate the hierarchical models for planning. At training time, we learn a perception model that reasons about interactions with the world, using our entire multi-robot dataset, and a dynamics model specific to the deployment robot, using only data from that robot. At test time, the perception and dynamics models are combined to form single integrated model, which a planner uses to choose the actions. Such hierarchically integrated models can leverage data from multiple robots for the perception layer, while also reasoning about the physical capabilities of the deployment robot during planning. This is because when the algorithm uses the hierarchically integrated model to plan, the dynamics model can "hint" to the perception model about the robot-specific physical capabilities of the deployment robot, and the planner can only select behaviors that the deployment robot can actually execute (according to the dynamics model). contrast to hierarchical policies, which may produce waypoint commands that are dynamically infeasible for the deploymen robot to achieve, our hierarchically integrated models take the robot's physical capabilities into account, and only permit physically feasible plans.
The primary contribution of this work Hint- hierarchically integrated models for acquiring image-basee based control policies from heterogeneous datasets. We demonstrate that Hint successfully learns policies from heterogeneous datasets in real-worle navigation tasks, and outperforms methods that use only one data source or use conventional hierarchical policies.
2 Related Work
Prior work demonstrated end-to-end learning for vision-based control for wide variety of appli- cations, from video games [4, 5] and manipulation [6, 7] to navigation [8, 9]. However, these ap- proaches typically require large amount of diverse data [10], which hinders the adoption of these algorithms for real-world robot learning. One approach for overcoming these data constraints is to combine data from multiple robots; While prior methods have addressed collective learning, they typically assume that the robots are the same [11], have similar underlying dynamics [12], or the data is from expert demonstrations [13, 14, 15). Our approach learns from off off-policy data gathered by robots with a variety of underlying dynamics. Prior methods have also transferred skills from simulation, where data is more plentiful [16, 17, 18, 19, 20, 21, 22]. In contrast, our method does not require any simulation, and instead is aimed at leveraging heterogeneous real-world data sources, though it could be extended to incorporate simulated data well.
Prior work has also investigated learning hierarchical vision-basch control policies for applications such autonomous flight [23, 2] and driving [1, 24, 3. 25]. One advantage of these conventional hierarchy approaches is that many can leverage heterogeneous datasets [26, 12, 23]. However, even if each module is perfectly accurate, these conventional hierarchy approaches can still fail because the low-level policy cannot communicate the robot's 's capabilities and limitations to the high-level policy. In contrast, because HInt learns hierarchical models, and performs planning on the integrated models at test time. HInt is able to jointly reason about the capabilities of the robot and the perceptual features of the environment.
End-to-end algorithms that can leverage datasets from heterogeneous platforms have also been in- vestigated by prior work; however, these methods typically require on-policy data, are evaluated in

---

visually simplistic domains, or have only been demonstrated in simulation [27, 28, 29]. In contrast, HInt works with real world off-policy datasets because at the core of HInt are predictive models, which can be trained using standard supervised learning.
3 Hlnt: Hierarchically Integrated Models
Our goal is to learn image- based mobile navigation policies that can leverage data from heteroge- neous robotic platforms. The key contribution of approach, shown in Fig. 2, is learn separate hierarchical models at training time, and combine these models into a single integrated model for planning at test time. The two hierarchical models include high-level, shared perception model, and a low-level, robot-specific dynamics model. The perception model can be trained using data gathered by a variety of different robots, all with possibly different dynamics, while the robot- specific dynamics model is trained only using data from the deployment robot. test time, because the output predictions of the dynamies model whhich are kinematic poses -are the input actions for the perception model, the dynamics and perception models can be combined into single inte- grated model. In the following sections, we will describe how HInt trains perception model and dynamics model, combines these models into single integrated model in order to perform planning, and conclude with an algorithm summary.
3.1 Perception Model
The perception model a neural network predictive model 88 (o,, 8p + ft:++ Shooww in Fig. 2- parameterized by model parameters 0. j takes as input the current image observa- tion On and a sequence of future changes in poses ōpt: = (öp, Sp.+1..... 0p:+H-1), and predicts H future rewards fI configuration of the robot. In our experiments, we used the robot's x position, y position, and yaw orientation the pose, though 3D configurations that include height, roll, and pitch are also pos- sible. This kinematic configuration provides a dynamics-agnostic interface between the perception model and the dynamies model. The perception model is trained using the heterogeneous dataset DeER by minimizing the objective:
(fz, fe+1...., Prt Ht+HH The pose characterizes the kinematic
C (0, DrEA)
Σ
P H m12
(o, H FEaRH)
fEA+ S (o,, Pp
The perception model can be trained with heterogeneous dataset consisting of data gathered by variety of robots, all with possibly different underlying dynamics. The only requirement for the data that the recorded sensors (e.g., camera) are approximately the same, and that we can calculate the change in pose-position and orientation-betweed each sequential datapoint, which could be done using visual or mechanical odometry, well the reward, which could be collected via onboard sensors or manual labeling. This ability to train using datasets gathered by heterogeneous platforms crucial because the perception model an image- based neural network, which requires large amounts of data in order to effectively generalize.
3.2 Dynamics Models
The dynamics model a neural network predictive model ( acr+ + 8p - shown in Fig. 2 parameterized by model parameters ø. f takes as input the current robot state S, and
Figure 2: During training, we learn two separate neural network models. The dynamics model takes as input the current robot state and a sequence of future actions. and predicts future changes in poses. This model is trained using data gathered by single robot. The per- ception model takes input the current image obser- vation and a sequence of future changes in poses, and predicts future rewards. This model is trained using data gathered by variety of robots that have the same image observations, but potentially different dynam- ics. When deploying, the dynamics and perception models are combined into a single integrated model ft-1-1 that is used for planning and executing actions that maximize reward.
Training
Deployment
Dynamics Model
tograti Monel
Dynamics Model
cp:
8-1
op:
6P
Perception Model
Perception Model
Ft-H-1
3

---

a sequence of H future actions a = (a,, a6+1. ....... a6+H-1). and predicts H future changes in poses ÖP (Óp, 8p ................................ OP:-H-1) The state can include any sensor information, such as the robot's 's battery charge or velocity, that may help the model to make more accurate predictions of future poses. The dynamics model is trained using the respective robot-specific dataset DDYN by minimizing the objective:
C (0, DUN)=
Σ
1PPLTH 8p::+H1l2
(Be Be Spec4H)
8P54+H fo f2 (st, ac(+H).
Although the dynamics model must only be trained using data collected by the deployment robot, the dynamics model dataset can be significantly smaller compared to the perception model dataset, because the dynamics model inputs are lower dimensional by orders of magnitude. Furthermore, the dynamics model dataset does not need reward signal, allowing for easier data collection. Though, because perception and dynamics are decoupled in HInt, the dynamics model can accurately model systems whose dynamics depend on low-dimensions state information only.
3.3 Planning and Control
In order to perform planning and control at test time, we first combine the perception and dynamics models into a single integrated model fa.e (or, St, a = f8tx (o,, fgy (s,, a+:++ - fez+ H+ shown in Fig. 2. These models can be combined without any additional training because the output of the dynamics model-changee This integrated model essential because enables the planner holistically reason about the entire system. In contrast, in conventional hierarchical control methods, where the high-level policy outputs goal for the low-level policy, the high-level policy could output dynamically infeasible reference trajectory for the low-level controller. Our approach would not suffer from this failure case because, with integrated planning. the perception model can only take as input saamiallly-fessbblee trajectories that are output by the dynamics model.
in kinematic poses-is is also the input to the perception model.
We then plan each time step for the action sequence that maximizes reward according to the integrated model by solving the following optimization:
a H arg max R(fra4H 6 a,.+H, gi)
Be:t+
PRc-H = f8.0 .COM (oz, St, a(=+H).
Here, R user-defined tpsk-specific reward function, and g: is a user-specified goal. We fol- low the framework of model predictive control, where cach time step, the robot calculates the best sequence of actions for the next steps, exccutes the first action in the sequence, and then repeats the process at the next time step. The action sequence that approximately maximizes the objective in Eqn. can be computed using any optimization method. In our implementation, we em- ploy stochastic zeroth-order optimization, which selects the best sequence among set of randomly generated trajectories, as is common in model- based RL [30, 31]. Specifically, we used either the cross-entropy method (CEM) [32] or MPPI [33]. depending on the computational constraints of the platform. While this approach does result in limitation for long-horizon reasoning, our main focus perform short-horizon navigation.
3.4 Algorithm Summary
We now briefly summarize how our HInt system operates during training and deployment, as shown in Fig. and Fig. 2.
During training, we first gather perception data using number of platforms. For each of these platforms, we save the onboard observations o, change poses öp, and rewards r into a shared dataset PER (see $B.1), and use this dataset to train the perception model feeR (Eqn. 1) (see $B.2). Then, using the robot we will deploy at test time, we gather dynamics data by having the robot act in the real world and recording the robot's onboard states actions and change in poses öp into the dataset DOYN (see $C. 1); we use this dataset to train the dynamics model fayN (Eqn. 2) (see $C.2).
When deploying Hlnt, we first combine the perception model ʃ and dynamics model JoYN into a single model foom The robot then plans a sequence of actions that maximizes reward (Eqn. 3), executes the first action, and repeat this planning process at each time step until the task is complete, as is standard in model- based RL with model predictve control [30, 31] (see SD).

---

Indoor Robot in Office
Outdoor Robot in Urban
Person in Industrial
Drone in Urban
Figure 3: Training data gathered by indoor Yujin Kobuki robot in office, outdoor Clearpath Jackal robot an urban enviroument, and person with a video camera in an industrial area. Hint is able to jointly train on this heterogencous data, which leads to improved navigation performance. Hint was deployed on the ourdnor Jackal robot as well as a Parrot Bebop drone in the urban environment. Because Hīnt is able to reason about the dynamies of the deployment robot during planning, it is able successfully control robots that are included in the perception dataset, such the drone.
Experiments
We now present an experimental evaluation of our hierarchically integrated models algorithm in the context of real world mobile robot navigation tasks. Videos and additional details about the data sources, models, training procedures, and planning can be found in the Appendix section or on the project website: https: //sitsssssss.e..///ss////e///ees google le com/ berkeereeeeeee ey sedu/hintt -alllt
In our experiments, we aim to answer the following questions:
Q1: Does leveraging data collected by other robots, in addition data from the deployment robot, improve performance compared to only learning from data collected by the deployment robot?
Q2: Does Hint's integrated model planning approach result in better performance compared to conventional hierarchy approaches?
In order to separately examine these questions, we investigate Q1 by training the perception module with multiple different real-world data sources, including data from different environments and dif- ferent platforms, and evaluating on a single real-world robot. To examine Q2, we deploy a shared perception module systems with different low-level dynamics.
4.1 QI: Comparison to Single-Source Models
In this experiment, we deployed robot in a number of visually diverse environments, including ones in which that robot itself has not collected any data. Our hypothesis is that Hint, which can make of heterogeneous datasets gathered by multiple robots, will outperform methods that can only train using data gathered by the deployment robot.
Perception data collected in three different environments using three different platforms (Fig. 3): a Yujin Kobuki robot in an indoor office environment (3.7 hours), a Clearpath Jackal in an outdoor urban environment (3.5 hours), and person recording video with hand-held camera in indus- trial environment (1.2 hours). The deployment robot is the Clearpath Jackal. The same dataset that provided the Jackal perception data as described above was also used for the Jackal dynamics data.
The robot's objective is to drive towards a goal location while avoiding collisions and minimizing action magnitudes. More specifically, the reward r is -1 for collision and 0 otherwise, the goal specifies GPS location, < ge) denotes the angle in radians formed by 5p and ge with respect to the robot's current position, and the reward function used for planning is
R(fu++ 5p ac.t ge)
(4)
H-1 Pus 0.3 L(6p Bi) 0.05 116P+41l1--
We compared HInt, which learns from all the data sources, with a variant of HInt which only leamed from a single data source. Both approaches use the same training parameters and neural network architecture for the integrated model in order to make the most fair comparison, with the only difference between the data with which they were trained. In our implementation, Hint and the single data source approach both build upon the method proposed in Kahn
[34]. Kahn et al [34] also uses a vision-based predictive model to perform planning, but differs from Hint in the following ways: (i) only data gathered by the deployment robot can be used for training and (ii) the integrated perception and dynamics model is trained end-to-end.
5

---

Perception Data Sources
Single-Source
Hint (ours)
Kobuki
Jackal
Human (Industrial)
Kobuki Jackal (Office) (Urban)
Human (Office) (Urban) (Industrial)
Kobuki Jackal
OOffice)
(Urban)
Test Env
Urban
0%
13%
N/A
100 %
100 %
Industrial
0%
N/A
0%
33%
87 %
Collisian
Office
Urban
Office
Industrial
42
31-
10
37
74
37
maters
52
13
20
2.
20
2.5
meters
meters
meters
meters
Figure 4: Comparison of single data source models vironment for the task reaching a goal location while avoiding collisions using the Clearpath Jackal robot. Note that HInt was trained with more datapoints than the single source approach, because Hint able to learn from data collected by other platforms in addition to the Jackal deployment robot. Fach approach was evaluated from the 3 same start locations in each environment (corresponding to the red, green, and blue lines), and was times from each start location. The images show top down view of the Urban (top row) and Industrial (bottom row) environments, with each column corresponding model. The quantitative results show what percentage of the trials successfully reached the goal.
HInt approach in urban and industrial
data sources used train perception
Urban
Industrial
Figure 5: Visualization of Hint at test time successfully reaching the goal location while avoiding collisions in urban (top) and industrial (bottom) environments.
Fig. 4 shows the results comparing Hint to the single data source approach. In all environments! our approach is more successful in reaching the goal. Note that even when the single data source method trained and deployed in the same environment, HInt still performs better, because learning- based methods benefit from large and diverse datasets. Furthermore, the row labeled "Industrial" illustrates well how HInt can benefit from data collected with other platforms: although the Jackal robot had never seen the industrial setting during training, the training set did include data collected by a person with video camera in this setting. The increase in performance from including this data ("oobuki + Jackal + Human ") shows that the Jackal robot was able to effectively integrate this data into its visual navigation strategy. Fig. 5 shows first-person images of Hint successfully navigating in both the urban and industrial environments.
4.2 Q2: Comparison to Conventional Hierarchy
In this experiment, we deployed perception module onto robots with different low-level dynam- ics, including ones that were not during the collection of the perception data. Our hypothesis is
We could not experiments in the office environment due to COVID-related closures.

---

that our integrated HInt approach will outperform conventional hierarchy approaches, because it is able to jointly reason about perception and dynamics.
We compared our integrated approach with the most commonly used hierarchical approach, in which the perception model is used output desired waypoints that then passed to a low-level con- troller [1, 23, 24, 2, 3]. In our prediction- and baaniing-basee framework, this modular approach is implemented by first planning over a sequence of positions 8pi.+ by using a kinematic vision- based model, and then planning over sequence of actions 2:--11 f1 so as to minimize the tracking error against these planned positions using a robot-specific dynamics model. This baseline repre- sents clean "apples-to-ap"" -apples"" comparison between our approach which directly combines both the dynamic and kinematic models into single model and conventional pipelined approach that separates vision- based kinematic planning with low-level trajectory tracking. Both our method and this baseline employ the same neural network architectures for the vision and dynamics models, and train them on the same data, thus providing for controlled comparison that isolates the question of whether our end-to-end approach improves over a conventional pipelined hierarchical approach. The planning process for the hierarchical baseline can be expressed as the following two-step opti- mization:
Pp H = arg max R(f HI, 6p ..)))
(5)
8PL11
Pu4HH = S0 (o,, 8pac+H)
11
a = arg min
6P - -pi+nal2
(6)
6 = fo (st, ac.+H).
We trained the perception model with three data sources from different robots ("Kobuki Jackal + Human" in Fig. 4), one of which is a Clearpath Jackal robot. We then deployed this perception model on the Jackal robot. The robot's objective is to go towards a goal location while avoiding collisions, using the same reward function for planning as Eqn. 5.
We also evaluated method the data collection process for the perception model. Note that this generalization to a new platform was only possible because the drone's 's image observations lies in-distribution within the perception training data. The drone's 's objective is to avoid collisions with minimal turning. The reward function for planning is
Parrot Behop drone, shown in Fig. 3, which was not included in
R(fe+ H. SP H: a
H-1
Σ Prth 116
The results in Fig. 6 compare HInt against the conventional hicrarchy approach on a Jackal robot with its normal dynamics and with its steering limited to 40% of its full steering range, well as on a Bebop drone flying at low speed (8 degrees forward tilt) and high speed (16 degrees forward tilt). While the approaches were able to achieve similar performance when evaluated on the normal Jackal and the low speed drone, our approach outperformed conventional hierarchy on the limited steering Jackal and on the high speed drone. The poor performance of the conventional hierarchy baseline in these experiments can be explained by the inability of the conventional higher-level planner which plans kinematic paths based on visual observations dynamics limitations of each platform. This can be particularly problematic near obstacles, where the kinematic model might decide that a last-minute turn to avoid an obstacle may be feasible, when in fact the dynamics of the current robot make it impossible. This is not an issue for the standard Jackal robot and the low speed drone, which are both able to make sharp turns. However, the limited steering Jackal was unable to physically achieve the sharp turns commanded by the higher-level model in the baseline method. Similarly, due to the aerodynamics of high-speed flight, the Bebop drone was also unable to achieve the sharp turns commanded by the baseline higher-level planner, leading to collision (see, e.g., the lower-right plot in Fig. 6). In contrast, in Hlnt, because the low- level model is able to inform the high-level model about the physical capabilities of the robot, the integrated model could correctly deduce that avoiding the obstacles required starting the turn earlier when controlling a less maneuverable robot, which allowed for successful collision avoidance.
account for the different
7

---

Normal Jackal
Lamited Steering Jackal
Low Speed Drone
High Speed Drone
Conventional Hierarchy HInt (ours)
80%
100%
20%
80%
100%
100
100 %
Conventional Hierarchy
Hint (ours)
Conventional Hierarchy
Hint (ours)
Low Speed Drone
Noraa Jackal
Limited Steering Jackal
High
Figure 6: Comparison of conventional hierarchy vs. Hint (ours) approaches in a real world experiment, on Jackal with normal dynamics and with steering limited 40%, as well as on a drone with low speed (8 degrees forward tilt) and high speed (16 degrees forward tilt). Both approaches were evaluated with 5 trials each from the same starting position. Our approach is able to achieve higher success rates for reaching the goal region when deployed the limited stecring Jackal and the drone, because the perception model was able reason about its dynamical limitations.
We also evaluated HInt and conventional hierarchy on a visual navigation task in simulation involv- ing robot with more drastic dynamical differences, such as limited steering, right turn only, and 0.25 seconds lag. More details can be found in the Appendix (see $E). In both the real world and simulation, we showed that conventional hierarchy can fail when deployed on dynamical systems that are not a part of the perception dataset. This is because the higher level perception-basee policy can waypoints for the lower level ttramics-based policy that are outside the physical capabilities of the robot. In contrast, HInt's integrated planning approach enables the dynamics model to inform the perception model about which maneuvers are feasible.
Discussion
We presented Hint, deep reinforcement learning algorithm with hierarchically integrated models. The hierarchical training procedure enables the perception model to be trained on heterogeneous datasets, which is crucial for learning image-based neural network control policies, while the inte- grated model at test time ensures the perception model only considers trajectories that are dynami- cally feasible. Our experiments show that HInt outperform both single-source and conventional hierarchy methods on real-world navigation tasks.
One of the key algorithmic ideas of HInt is that sharing perception data from heterogeneous plat- forms could improve generalization and reduce the amount of training data needed from the deploy- ment robot. As with all machine learning methods, within the distribution of the training data. So if training data comes from a variety ground robots expect good generalization
approach generalizes to situations that
with similar cameras, but not, for example, to walking robots. We conjecture that this problem can be solved by learning from bigger and more diverse dataset. An exciting direction for future work could be to scale up our experimental evaluation to a much larger number and diversity of platforms (e.g. training the perception model with data from 1000 different cars, drones, and legged robots). Such an approach could lead to a general and robust perception system that can be directly deployed on wide range of mobile robots without needing to collect additional perception data from the deployment robot. We believe that enabling robot platforms to learn from large and diverse datasets is essential for the success of robot learning, and that Hint is promising step towards this goal.
other ground robots with similar cameras,

---

Acknowledgments
We thank Anusha Nagabandi and Simin Liu for insightful discussions, and anonymous reviewers at RAIL for feedback on early drafts on the paper. This research was supported by DARPA Assured Autonomy and ARL DCIST CRA 191NFF-7---0111. KK and GK are supported by the NSF GREP.
References
[1] W. Gao, D. Hsu, W. S. Lee, S. Shen, and K. Subramanian. Intention-net: Integrating planning and deep learning for goal-directed autonomous navigation. In CoRL, 2017.
[2] E. Kaufmann, M. Gehrig, Foehn, Ranftl, A. Dosovitskiy, V. Koltun, and Scaramuzza. Beauty and the beast: Optimal methods meet learning for drone racing. In ICRA, 2019.
[3] S. Bansal, V. Tolani, S. Gupta, Malik, and C. Tomlin. Combining optimal control and learning for visual navigation novel environments. In CoRL, 2019.
[4] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1eXi::1312.660,, 2013.
[5] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
[6] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
[7] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. In CoRL, 2018.
[8] S. Ross, N. Melik-Barkhudarov, K. S. Shankar, A. Wendel, D. Dey, J. A. Bagnell, and M. Hebert. Learning monocular reactive uav control in cluttered natural environments. In ICRA, 2013.
[9] A. Kendall, Hawke, D. Janz, Mazur, D. Reda, J.-M. Allen, V.-D Lam, A. Bewley, and A. Shah. Learning drive in day. In ICRA, 2019.
[10] M. Hessel, J. Modayil, H. Van Hasselt, Schaul, G. Ostrovski. W. W. Dabney, Horgan, B. Piot,
M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning.
In AAAI, 2018.
[11] S. Levine, Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. IJRR, 2018.
[12] Dasari, Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, Singh, Levine, and
C. Finn. RoboNet: Large-scale multi-robot learning. In CoRL, 2019.
[13] A. Edwards, H. Sahni, Y. Schroecker, and C. Isbell. Imitating latent policies from observation. In ICML, 2019.
[14] F. Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. In ICML Workshop on Imitation and Intent, 2018.
[15] W. Sun, A. Vemula, B. Boots, and J. A. Bagnell. Provably efficient imitation learning from observation alone. ICML, 2019.
[16] Sadeghi and S. Levine. (CAD)2 RL: Real Single-Image Flight without Single Real Image. RSS, 2017.
[17] A. A. Rusu, M. Večerík, T. Rothörl, Heess, R. Pascanu, and Hadsell, Sim-to-real robot learning from pixels with progressive nets. In CoRL, 2017.

---

[18] F. Zhang, J. Leitner, M. Milford, and P. Corke. Modular deep q networks for sim-to-real transfer of visuo-motor policies, 2017.
[19] K. Boussmalss A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, et al. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In ICRA, 2018.
[20] X. Meng, N. Ratliff, Y. Xiang, and D. Fox. Neural Autonomous Navigation with Riemannian Motion Policy. In ICRA, 2019.
[21] H.-T. L. Chiang, A. Faust, M. Fiser, and A. Francis. Learning navigation behaviors end-to-end with autorl. RA-L, 2019.
[22] J. Zhang. L. Tai, P. Yun. Y. Xiong, M. Liu, J. Boedecker, and W. Burgard. VR-goggles for robots: Real-to-sim domain adaptation for visual control. RA-I, 2019.
[23] A. Loquercio, A. I. Maqueda, C. R. Del-Blanco, and D. Scaramuzza. Dronet: Learning to fly by driving. In RA-L, 2018.
[24] M. Müller, A. Dosovitskiy, B. Ghanem, and V. Koltun. Driving policy transfer via modularity and abstraction. In CoRL, 2018.
[25] X. Meng, N. Ratliff, Y. Xiang, and D. Fox. Scaling Local Control to Large-Scale Topological Navigation. arXiv preprint arXiv:1909.13322 2019.
[26] D. Hejna, Abbeel, and L. Pinto. Hierarchically decoupled imitation for morphological trans- fer. In ICML, 2020.
[27] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. ICRA), 2017.
[28] O. Nachum, S. S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. In NeurlPS, 2018.
[29] M. Wulfmeier, A. Abdolmaleki, R. Hafner, J. Springenberg, M. Neunert, T. Hertweck, T. Lampe, N. Siegel, N. Heess, and M. Riedmiller. Compositional Transfer in Hierarchical Reinforcement Learning. In RSS, 2020.
[30] A. Nagabandi, G. Kahn, based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559-7566. IEEE, 2018
Fearing, and S. Levine. Neural network dynamics for model-
[31] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. arXiv preprint arXiv: 2018.
[32] R. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1999.
[33] G. Williams, A. Aldrich, and E. Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv:1509.01 149, 2015.
[34] G. Kahn, Abbeel, and S. Levine. BADGR: An autonomous self-supervised learning- navigation system. arXiv:2002.05700, 2020.
[35] M. Goslin and M. Mine. The panda3d graphics engine. In IEEE Computer, 2004.
[36] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
10


Reviews:
[{"review_id": "WIHYtwpbt5_", "reviewer": "Reviewer_cpmD", "summary": "This paper presents a control strategy for mobile robots that leverages a neural network forward model in an MPC fashion. The model consists of two parts: a perception module which is trained using datasets collected from different robots and controllers and a dynamics model of a specific robot being controlled.\nThis paper tries to enrich the dataset for off-line/off-policy reinforcement learning for visual control tasks. Typically, when training a policy for a new robot, we have to create a new dataset collected by the same robot and low-level tracking controller. This paper tries to reduce this time-consuming process by reusing datasets collected by different platforms\n\nThe main contribution of this paper is the construction of modular control architecture that separates perception and motion prediction. The perception model predicts future rewards (collision event) given the future pose sequences. By construction, the perception model is independent of the dynamics of the robot & tracking controller and only focuses on the scene understanding. The motion prediction model (dynamics model) is trained separately for the tested robot to predict the future pose sequences given the future actions. The dynamics model learns the tracking behavior of the robot and its low-level controller. In the test time, the output of the dynamics model is fed to the perception model and a zeroth-order optimizer is used to generate action sequences that minimize the future rewards.\n\nWith experimental results, authors claim that:\n    - (Q1) Using diverse datasets collected in different environments and using different robots improves the performance of a navigation policy.\n    - (Q2) Proposed architecture performs better than the conventional hierarchical approaches.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- This paper is tackling an interesting problem. For autonomous driving systems, the problem is usually tackled by leveraging explicit intermediate representations such as semantic segmentation or lane detection. In this work, the pose sequence acts as an intermediate representation that connects the two separate models.\n- The proposed method and experimental results seem to be limited to specific setups. All the datasets are collected from similar perspectives and the proposed approach cannot tackle one of the main problems of reusing existing data: different perspectives, FOVs, and image sensors across datasets. As shown by \"Person in Industrial\" in Fig. 3, the data used by this work seem to be collected in similar perspectives and the generalizability of the proposed approach is questionable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper tackles an important problem in applying reinforcement learning using real-world data. The real-world experiments prove that the proposed architecture can improve the performance of a RL-trained policy as it enables using richer data and stabilizes the perceptive planning by only using (likely) feasible trajectories predicted by a dynamics model. However, the approach and experimental result cannot fully support the promises made in the introduction. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "- The result in Figure 4 cannot be used to answer question Q1. The experiment in Q1 is not fairly comparing the Multi-source v.s. Single-source. The multi-source model and the single-source model that the paper is comparing have different control architectures. One is using a hierarchical model while the other is using an end-to-end model. For a fairer comparison, the authors have to compare two models of the same control architecture that are trained with multi-source data and single-source data. The effects of different architecture and different data sources are blended in the presented experiment.\n- The dynamics model does not consider perception and can have a large variance. To predict future poses, we need environmental information such as ground friction or obstacle location. For example, when commanded to go straight into a wall, the future pose will differ depending on the distance to the wall. This sort of information cannot be predicted using the proposed dynamics model. One of the main claims of this paper is the decoupling of dynamics from the perception, which is supported by Q2, but the dynamics model should still be dependant on the perception. The paper should clearly present this limitation and in which environment/setup the datasets for dynamics models are collected. \n- In the table in Figure 4, It would be better to clarify that \"Jackal\" is used for testing.\n- Typo in like 242: \"drone drone\""}}, {"review_id": "NhcKTTO7mUy", "reviewer": "Reviewer_DdRP", "summary": "The authors propose a new machine learning paradigm for building autonomous robot navigation systems. Specifically, they propose HInt, a hierarchical approach which consists of two separately learned models: (1) a high-level \"perception\" module, and (2) a low-level \"dynamics\" module. The authors propose to train (1) from data gathered from potentially multiple robots/environments, and to train (2) using data gathered on the deployment platform itself. During deployment, the composition of these learned models is used to define a scoring function that a planner may use when searching for control sequences. The authors validate their method using real-world experimental data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\n(S1) The authors have focused on an important problem in the robotics community (vision-based navigation), and have proposed a novel machine learning method with an important benefit (ability to leverage traning data from heterogenous platforms) to address it.\n\n(S2) The authors have performed an impressive number of real-world experiments that have yielded data to support their claims.\n\nWEAKNESSES\n\n(W1) The presentation of the method itself is a bit unclear:\n\na. First, very little upfront intuition is given regarding why, exactly, one should expect the high-level perception module to \"work\" (ie, provide meaningful scores) at all. My guess is that the image and a short enough sequence of pose changes provides enough information to tell whether or not the robot will collide and/or make progress to the goal. If that's correct, the authors should very clearly state this upfront or, if not, the authors should provide some other intuition regarding why we could expect a function with these inputs to reasonably be able to learn what they claim it learns.\n\nb. Second, the presentation of the concepts surrounding Equation (1) is a bit murky. It's not until much later in the paper that one learns where the (supervised) reward information comes from, and that seems very important to the method.\n\n(W2) The experiments section is missing important information regarding how, exactly, the training data is collected. This is somewhat related with (W1b) above--if the method uses supervised learning for the reward prediction problem, then it stands to reason that a variety of rewards _must_ be present in the dataset, including a potentially large number of collisions. Can the authors comment on this? Regardless of the answer, more information regarding how training data was/should be collected needs to be provided to the reader.\n\n(W3) I find the title to be a bit misleading in the following sense. Typically, the phrase \"multi-robot,\" like \"multi-agent,\" refers to methods that seek to address problems that involve the simultaneous control of multiple agents. Here, instead, it appears that the authors are using the term to indicate that many robots are used to _gather training data_. While the proposed method is indeed an interesting contribution to the literature, I think the term \"multi-robot deep reinforcement learning\" should be reserved for methods that control multiple agents, as seems to be standard in the community. The authors should revise the title--perhaps the term \"heterogeneous\" could be more appropriate here?\n\n(W4) While I applaud the authors for the impressive real-world evaluation done here, I do wonder how the results should be interpreted in the context of autonomous navigation more broadly. What does the proposed system (e.g., a vision-based navigation system) allow the system to do in these environments that could not be accomplished by existing robot navigation systems from across the entire community? For example, is there something challenging about these environments where the requested navigation tasks could not be accomplished by a classical planning-based system using lidar? I think the authors should more clearly articulate--either in the experiments section or perhaps in the introduction--the exact benefit that machine learning provides here.\n\nMINOR COMMENTS\n\n(MC1) Figure 4 is missing important information. In particular, it's not clear what exactly is going on in the part of the figure with the aerial photos. What does each photograph correspond to? The authors have stated in the caption that red/blue/green indicate trials starting from different positions, but which environment does each row correspond to? Does the word above each picture correspond to the training data used for the trials shown? I couldn't find answers to these questions in either the caption or the text.\n\n(MC2) The \"angle\" symbol in Equation (4) needs to be precisely defined.\n\nPOST-RESPONSE COMMENTS:\n\nThanks to the authors for providing additional detail and addressing many of my concerns. I'm willing to raise my score conditioned on adding detail (either in the main paper or the appendix) about just how many collisions were required in the training data in order to achieve the results shown in the paper. I think this is a critical detail.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My current recommendation is \"weak reject.\" I very much like the overall thrust of the work, and I applaud the authors for proposing an interesting new method that can potentially allow system designers to take advantage of a wide variety of vehicle video data in service of learning for navigation. That said, I think there are several important ways in which the paper needs to be improved before I would feel comfortable arguing for acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I'm willing to revise my score upwards if the authors are able address the issues laid out in the \"weaknesses\" section above.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "1Pktr4oC75U", "reviewer": "Reviewer_JJFZ", "summary": "This paper proposed a novel DRL framework to efficiently share multiple robots' samples. The proposed framework has two separate models: perception and dynamics model to learn the observation-reward map and the robot dynamics respectively. This double-model framework enable the agent to learn a wide range of observation while considering the current robot behaviors. The experiment was conducted in multiple environments and robots and clearly show the advantages of the proposed framework. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper clearly shows its motivation and contribution. It has a well-written introduction and related works.\n2. The approach section is well designed with figures.\n3. The experiment is attractive and informative. The support materials are helpful to understand the details of experiment.\n\nWeaknesses\nThe reviewer's concern are following:\n1. Sharing samples collected from multiple robots is a good idea. However, the reviewer wonders whether and how the image samples collected from multiple robots are suitable to guide a specific one in an unknown environment. From Fig. 3, the cameras were fixed in different position of different robots, therefore, the samples collected from some robots (drone) may never be seen by others (UAV). How did the experience of a drone turns to help a UAV in an area explored by the drone only? Although the results show a improvement, the generalization capability of perception model should be further addressed.\n2. For the dynamics model, the reviewer found that the state s, one input of f^{DYN} was set to empty (\"There is no inputted state\"). If so, the reviewer believes f^{DYN} may only learned very limited \"dynamics of each robot\" which may be insufficient to predict robots' movement. The author should further explain and discuss it with details (why set state to empty).\n3. The reviewer also find that the baseline approach had a poor performances when the sample collection and evaluation are both in Jackal (Urban). It should have a close performance to the proposed approach since there is no samples collected by other robot/environment. The author should give a reasonable explanation and make the comparison fair if necessary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper has clear contribution. It can be accepted once the pointing issues are addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "All issues have been pointed in weaknesses.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "1AE5psdnLc", "reviewer": "Reviewer_KezA", "summary": "The authors address the problem of autonomous navigation on heterogeneous robotic platforms. One of the limitations of many current approaches is the need to train an algorithm in the same specific platform in which it is intended to be used. To overcome this problem, HInt, a new model-based Deep Reinforcement Learning approach, is proposed. HInt has two main components: the dynamics model, trained with data from a specific robotic platform, and the perception model, trained instead with data collected using different sources. Combining together these two components, the resulting model is able to take into account the dynamics of the robots in which it is deployed while, at the same time, leverage heterogeneous dataset gathered by different platforms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is in general well written and easy to follow. The problems addressed by the work are relevant to the Robotics community and are clearly defined by the authors. The literature review is also adequate. The proposed approach is interesting and promising even if requires a real-world data collection, which can be time and resource expensive. The training procedure and model structure are also clearly described but it would be useful to better explain the optimization method used to learn the reward signal and to add a more formal definition of the overall algorithm. Finally, the experiments show the performances of the proposed approach against a recent state-of-the-art method [34] in different real scenarios. While the comparison with the other method demonstrates the superior adaptation capabilities of HInt, it would be interesting to test it against a version of itself trained with a limited set data. In this way, the importance of a heterogeneous dataset for the perception model training would be explicitly measured.\n\n[34] G. Kahn, P. Abbeel, and S. Levine. BADGR: An autonomous self-supervised learning-based navigation system. arXiv:2002.05700, 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is original, technically sound and clearly written. The proposed approach is interesting and my overall recommendation is positive.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1) To help the reader to better understand the training procedure, a brief description of the stochastic zeroth-order optimization method is suggested;\n2) Section 3.4, containing the algorithm summary, is useful as a recap, but should be accompanied by the algorithm pseudocode for better clarity;\n3) Three versions of HInt trained with a single source dataset (i.e., Urban, Office, and Industrial) could be compared against its final version and [34]."}}]

Author rebuttal:

Meta-review: This paper has quite mixed reviews. On the one hand, the experiments on real robots show the value of a decoupled dynamics/collision architecture in generalization. However, the reviewers raised concerns about generality, how exactly the model is trained, and more broadly and fundamentally, about why this is necessary when quite conventional approaches have been doing similar things in robotics for decades. Addressing these questions in the rebuttal would be very helpful for the subsequent discussion. 

--- post-rebuttal, some of the reviewers have raised their scores, and we are on a majority accept. Having looked at the authors' rebuttal, I believe they have answered most of the outstanding questions raised.
Decision: Accept (Poster)