Rough Terrain Navigation Using Divergence Constrained Model-Based Reinforcement Learning
Authors: Sean J Wang
Samuel Triest
Wenshan Wang
Sebastian Scherer
Aaron Johnson
Venue: CoRL2021 Poster (2021)
Keywords: Rough Terrain Navigation
Model-Based Reinforcement Learning
Model Uncertainty

Abstract: Autonomous navigation of wheeled robots in rough terrain environments has been a long standing challenge. In these environments, predicting the robot's trajectory can be challenging due to the complexity of terrain interactions, as well as the divergent dynamics that cause model uncertainty to compound and propagate poorly. This inhibits the robot's long horizon decision making capabilities and often lead to shortsighted navigation strategies. We propose a model-based reinforcement learning algorithm for rough terrain traversal that trains a probabilistic dynamics model to consider the propagating effects of uncertainty. During trajectory predictions, a trajectory tracking controller is considered to predict closed-loop trajectories. Our method further increases prediction accuracy and precision by using constrained optimization to find trajectories with low divergence. Using this method, wheeled robots can find non-myopic control strategies to reach destinations with higher probability of success. We show results on simulated and real world robots navigating through rough terrain environments.

Rough Terrain Navigation Using Divergence Constrained Model-Based Reinforcement Learning
Sean J. , Triest,, Wenshan Wang", Sebastian Scherer and Aaron M. Johnson'
DDppartment of Mechanical Engineering. 2 Rohotics Institute Camegie Mellon University Pittsburgh, PA, USA
Abstract: Autonomous navigation of wheeled robots in rough terrain environ- ments has been long standing challenge. In these environments, predicting the robot's trajectory is challenging due to the complexity of terrain interactions and the divergent dynamics that cause model uncertainty compound. This inhibits the robot's long horizon decision making capabilities and often lead to short- sighted navigation strategies. We propose moodll-based reinforcement learning algorithm for rough terrain traversal that trains a probabilistic dynamics model to consider the propagating effects of uncertainty. Our method increases prediction accuracy and precision by using tracking controller and by using constrained optimization find trajectories with low divergence. Using this method, wheeled robots can find non-myopic control strategies to reach destinations with higher probability of success. We show results on simulated and real world robots navi- gating through rough terrain environments.
Keywords: Rough Terrain Navigation, Model-Based Reinforcement Learning, Model Uncertainty
Introduction
Autonomous wheeled robots are useful for a variety of applications such as environmental monitor- ing, package delivery, and warehouse operation. Although these robots are capable of autonomously navigating through flat environments, rough terrain still poses challenge as it requires non-myopic decision making to avoid getting trapped. This is challenging as the interactions between the robot and the unstructured terrain are complex and difficult to model [1]. Furthermore, the highly diver- gent dynamics can cause modeling errors to compound and propagate poorly along trajectories.
Methods that rely on accurate handcrafted dynamics models for decision making, e.g. [2, 3, 4], often perform poorly when predictions contradict reality. Some methods aim combat this issue by formulating control policies that are robust against model uncertainty [5, 6, 7, 8). Others abstract the robot's dynamics by using traversability maps to provide heuristics for how difficult different regions are to traverse [9, 10, 11, 12, 13, 14]. However, these approaches are limited to coarse approximations of traversability and don't 't consider dynamic interactions.
Alternatively, reinforcement learning can be used for rough terrain traversal. Some methods use model-free reinforcement learning (MFRL) [15, 16, 17], where control policies are directly opti- mized. Others use model- reinforcement learning (MBRL) [18], where predictive dynamics models are learned and then used for decision making. Due to their higher sample efficiency, MBRL methods are generally more practical for real world systems where training data is limited and expen- sive to collect. However, MBRL methods often have poorer performance as their decision making processes may exploit model inaccuracies. Previous MBRL algorithms have addressed this issue by considering prediction uncertainty during decision making [19, 20], though characterizing pre- diction uncertainty for rough terrain navigation is challenging since the divergent dynamics cause model uncertainty to propagate poorly over long prediction horizons.
We propose a MBRL algorithm for rough terrain navigation, summarized in Sec. 3 and Fig. 1. Our algorithm's unique method of handling uncertainty allows the robot to more accurately optimize
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Closed-Loop Trejectory Prediction
Optimized Trajectory
Actual Trajectory
Closed Loop Probabilatic Dynamics
Closed Loop
Action
Model
8r+1
Pa (1 18,, oz)
Enjecttry Tracking Controller
Strack
Open-Loop Unconstrained
Closed Loop Constrained
80,, O02 OT
Diveegence Comminainend
Starting State Feed Forword Actions
Optimization
Closed-Loop
Trajectories
Figure 1:
Left: An overview of the method. The probabilistic dynamics model is trained using multistep loss that considers how uncertainty propagates. The trajectory tracking controller is used to predict a distribution of closed-loop trajectories. The divergence constrained optimizer is used to find closed-loop trajectory with low divergence. Center: Example trajectory distribution found using our method (green) and a trajectory distribution found without using closed-loop predictions or constrained optimization (red). Right: The actual results of running the two optimized trajectories.
longer trajectories. Like previous methods [19, 20]. our method performs trajectory optimization using learned probabilistic dynamics models to predict trajectories and characterize prediction un- certainty, However, we train the probabilistic dynamics model using a multistep loss that considers how model uncertainty propagates along trajectories (Sec. 4). We perform closed-loop trajectory prediction, where the capability of the robot's trajectory tracking controller is considered during trajectory prediction (Sec. 5). The use of multi-step loss and closed-loop trajectory prediction decreases long horizon trajectory prediction uncertainty and allows the robot to be more selective during optimization. Our algorithm uses constrained optimization in high divergence (Sec. 6), similar to Convergent Planning [8]. Through simulation and hardware experiments, we show that these improvements allow the robot find non non-myopic trajectories with high prediction accuracy and precision, increasing the likelihood of successful navigation (Sec. 7).
avoid trajectories that result
2 Related Works
Probabilistic Ensembles with Trajectory Sampling (PETS) [19] aims to make MBRL more robust to modeling errors by characterizing uncertainty with probabilistic ensemble models and propogating uncertainty with trajectory sampling. PETS optimizes trajectories based on expected rewards and performs well on benchmark tasks [21]. However, using PETS rough terrain is challenging due to the high variance in predictions, which makes it difficult to estimate expected reward. Furthermore, high reward expectation with high variance does not mean the robot will actually perform well.
Kahn et al. [20] use uncertainty in a model-based fashion for collision avoidance in navigation. They train an ensemble of models (via variational dropout [22]) to predict the probability of a col- lision. They then use this uncertainty to augment their task-specific cost function via a risk-averse prediction of the collision probability, which factors in the uncertainty of the collision predictor en- semble. This collision probability used their reward function, which solved using MPC. This approach is shown to be effective with real-world experiments.
Recent work by al. [23] takes more general approach uncertainty-aware MBRL by learning an ensemble of dynamics models and augmenting arbitrary cost functions with a scaled penalty based on the uncertainty of the ensemble. The authors show that such an approach allows them to perform offline reinforcement learning, or reinforcement learning without access to the environment.
3 Method Overview
We formulate the rough terrain navigation problem as a Markov decision process (MDP). Our prob- lem is defined as P,,) where S is the state space, A is the action space, P(s' |s a) is the non-deterministic discrete time transition dynamics, C(s, a) the cost function, and 7 is the discount factor. We assume that the state includes terrain features (e.g. terrain height map).

---

Our approach, Fig. 1, falls under the MBRL paradigm. We train probabilistic neural network model the unknown transition dynamics P using previously collected data where d; = (so.... .., ST: do,,.., aT-1) is a state-action trajectory. We train the model to capture both aleatoric and epistemic uncertainty using a multistep training loss that considers how model uncer- tainty propagates along trajectories (Sec. 4). The trained probabilistic model is used to predict distribution of the robot's trajectory under the influence of a tracking controller (Sec. 5). Finally, low cost trajectory to the goal is generated using constrained optimization. The solution is con- strained based on the probabilistic model to avoid steps where the dynamics significantly diverges to ensure that the tracking controller can actually achieve the trajectory (Sec. 6).
{do. .... das ).
4 Probabilistic Dynamics Models
We propose a method for training probabilistic dynamics model to predict long horizon trajectories. Similar to [19], we train the model capture both epistemic and aleatoric uncertainty, however here we propose a multistep loss that considers how prediction uncertainty propagates along a trajectory.
4.1 Aleatoric Uncertainty
In the learned model we aim to capture aleatoric uncertainty by modeling the true transition dynam- ies P with a probabilistic neural network [24] parameterized by 9. Given a state and action ag time t, the model predicts a probability distribution of possible next states. For our implementa- tion, the network predicts the mean and log-variance of an uncorrelated gaussian distribution. The estimated likelihood of next state 8141 conditioned on 8, and or is denoted as po (#++1 |8, at). The training method in [19] uses a single step likelihood loss. However, this loss fails to consider how prediction uncertainty propagates when multiple single step predictions are chained for trajectory prediction. Furthermore, [25, 26] describe how states generated from earlier predictions may fall out of the distribution of training data, leading to poor predictions for later steps in trajectory. We adapt [25, 26] to non-deterministic environments by proposing probabilistic multistep loss that considers the propagation of model uncertainty instead of model error.
The multistep loss we use aims to minimize the negative log likelihood of states conditioned on all prior actions and the starting state of the trajectory, i.c. (ms(s A, 0) == Pog where A = (8o, 40. .... 01- The likelihood of states 81; .1:... conditioned on A can be found as follows:
(0 (0 (so, aa)
po A) = E (02 an)]
(s:] AA) = Es (4) (se (s- a.-))
Calculating Lms exactly is intractable due to the expectations required. Instead we define upper bounds that can be used during training instead, denoted as L. where N N,
N
C (st, A, == log N
po p0 at-1)
(2)
where 8 is sampled recursively from po(s aj-1) for j t. As N approaches infinity, C.N approaches the true multistep loss, L.s-- Furthermore, due to Jensen's 's inequality:
E [[C (st. A, 0)] > E [c.. (st, A,0] 2 2 (ms
(3)
As such, minimizing C.N. using any stochastic optimization methods such Stochastic Gradient Descent (SGD) [27] will effectively also minimize the true loss. Any value could be used for N, however larger values give more realistic estimates at the expense of increased computation.
4.2 Epistemic Uncertainty
Epistemic uncertainty defined as model uncertainty due to a poorly distributed or lack of training data. In the context of deep learning, it is typically quantified using a prediction disagreement
3

---

metric via an ensemble of neural network models. Some popular methods use explicit ensembles of models [19], while other methods such as Bayesian neural networks [28] or hypernetworks [29, 30] parameterize models via uncertainty in the weights of the neural network.
In work, we use variational dropout characterize epistemic uncertainty [31, 22]. With this approach, inputs to the neural network's 's fully connected layers are randomly dropped (set to zero) with some probability. We define θ the parameters of the neural network dynamics model with no dropout applied. Using dropout on this model will effectively result in distribution of models. We denote the probability of sampling model, parameterized by A as qe
During training, we randomly sample parameters from ge (0) and calculate the multistep training loss described in Sec. 4.1. We then update θ to minimize training loss. The complete model training procedure is summarized in Algorithm
5 Closed-Loop Trajectory Prediction
Imprecise trajectory predictions are problematic for two reasons. First, predicted trajectory distribu- tions with high variance provides little insight into the robot's true outcome. Second, the standard error of Monte Carlo predictions scales linearly with prediction variance and inversely with the square root of number of samples. This means that using Trajectory Sampling [19) may require an intractable number of samples when prediction variance is high. We observe that it is common to use a low-level trajectory tracking controllers to correct for deviations of the true trajectory from the desired (predicted) trajectory. Our method aims predict robot trajectories under the influence of low-level trajectory tracker controllers. Doing so allows prediction of less divergent robot trajectory distributions and better captures the capabilities of the closed-loop robot control system.
We assume that the controllers take the form:
(4) where S0.-..; 8, the robot's actual states, and d (So. 8T, ag ..... Ro...... a1- is reference tra- jectory containing reference states and feed forward actions. a is the combination of the planned, feed-forward action a and the corrective action applied to the robot to track the reference trajectory.
ag = ā. + ftrack (So. St. d),
Training a dynamics model to directly predict the robot's closed loop behavior is difficult as the controller's 's inputs are high dimensional and unbounded. Instead, our method first predicts nominal reference trajectory given a starting state and feed forward actions,
B [[s4+1 ~ (5 az)]
(5)
Our model outputs the predicted distribution's 's mean which is the expected value above.
A distribution of closed-loop trajectories is then predicted by propagating a set of particles for- ward in time using the probabilistic dynamics model from Section 4. At each time step, corrective actions for each particle calculated using ftrack based on the predicted nominal reference trajec- tory and the feed forward actions. We denote the predicted closed-loop trajectory distribution as
δ = {dd) idd,,...?? d where d' ($) ..... st., ab. .... 1) ) is particle's state-action trajectory. The procedure for predicting closed-loop trajectories summarized in Algorithm 2.
Divergence Constrained Optimization
In MBRI., low cost trajectories are found via trajectory optimization through the learned model. However, naively minimizing cost allows the optimizer to exploit modeling errors, resulting in tra- jectory with low predicted cost, but high cost under the true dynamics. Previous work [23] prevented the optimizer from exploiting modeling errors by penalizing trajectories with high prediction uncer- tainty. This approach may be problematic as the penalty opposes the cost objective. For example, if a robot remains stationary, its trajectory has low prediction uncertainty, but high task cost.
Instead of penalizing uncertainty, our method uses constrained optimization to find low cost tra- jectories with divergence bounded by some U, as trajectory trackers are usually effective as long as the true trajectory does not diverge too much from the reference. Values for U are dependant on the trajectory tracker used, our experiments, we selected U based on the tracker's observed capabilities.

---

Algorithm 1: Probabilistic Model Training
Algorithm 2: Closed-Loop Prediction
for number of epochs do
Input: Starting State 80,
4-
Feed-Forward Actions
for training batch size(b) do
// Predict Nominal Trajectory
G ~ pe(()
5o f 90 for t [1 do
sample
{s0, ao ...,,.., aT-
8ü 3 f- 80
8; - Ess ~ a-11
for 1 E [1, do
Closed-Loop Prediction for i do
for E N] do
8 ~ P0 8 411, N
0f ~'>pe(1) 8). ab 80, 8o
N Σラ i= (s a(-1)
for l E [17]] do
C log
8) Po Po 8 8 0 1)1
- update using gradient Vet f
0j t a + Strack ()).. Sl,d)
Algorithm 3: Divergence Constrained Optimization
Input: Starting State So
ao. .... 0T random initialization
for number of Augmented Lagrangian iterations do
for number of gradient descent iterations do
ò - predict nominal and closed-loop trajectories using Algorithm 2
max. 1
8el1
+ c(d) + À maz uu U,0)
40. aT + update using gradient ag C
Increase À
Given a nominal trajectory and predicted closed-loop trajectory distribution D, we define diver- gence as follows:
u(d, D) max 1 -1 11 8
The constrained optimization problem we aim to solve is defined as follows:
min To...,TT--
C(d)
s.t. u( <U
We use Augmented Lagrangian optimizer to solve (7), shown in Algorithm 3. During opti- mization, the estimated divergence is thresholded, mar uu U, 0j. scaled by a penalty factor and added to the nominal trajectory cost. À starts small, but is progressively increased.
7 Experiments
We compare our navigation method against others in simulation and the real world. We used Py- Bullet [32] for our simulation environment and MuSHR robot [33] for our real world experiment. Simulation training data was collected using random actions
noise) random environments (similar to Figure 3). Real world data collected using human selected actions (with Gaussian noise) in one environment (similar to Figure 4a with different arrangement). Actions corresponded to motor commands (throttle and steering). Training data consisted of terrain height maps and state-action trajectories. Simulation states included pose (in (3)) and body velocity 6 twist vector). Real world states included pose (in SE())) estimated using motion capture.
We trained the dynamics models to predict state transition probability, given an action and an obser- vation. Simulation observations included robot tilt (R&3 gravity vector) and body velocity (R& twist vector). State transitions consisted of change pose (in SF)))), and new body velocity. Real world observations included previous state transition. State transitions included change in pose (in SE(2)).

---

(a) Single Step
(b) Single Step Tracked
(c) Multistep
(d) Multistep Tracked
Figure 2: Predicted robot trajectories using different prediction methods The predicted trajectories shown in red, and the true trajectories are shown in green.
All observations also included a robot centric terrain height map cropped from the world map. Tra- jectories were predicted by iteratively calculating observations given current states, sampling state transitions from model predictions, and transitioning current states to next states. The models were trained using our proposed multistep loss, L with Algorithm using 5 step prediction horizons.
We compared three different trajectory optimization frameworks: 1) an optimizer that only mini- mizes the goal cost, (d).. 2) an optimizer that adds a scaled divergence penalty, C(d) + Au(d, D), and 3) the proposed divergence constrained optimizer (Algorithm 3). We used squared distance to the goal as optimization and evaluation cost. The number optimization gradient steps were kept consistent between methods. We tracked optimized trajectories using simple PD controller (de- tailed definition of ferack provided in Sec. B). The effects of the tracker were considering during op- timization using closed- loop predictions (Algorithm 2). We treated the threshold U for constrained optimization as a hyperparameter, which we adjusted based on the observed abilities of the tracker.
Trajectory Prediction Accuracy
In simulation, we evaluated the effects of using multistep loss and making closed-loop predictions on trajectory prediction accuracy. We trained one model using a single-step log-likelihood loss, and another model using proposed multistep loss, L with Algorithm We used these two models to estimate the probability distribution the robot's s state the end of a 32 step trajectory by propagating 512 particles using the trained models and averaging the predicted distribution from all particles. We simulated the robot's true trajectory and recorded the log-likelihood of the true state given the estimated distributions. We repeated this process with the closed-loop trajectory predictions using Al Algorithm 2. The average log-likelihoods over 100 trials are listed in Table 1, which shows improvement from both the multistep loss and the tracking controller. The true trajectories and some example particle trajectories are shown in Fig. 2.
Navigating Over Simulated Terrain
We compared the performance of different navigation methods for the task of driving randomly generated terrain with large obstacles to a goal 7m away. Besides the three aforementioned opti- mization frameworks,
also included A path planning method with tracking using the terrain cost from [34], which considered local terrain smoothness, slope, and curvature. We also evaluated the sensitivity of the optimizers trajectory prediction accuracy by performing closed-loop pre- dictions. For 250 trials, Table 2 shows each method's success rate, defined as ending within 0.5m of the goal. Table shows the actual cost and standard deviation for each method with tracking.
Overall, we found that the constrained optimization has the highest success rate and lowest final cost. Furthermore, seeding the constrained optimization with the planner resulted in an even higher
Single-Step
Single-Step Tracked
Multistep
Multistep Tracked -31.1 + 48.5
Log-I Likelihood -692.7 + 470.0 -242.5 + 364.6 -110.0 + 166.6
Table 1: Mean and standard deviation of prediction log- likelihod for 100 trials.

---

A Planner 72 %
No Uncertainty
Penalty 60 %
Constraint
Tracking
66 %
82 %
No Tracking
62
32 %
64
Table 2: Random terrain results. Success percentage over fifty random terrains, using the same
terrain for all methods. The A planner does not provide actions, so tracking was required.
A Planner
No Uncertainty
Penalty 0.9561
Constraint
Cost (Squared distance to goal) Divergence
2.0892
1.8792 0.2350
0.9109
0.2039
0.1220
0.1529
Table 3: Results of random terrain testing. Despite having higher divergence than the penalty method, the constrained method has lower final cost.
success rate (88%), suggesting that future work should address issues with local minima. As pected, the penalty method has the lowest divergence since directly minimizes it, however this did not translate to higher success rate. We also found that prediction accuracy was especially important to uncertainty aware optimization methods. Not including tracking during prediction significantly decreased the success rate of both the penalty and constrained optimization methods.
considered, the optimizer exploits predic- tion error and finds risky solution with high divergence. When executed, the robot collides with an obstacle and rolls over. Conversely, the penalty method finds an overly conservative trajectory since the optimizer minimizes divergence at the expense of task cost. The trajectory has low divergence, but does not reach the goal. The divergence constrained method strikes middle ground finding safe low-divergence trajectory to the goal. This behavior was exhibited throughout most of our experiments. Sec E shows more trials and examples.
Figure shows one example trial. When divergence
Real World Navigation
We also compared the three optimization frameworks in the real world, Fig. 4. The robot was tasked with navigating across obstacle field to goal 5m away. The field consisted of traversable short obstacle, non-traversable tall obstacle, and a ramp whose traversability depended on the robot's maneuver. Table 4 shows the actual cost and success rate for 10 trials.
When no uncertainty was considered, the optimizer exploited nominal prediction errors and found impossible trajectories through the tall obstacle. For 9 out of 10 trials, the robot got stuck on the tall obstacle. For trial, the true trajectory deviated from the nominal enough for the robot to catch the edge of the ramp and make it to the goal. When penalizing divergence, the optimizer avoided the ramp 6 times to minimize divergence. When using divergence constrained optimization, the robot made it over the ramp 8 times, and average got closer to the goal.
Discussion & Conclusion
We propose a novel MBRL algorithm for rough terrain traversal. The main advantage of our method is its ability to accurately optimize long horizon trajectories and thus perform non-myopic decision making. Our method uses divergence constrained optimization to find trajectories with low diver- gence, ensuring that the robot can faithfully execute the trajectory. However, the benefits of using divergence constrained optimization are less apparent when trajectory predictions are inaccurate. Therefore, our method trains dynamics models to consider uncertainty propagation and uses those models to predict closed-loop trajectories.
No Uncertainty
Penalty Constraint
Actual Trajectory Cost (Squared distance to goal)
3.57
4.11
1.68+1.42
Success Rate Table 4: Real world navigation results. Actual cost values mean and standard deviation for 10 trials. Success is defined as getting within 1.5m of the goal.
10%
40%
70%
7

---

(a) No uncertainty
(b) Uncertainty penalty
(c) Uncertainty constraint
Figure 3: One trial in random terrain. Predicted nominal trajectories shown in red, and the true trajectories are shown in green. The vertical green lines mark the goal.
(a) Test Environment
(b) No Uncertainty
(c) Penalty
(d) Constraint
Figure 4: One trial of navigating a real world robot across obstacle field. The goal (red cross), predicted closed-loop trajectories (red lines with last time step marked with dots), predicted nominal trajectory (black line), and actual robot trajectory (green line) shown over height map.
Additionally, our method is practical to implement and scalable. First, our method lends itself to offline settings where the agent unable interact with the environment online. The divergence constrained optimization naturally avoids actions that result in high epistemic uncertainty. As result, the robot's policy only performs maneuvers that are well supported by the dataset. Second, our algorithm is computationally feasible. Although multiple trajectory predictions are required to propagate uncertainty, this process is easily parallelized for GPU computation. Furthermore, our approach can allow for a much longer horizon when using model predictive control and thus more time for optimization. In our experiments, the algorithm could accurate optimize trajectories with horizons long enough (roughly 100 steps) not necessitate model predictive control. Lastly, method is highly modular. Since the dynamics model is agnostic to the tracker used and task's rewards, these components can be changed online without the need to retrain the model.
In the future, improvements the constrained optimization could increase performance further. In our experiments, the constrained optimization's 's solution reached local minimum and that seeding the optimization with trajectories from a planner resulted in higher success. Besides seeding, the optimization could potentially be improved with the use of motion primitives.
In the experiments, robot observations included robot-centric terrain height maps cropped from world height maps. We believe the assumption of world height map knowledge is reasonable since common to either build maps priori or online from sensor data. However, these maps are often imperfect, especially in areas with sparse sensor coverage. Future work will incorporate map uncertainty into trajectory predictions.
Finally, we believe that this approach could also be useful for other tasks. Sec. A describes how the algorithm could be extended to any Partially Observable Markov Decision Process. This could be especially promising for tasks with stochastic dynamics that require non-myopic decision making.

---

References
[1] K. Nagatani, G. Ishigami, and Y. Okada. Modeling and control robots on rough terrain. In Springer handbook of robotics, pages 1267-1284. Springer, 2016.
[2] A. Kelly and A. Stentz. Rough terrain autonomouss mobility part 2: An active vision, predictive control approach. Autonomous Robots, 5(2):163-198, 1998.
[3] T. M. Howard and A. Kelly. Optimal rough terrain trajectory generation for wheeled mobile robots. The International Journal of Robotics Research, 26(2):141-166, 2007.
[4] T. Howard, M. Pivtoraiko, R. A. Knepper, and A. Kelly. Model-predictive motion planning: Several key developments for autonomous mobile robots. IEEE Robotics & Automation Mag- azine, 21(1):64-73, 2014.
[5] L. P. Kaelbling, M. L. Littman, and Cassandra. Planning and acting in partially observ- able stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
[6] A. T. Schwarm and M. Nikolaou. Chance-constrained model predictive control. AIChE Jour- nal, 4543):1743-1752, 1999.
[7] G. Kewlani, G. Ishigami, and K. lagnemma. Stochastic mobility-based path planning in uncer- tain environments. In IEEE/RSJ International Conference Intelligent Robots and Systems, pages 1183-1189,0001
[8] A. M. Johnson, J. E. King, and S. Srinivasa. Convergent planning. JEEE Robotics and Au- tomation Letters, 1(2):1044-100 2016.
[9] P. Papadakis. Terrain traversability analysis methods for unmanned ground vehicles: A survey. Engineering Applications of Artificial Intelligence, 26(3): 2013.
[10] M. Agrawal, K. Konolige, and R. C. Bolles. Localization and mapping for autonomous nav- igation in outdoor terrains: A stereo vision approach. In IEEE Workshop on Applications of Computer Vision, pages 7-7, 2007.
[11] D. Maturana, P.-W. Chou, M. Uenoyama, and S. Scherer. Real-time semantic mapping for autonomous off-road navigation. In Field and Service Robotics, pages 335-350. Springer, 2018.
[12] N. Hirose, A. Sadeghian, M. Vázquez, P. Goebel, and S. Savarese. GONet: A semi- semi-suppreiied deep learning approach for traversability estimation. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3044-3051, 2018.
[13] R. O. Chavez-Garcia. J. Guzzi, L. M. Gambardella, and A. Giusti. Learning ground traversabil- ity from simulations. IEEE Robotics and Automation letters, 16). 2018
[14] D. Kim, Sun, S. M. Oh, M. Rehg, and A. F. Bobick. Traversability classification us- ing unsupervised on-line visual learning for outdoor robot navigation. In IEEE International Conference on Robotics and Automation, pages 518-525, 2006.
[15] K. Zhang, F. Niroui, M. Ficocelli, and G. Nejat. Robot navigation of environments with un- known rough terrain using deep reinforcement learning. In IEEF International Symposium on Safety, Security, and Rescue Robotics, pages 1 1-7, 2018.
[16] Josef and A. Degani. Deep reinforcement learning for safe local planning of a ground vehicle in unknown rough terrain. IEEE Robotics and Automation Letters. 5(4):6748-6755 2020.
[17] Blum and K. Yoshida. PPMC RL training algorithm: Rough terrain intelligent robots through reinforcement learning. arXiv preprint arXiv:2003.02655, 2020.
[18] G. Kahn, P. Abbeel, and S. Levine. BADGR: An autonomous self supervied Baaming-based navigation system. IEEE Robotics and Automation Letters, 602):1312-139, 2021.
[19] K. Chua, R. Calandra, R. McAllister, and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. arXiv preprint 1805. 2018.

---

[20] G. Kahn, A. Villaflor, V. Pong, P. Abbeel, and S. Levine. Uncertainty- aware reinforcement learning for collision avoidance. arXiv preprint arXiv:/702.011 2017.
[21] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, Schulman, J. Tang, and W. Zaremba. OpenAl gym. arXiv preprint arXiv: 2016.
[22] Y. Gal and Z. Ghahramani. Dropout bayesian approximation: Representing model uncer- tainty in deep learning. In International Conference on Machine Learning, pages 1050-1059. PMLR, 2016.
[23] Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. MOPO: Model- based offline policy optimization. arXiv preprint arXiv: 13239 2020.
[24] D. F. Specht. Probabilistic neural networks. Neural networks, 3(1):109-118 1990.
[25] E. Talvitie. Self-correcingg models for model-based reinforcement learning, In AAAI Confer- ence on Artificial Intelligence, volume 31()), 2017.
[26] E. Talvitie. Model regularization for stable sample rollouts. UAI, pages 780-789, 2014.
[27] S. Amari. Backpropagation and stochastic gradient descent method. Neurocomputing, 5(4-5): 185-196, 1993.
[28] C. Blundell, Comneiiee, Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613-1622. PMLR, 2015,
[29] D. Ha, A. Dai, and Q Le. Hypernetorrss. arXiv preprint arXiv:1609.09106, 2016.
[30] D. Krueger, C.-W. Huang, R. Islam, R. Turner, A. Lacoste, and A. Courville. Bayesian hyper- networks. arXiv preprint 1rXiv:1710.04759, 2017.
[31] D. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameteriza- tion trick. arXiv preprint arXiv: 150..05555 2015.
[32] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybulle. org, 2016-2019
[33] S. Srinivasa, P. Lancaster, J. Michalove, M. Schmittle, C. Summers, M. Rockett, J. R. Smith, Choudhury, C. Mavrogiannis, and F. Sadeghi. MuSHR: low-cost, open-source robotic racecar for education and research. arXiv preprint 1908::190.00000 2019.
[34] P. Fankhauser, M. Bjelonic, C. D. Bellicoso, T. Miki, and M. Hutter. Robust rough-terrain locomotion with a quadrupedal robot. In IEEE International Conference on Robotics and Automation, pages 5761-5765 2018
[35] E. V. Kumar and J. Jerome. Robust lqr controller design for stabilizing and trajectory tracking of inverted pendulum. Procedia Engineering, 64:169-178, 2013.
[36] C. Liu, J. Pan, and Y. Chang. Pid and lar trajectory tracking control for an unmanned quadrotor helicopter: Experimental studies. In 2016 35th Chinese Control Conference (CCC), pages 10845-10850. IEEF, 2016.
[37] A. Nagariya and S. Saripalli. An iterative lqr controller for off-road and on-road vehicles using a neural network dynamics model. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 1740-1745. IEEE, 2020.
[38] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for plamning from pixels. In International Conference on Machine Learning, pages 2555-2565. PMLR, 2019.
[39] D. Ha and J. Schmidhuber. World models. arXiv preprint arXiv: 1803.10122111 2018
10


Reviews:
[{"review_id": "tvYzn3sacRD", "reviewer": "Reviewer_HWwA", "summary": "Motivated by the problem of building autonomous navigation systems capable of allowing wheeled robots to traverse rough terrain, the authors propose a new system based on model-based reinforcement learning. Specifically, the authors propose to learn a probabilistic dynamics model that provides better predictions of future trajectories (given open-loop control sequences) and also a new planning method in order to select good trajectories. The authors provide interesting real-world experiments that highlight the efficacy of their approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\n(S1) The authors study a compelling and difficult problem in robotics, and clearly showcase the benefit of using machine learning in order to approach it.\n\n(S2) The proposed approach (the particular loss function and sampling-based approach to evaluating/deploying) seems novel and has the potential to be quite impactful.\n\n(S3) The authors have performed impressive real-world experimentation.\n\nWEAKNESSES\n\n(W1) The experiment details and metrics used need to be re-hashed.\n\n(a) The authors need to provide more details regarding how training data was collected. In the current submission, only the second paragraph of Section 7 seems to discuss this, and the details are sparse. How much training data must be collected? Are \"random actions\" really all that are required? It seems doubtful that such an approach would generate diverse enough coverage of the state space to be useful.\n\n(b) \"Final cost\" is a reported metric, but its unclear as to what that really means for the task at hand. The authors should use a more easily-interpreted metric (time to destination?) in order to measure performance, as \"cost\" is really whatever the authors define it to be.\n\n(W2) It's not clear from the submission as to what role partial observability plays in the proposed technique. For example, does the proposed approach require a full global map in order to be used? What if this information is not available?\n\nMINOR COMMENTS\n\n(MC1) It's unclear from Figure 3 as to what the difference between (c) and (d) is. This should be highlighted better in the caption.\n\nPOST-DISCUSSION COMMENTS\n\nThe authors have addressed several of my concerns, especially with respect to the experimental details section. The paper addresses an important topic, and the work seems strong experimentally.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I've recommended \"weak accept\" because I feel that the paper considers and important problem and appears to make a reasonable contribution towards solving it. I do think the authors have a few points to address, but overall the paper seems strong.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "(see \"weaknesses\" section above)", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "kPYKmn3vvH9", "reviewer": "Reviewer_8wj7", "summary": "This paper is investigating the challenge of using model-based reinforcement learning on rough terrain navigation problems with wheeled robots. The challenge here is that in rough terrain environments with wheels there's a number of different changes in contacts and other dynamics that make prediction difficult and the overall uncertainty of being able to predict into the future larger. The method in the paper then proposes a combination of methods that train over sequences of states into the future to help capture the compounding uncertainties better and preferences in the optimization procedure to avoid using predicted states that have high uncertainty. This combination of methods appears to result in more controled and accurate behavior driving a wheeled robot in a simulated environment and in the real world.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n- The method proposed in the paper appears to be well-founded and the analysis in the paper also shows the positive effects of using each of the components in the overall method.\n- The changes for the method are also clean and simple and easy for other people to be able to adopt making the reproduction of the work in this paper simple for others to use.\n\nCons\n- However then part of the challenge here is the methods that are used in this paper have somewhat been explored in other prior work. Then the author in this case needs to better highlight the importance of the unique combination of these methods to be able to solve these particular tasks.\n- This method is also primarily validated on the novel wheeled terrain navigation tasks that are shown in the paper. Therefore in order to compensate for some of the cons in the paper, it is important to run the proposed method across the environments that are explored in the PETs paper. This will show that the method also outperforms some other model-based methods and that it's not the case that the new method only performs well on the new custom environment that was created for this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "See the above comments.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See the above comments.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "EJoxoCmy2kl", "reviewer": "Reviewer_79eD", "summary": "This paper presents a method to control wheeled robots to navigate across rough terrain. Uncertainty aware models are trained to model the dynamics and a tracking controller is used to improve prediction accuracy. An constrained optimization is the used to find trajectory with low divergence.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n\n1. A model learning algorithm that takes uncertainty into account.\n\n2. A divergence constrained optimization algorithm to find suitable trajectory.\n\n3. Demonstration on a real robot.\n\nCon:\n\nNothing major.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a method to control a wheeled robot to navigate across rough terrain. New algorithms are proposed to address issues common in model-based RL and the experiments are well executed, including comparison to alternatives and real robot tests.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I think the paper is well written and the proposed method is well executed. No major issues need to be addressed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "5HJNjHfu1f", "reviewer": "Reviewer_stKS", "summary": "The paper presents a model-based RL (MBRL) approach that combines a stochastic state transition model and uncertainty-constrained optimization. \nThe paper claims to have three new ideas. (1) considering the propagation of model uncertainty along the trajectory (2) combining low-level feedback controller in the trajectory prediction (3) Constrained optimization for uncertainty-bound planning.\n\nThe proposed method uses a stochastic state transition model p(s_{t+1}|s_{t}, a_{t}) to predict the evolution of the dynamics. The stochastic model is able to quantify the Aleatoric uncertainty, and the Epistemic Uncertainty is measured by using stochastic dropout. In addition to the naive trajectory prediction, they included a low-level tracking controller (denoted as f_{track} in the paper) in the prediction loop to reduce the variance (This is unclear to me if this is a valid approach).\nThe estimated uncertainties are then used for planning. Using a constrained optimization solver, they generate the action sequence with the thresholded estimated divergence. \n\nThe approach is tested on a wheeled robot both  in simulation and in the real-world. The experimental results & ablation results show that the proposed approach improves the success rate and can be deployed on the real robot.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The presented approach is similar to most of the uncertainty-aware MBRL methods except for the \"Closed-Loop Prediction\" component. I'm unsure about the validity of the \"Closed Loop Prediction\" part as I argued in the Issues.\n- Several important technical details are missing. I could not find any information about the function \"f_{track}\" which is claimed to be one of the key components of this work. I could not find how the threshold U is defined. As the approach is aimed at the robotics application, I would also be interested in the computational efficiency and sample efficiency of the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I admire the effort put into this paper with the real-world experiment and implementation of the whole pipeline. However, the paper seems a bit incomplete. It lacks technical details and some of the claims/ideas are not supported well. (In the initial submission) ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- Minimizing loss function in Eq. 2 maximizes the log-likelihood of the state transitions per each step given sampled previous states. \nUsing a stochastic state transition model and penalizing the whole trajectory prediction is a widely used idea and cannot be seen as a novel idea. For one example, this paper:  \"Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic\"  ([https://arxiv.org/pdf/1901.02705.pdf](https://arxiv.org/pdf/1901.02705.pdf)).\n- The comparison between the \"Penalty\" method and \"Constraint\" is not valid. The latter is implemented as an augmented Lagrangian method. The problem is that the augmented Lagrangian optimization is basically the same as the Penalty method in principle. So the paper is comparing a penalty method with a fixed scale to a penalty method with increasing scale.\n- I could not find any information about f_{track} in the submission. How is f_{track} defined? Is it only conditioned on the previous states and actions? What happens if the tracking performance differs depending on the terrain conditions?\n- The authors claim that the Influence of the low-level controller is considered by looping f_{track} in the trajectory prediction. This is unclear to me. Isn't the influence already learned by the state transition model? The proposed Closed-Loop Prediction can be seen as applying the same controller twice per state transition.\n- Typo line 155: does not \"divergence\" too much."}}]

Author rebuttal:

Meta-review: The paper a model-based RL (MBRL) approach that takes into account the effect of uncertainty during the planning process. All reviewers agree on the importance of the problem and the good quality of the contribution. 

- The paper will benefit from clearly highlighting its key contributions. The paper proposes a combination of improvements in MBRL methods; however, some of these variants have been explored in the literature before. What unique advantages does the proposed combination have? Where are these gains coming from? Which part of the combination? More ablation studies to answer these questions might significantly strengthen the paper.

- A clear discussion on the scalability and applicability of the proposed approach, especially during the uncertainty propagation and the planning phases. 

- Demonstration of the proposed approach in simulation (and/or in experiments) on diverse systems and environments will help in understanding the advantages, the key limitations, and the applicability of the proposed approach compared to the state-of-the-art MBRL methods.

Besides addressing the major issues mentioned above, the authors should also revise the manuscript according to the other clarifications
requested and the suggestions provided. 

===== Post rebuttal =====

The authors have sufficiently addressed reviewers' concerns; specifically, the discussion on the related work was particularly helpful. I recommend an acceptance.

Decision: Accept (Poster)