Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a model-based RL (MBRL) approach that combines a stochastic state transition model and uncertainty-constrained optimization. 
The paper claims to have three new ideas. (1) considering the propagation of model uncertainty along the trajectory (2) combining low-level feedback controller in the trajectory prediction (3) Constrained optimization for uncertainty-bound planning.

The proposed method uses a stochastic state transition model p(s_{t+1}|s_{t}, a_{t}) to predict the evolution of the dynamics. The stochastic model is able to quantify the Aleatoric uncertainty, and the Epistemic Uncertainty is measured by using stochastic dropout. In addition to the naive trajectory prediction, they included a low-level tracking controller (denoted as f_{track} in the paper) in the prediction loop to reduce the variance (This is unclear to me if this is a valid approach).
The estimated uncertainties are then used for planning. Using a constrained optimization solver, they generate the action sequence with the thresholded estimated divergence. 

The approach is tested on a wheeled robot both  in simulation and in the real-world. The experimental results & ablation results show that the proposed approach improves the success rate and can be deployed on the real robot.
strengths_and_weaknesses: - The presented approach is similar to most of the uncertainty-aware MBRL methods except for the "Closed-Loop Prediction" component. I'm unsure about the validity of the "Closed Loop Prediction" part as I argued in the Issues.
- Several important technical details are missing. I could not find any information about the function "f_{track}" which is claimed to be one of the key components of this work. I could not find how the threshold U is defined. As the approach is aimed at the robotics application, I would also be interested in the computational efficiency and sample efficiency of the proposed approach.
summary_of_recommendation: I admire the effort put into this paper with the real-world experiment and implementation of the whole pipeline. However, the paper seems a bit incomplete. It lacks technical details and some of the claims/ideas are not supported well. (In the initial submission) 
issues: - Minimizing loss function in Eq. 2 maximizes the log-likelihood of the state transitions per each step given sampled previous states. 
Using a stochastic state transition model and penalizing the whole trajectory prediction is a widely used idea and cannot be seen as a novel idea. For one example, this paper:  "Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic"  ([https://arxiv.org/pdf/1901.02705.pdf](https://arxiv.org/pdf/1901.02705.pdf)).
- The comparison between the "Penalty" method and "Constraint" is not valid. The latter is implemented as an augmented Lagrangian method. The problem is that the augmented Lagrangian optimization is basically the same as the Penalty method in principle. So the paper is comparing a penalty method with a fixed scale to a penalty method with increasing scale.
- I could not find any information about f_{track} in the submission. How is f_{track} defined? Is it only conditioned on the previous states and actions? What happens if the tracking performance differs depending on the terrain conditions?
- The authors claim that the Influence of the low-level controller is considered by looping f_{track} in the trajectory prediction. This is unclear to me. Isn't the influence already learned by the state transition model? The proposed Closed-Loop Prediction can be seen as applying the same controller twice per state transition.
- Typo line 155: does not "divergence" too much.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: Motivated by the problem of building autonomous navigation systems capable of allowing wheeled robots to traverse rough terrain, the authors propose a new system based on model-based reinforcement learning. Specifically, the authors propose to learn a probabilistic dynamics model that provides better predictions of future trajectories (given open-loop control sequences) and also a new planning method in order to select good trajectories. The authors provide interesting real-world experiments that highlight the efficacy of their approach.
strengths_and_weaknesses: STRENGTHS

(S1) The authors study a compelling and difficult problem in robotics, and clearly showcase the benefit of using machine learning in order to approach it.

(S2) The proposed approach (the particular loss function and sampling-based approach to evaluating/deploying) seems novel and has the potential to be quite impactful.

(S3) The authors have performed impressive real-world experimentation.

WEAKNESSES

(W1) The experiment details and metrics used need to be re-hashed.

(a) The authors need to provide more details regarding how training data was collected. In the current submission, only the second paragraph of Section 7 seems to discuss this, and the details are sparse. How much training data must be collected? Are "random actions" really all that are required? It seems doubtful that such an approach would generate diverse enough coverage of the state space to be useful.

(b) "Final cost" is a reported metric, but its unclear as to what that really means for the task at hand. The authors should use a more easily-interpreted metric (time to destination?) in order to measure performance, as "cost" is really whatever the authors define it to be.

(W2) It's not clear from the submission as to what role partial observability plays in the proposed technique. For example, does the proposed approach require a full global map in order to be used? What if this information is not available?

MINOR COMMENTS

(MC1) It's unclear from Figure 3 as to what the difference between (c) and (d) is. This should be highlighted better in the caption.

POST-DISCUSSION COMMENTS

The authors have addressed several of my concerns, especially with respect to the experimental details section. The paper addresses an important topic, and the work seems strong experimentally.
summary_of_recommendation: I've recommended "weak accept" because I feel that the paper considers and important problem and appears to make a reasonable contribution towards solving it. I do think the authors have a few points to address, but overall the paper seems strong.
issues: (see "weaknesses" section above)
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper is investigating the challenge of using model-based reinforcement learning on rough terrain navigation problems with wheeled robots. The challenge here is that in rough terrain environments with wheels there's a number of different changes in contacts and other dynamics that make prediction difficult and the overall uncertainty of being able to predict into the future larger. The method in the paper then proposes a combination of methods that train over sequences of states into the future to help capture the compounding uncertainties better and preferences in the optimization procedure to avoid using predicted states that have high uncertainty. This combination of methods appears to result in more controled and accurate behavior driving a wheeled robot in a simulated environment and in the real world.
strengths_and_weaknesses: Pros
- The method proposed in the paper appears to be well-founded and the analysis in the paper also shows the positive effects of using each of the components in the overall method.
- The changes for the method are also clean and simple and easy for other people to be able to adopt making the reproduction of the work in this paper simple for others to use.

Cons
- However then part of the challenge here is the methods that are used in this paper have somewhat been explored in other prior work. Then the author in this case needs to better highlight the importance of the unique combination of these methods to be able to solve these particular tasks.
- This method is also primarily validated on the novel wheeled terrain navigation tasks that are shown in the paper. Therefore in order to compensate for some of the cons in the paper, it is important to run the proposed method across the environments that are explored in the PETs paper. This will show that the method also outperforms some other model-based methods and that it's not the case that the new method only performs well on the new custom environment that was created for this paper.
summary_of_recommendation: See the above comments.
issues: See the above comments.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a method to control wheeled robots to navigate across rough terrain. Uncertainty aware models are trained to model the dynamics and a tracking controller is used to improve prediction accuracy. An constrained optimization is the used to find trajectory with low divergence.
strengths_and_weaknesses: Pro:

1. A model learning algorithm that takes uncertainty into account.

2. A divergence constrained optimization algorithm to find suitable trajectory.

3. Demonstration on a real robot.

Con:

Nothing major.
summary_of_recommendation: This paper presents a method to control a wheeled robot to navigate across rough terrain. New algorithms are proposed to address issues common in model-based RL and the experiments are well executed, including comparison to alternatives and real robot tests.
issues: I think the paper is well written and the proposed method is well executed. No major issues need to be addressed.
review_title: null

Meta-review: The paper a model-based RL (MBRL) approach that takes into account the effect of uncertainty during the planning process. All reviewers agree on the importance of the problem and the good quality of the contribution. 

- The paper will benefit from clearly highlighting its key contributions. The paper proposes a combination of improvements in MBRL methods; however, some of these variants have been explored in the literature before. What unique advantages does the proposed combination have? Where are these gains coming from? Which part of the combination? More ablation studies to answer these questions might significantly strengthen the paper.

- A clear discussion on the scalability and applicability of the proposed approach, especially during the uncertainty propagation and the planning phases. 

- Demonstration of the proposed approach in simulation (and/or in experiments) on diverse systems and environments will help in understanding the advantages, the key limitations, and the applicability of the proposed approach compared to the state-of-the-art MBRL methods.

Besides addressing the major issues mentioned above, the authors should also revise the manuscript according to the other clarifications
requested and the suggestions provided. 

===== Post rebuttal =====

The authors have sufficiently addressed reviewers' concerns; specifically, the discussion on the related work was particularly helpful. I recommend an acceptance.

Decision: Accept (Poster)