Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors propose a GP-based goal-driven BRL problem formulation for goal-driven planning in partially unknown environments. They formulate goal-driven planning in unknown environments as model-based  BRL with clear expressibility and computational efficiency.
strengths_and_weaknesses: Strengths:
- Paper is well written and technically sound. Motivation of the paper is solid.
- The belief update in the theoretical analysis in section 4.4 is clearly explained but I have few questions. I like the figure 1.  

Weakness:
- Refer to my questions in Issues sections. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Technically sound paper with incremental research. 
issues: I have few questions on theoretical analysis from U-MDPs to GP-BAMDPs

- I couldn't find the reward function in the paper. Can you add them in the rebuttal? 
- Can you explain how the reward is back-propagated up the tree, and the branch’s visited counters are increased? 
- Is the transition function assumed to be deterministic here and independent from the objective function ?
- I was not clear on how the observation function O is sampled along trajectories? Can you elucidate it further?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The article proposed RL based planning algorithm for unknown environment with dynamics which could affect the cost of a planned trajectory. The proposed method was compared to two baseline approaches and demonstrated advantages through simulations.
strengths_and_weaknesses: Strengths:
The comparisons with two major approaches as baseline is great, which reveals the strength of the proposed approach. 


Weaknesses:
The experiments part seems a bit tricky, while the total number of runs in each setup is relatively small number (10), given this is in simulation. Also the baseline approach which seems not to benefit much when more time budget is allowed. Normally if the budget increase, the performance of each method should also improve to some extent.
The multi goal/target setup is interesting, however maybe also introduce a single goal/target version to allow the audience see the full picture.



robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I understand the proposed approach may outperform the baseline according to the experiments (in simulation), however I am not sure if the baseline approach were setup correctly so even the results looks good, I am not sure if we are seeing the full picture.
issues: Could you double check on the experiments and especially on the baselines. The performance should (at least slightly) increase when more budget/planning time allowed. 
Also consider adding a single target environment to enhance the comparisons.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies a robot motion planning in a partially known environment.  A motivating application is that of an underwater robot where the location of the robot is known but the ocean currents are not.  The unknown ocean currents affect the robot dynamics.  This problem is formalized as a Markov Decision Process with a partially unknown transition function.  The authors utilize Gaussian Processes to model the belief over the unknown state.  They then adapt the Bayes-Adaptive Monte Carlo planning (BAMCP) to their problem in which there are GP beliefs.  They utilize an existing technique called “root sampling”, which avoids GP updates during rollouts.  The authors extend the technical result in [19] to show that the distribution of rollouts is, in some sense, still correct.

The results are applied to two robotics problems, a radiation domain where a robot navigates between rooms with different radiation levels, and an underwater currents domain.

strengths_and_weaknesses: Strengths:
- The simulation environment and results are quite interesting.  I appreciated that two different domains were shown.  The results compared to the two baselines are impressive.

- The main contribution is the extension of root sampling to GPs.  While this follows the development from [19], but it is a nice result that the method is applicable in this setting.

Weaknesses:
- The abstract and introduction overstate the generality of what is studied in the paper.  I recommend the introduction clearly sets the problem of a discrete state space, uncertainty only in the transition function and not cost, and with full observability of the current state.  These assumptions should be clearly laid out early in the paper.

- The notation and use of acronyms is quite heavy in places, making the paper a hard read.  I do not have a concrete suggestion for improving this as I believe much of the notation follows from [19], but it was something I struggled with in reading the paper.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The main contribution is to apply the results in [19] to GP beliefs over a scalar unknown quantity.  On the technical side, the extension of the results from [19] is a good contribution but does appear to follow in a straightforward manner from Lemma 1 in [19].  The simulation results are a strong point, with two interesting scenarios for testing the proposed approach.  I have some concerns about the practical applicability of the proposed approach as it requires perfect localization and that the robot remains on the gridded set of states.  However, I would like to see the paper accepted as I think the paper does have significant strengths.
issues: - In the introduction, lines 27-30, I found this discussion of RL a bit misleading as I don’t think anyone would think of training an RL agent on every possible environment.  It feels like a bit of a straw man argument.

- Lines 41-42 it is stated that BRL is known to optimally trade off exploration and exploitation.  A reference for this should be included.

- Root sampling is stated as a contribution in lines 50-51, but the concept has not yet been introduced at this point..  

- In Section 4.2 the authors mention that the unknown state is considered scalar for notational simplicity.  It would be good to clarify that more than one unknown state feature is considered in the experiments.

- Line 98, the second half of the sentence appears to be repeated in the following sentence.

- Line 154, there is a discussion of implicitly making regularity and Lipschitz assumptions.  This requires some expansion as it is not clear what function must be Lipschitz continuous.

- Would it make sense to put equation (2) in a claim or lemma environment and then move the derivation to a proof? 

- Line 277, typo “stoasticity”

- In the limitations there is a discussion of sim-to-real transfer.  Is imperfect localization going to be the biggest hurdle here?  This is a strong assumption and I’m unsure how it would be handled in a real implementation (say for an underwater vehicle).
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper extended the existing Bayesian reinforcement learning approaches to the usage of Gaussian process belief by introducing GP model to the MDP with Unknown Feature Values. The proposed method GP-BAMCP was evaluated in two simulation tasks wiith unknown environmental features and demonstrated lower cost to goal compared with MCTS based approaches.
strengths_and_weaknesses: Strengths:
This paper addressed on an interesting topic and successfully attracted the reviewer from the attached video. The structure of the paper is overall good.

Weaknesses:
The reviewer believes the approach was not detailed enough. The current paper mainly focused on introducing the MDP with GP belief but did not explained how to solve it using RL. From Algorithm 1,  the reviewer see the planning of GP-BAMCP will minimize the Q function, but there is no information about this Q function. The reviewer wish to see how the approximatied Q function and the GP model affected each other during the learning process and how the related policy and value function were built.

The experimetal result is also limited in the current version. It is difficult to know the learning capability of the proposed method without learning curve. The comparsiosn to MCTS based approaches is also not enough since both two simulation tasks are not that challenging and should be solved by many baseline RL approach even without neither Bayesian view nor GP model.  The author should clearly explain why other methods were not compared.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: My recommendation is weak reject due to the weakness listed above.
issues: 1. the proposed method should be detailed in the form of an RL approach. The current paper mainly focused on MDP.
2. experiment should be exteneded with more analysis and additional comparsions.
review_title: null

Meta-review: metareview: The paper received mixed reviews. While reviewers appreciated the technical contribution as an extension of the results in [19], and the simulation results reported, they mentioned several issues that should be addressed during the rebuttal phase. In particular:

1. Clarity can be significantly improved in particular details for the Q function in Alg 1.
2. Better problem definition detailing the state space, transition function and state observability
3. Improve notation
4. Details on the baselines and adding a single target environment 

================================
Post rebuttal update

The authors have adequately addressed most of the concerns by the reviewers in the revised version.  
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)