Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents a Bayesian residual policy which improves a ensemble of expert policies by learning to reduce uncertainty. The algorithm is designed for reducing uncertainty due to the occluded objects and uncertainty about tasks. It is verified on two problems, cheese finding and door findiing, and compared with several different baselines. 

The idea of the paper is good and Algorithm 1 sets out to learn the exploration policy when the expert policies do not agree. The exposition and writing are clear. The experiments are details and convey that the proposed method outperforms the baselines.

That said, the formulation of the task is a bit unusual and too specific, making me wonder if the method works for other tasks. Some questions to clarify the task formulation:
1. Do agent start locations and cheese locations change during the training and evaluation? The figures suggest they remain the same, in which case the generality is limited.

2. When an agent senses for cheese, does it receive orientation or only the distance? If it receives the distances, will that not be a signal that matches the goals with some noise. In other words, why does the agent to sense several times at the beginning to associate which expert policy should be active, and then follow that policy.

3. Why and how was the reward for the cheese finding task determined? It seems very specific.

4. I would be helpful to provide some intuition about \psi

Overall an interesting paper, but not sure how well it would perform on a wider set of tasks.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors motivate and propose a learning algorithm, called Bayesian Residual Policy Optimization (BRPO), for Bayesian reinforcement learning problems. Experiment results are demonstrated in Section 5.

The paper is well written in general, and the proposed algorithm is also interesting. However, I think the paper suffers from the following limitations:

1) This paper does not have any theoretical analysis or justification. It would be much better if the authors can rigorously prove the advantages of BRPO under some simplifying assumptions.

2) It would be better if the authors can provide more experiment results, like experiment results in more games.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper considers Bayesian Reinforcement Learning problem over latent Markov Decision Processes (MDPs). The authors consider making decisions with experts, where each expert performs well under some latent MDPs. An ensemble of experts is constructed, and then a Bayesian residual policy is learned to balance exploration-exploitation tradeoff. Experiments on Maze and Door show the advantages of residual policy learning over some baselines.

1. The Bayesian Reinforcement Learning problem this work considered is important. However, using experts immediately make the problem much easier. The original Bayesian Reinforcement Learning problem is then reduced to making decision with experts. Under this setting, there are many existing work with respect to exploration-exploitation tradeoff (OFU, Thompson Sampling) with theoretical guarantees. I did not see why using this residual policy learning (although as mentioned residual/boosting is useful under other settings) is reasonable here. There is not theoretical support showing that residual learning enjoys guaranteed performance. The motivation of introducing this heuristic is not clear.

2. The comparisons with UPMLE and BPO seems not convincing. Both BPO and UPMLE do not use experts, and ensemble of experts outperforms them as shown in the experiments. And the ensemble baseline here is kind of weak (why sensing with probability 0.5 at each timestep?) Always 0.5 does not make sense (exploration should decrease as uncertainty reduced). Other exploration methods should be compared, to empirically show the advantages/necessities of residual policy learning.

Overall, I consider the proposed BRPO a simple extension of BPO, with a heuristic of learning ensemble policy to make decisions. BRPO is lack of theoretical support, and it is not clear why residual policy learning here is necessary and what exactly the advantage is over other exploration methods. Comparisons with simple baseline like exploration with constant probability is not enough to justify the proposed method.

=====Update=====
Thanks for the rebuttal. The comparison with PSRL improves the paper. However, I still think this paper needs more improvement as follows.
Theorem 1 looks hasty to me. Batch policy optimization Alg is going to solve n_{sample} MDPs, which are generated from P_0. But Eq. (6) or Theorem 1 does not contain information about P_0, implying that P_0 has no impact, which is questionable (an uniform P_0 that can generate different MDPs and a deterministic P_0 can only generate one MDP should be very different). I suggest the authors do more detailed analysis.
On the other hand, I expected whether this special "residual action" heuristic has any guarantees in RL? Can decomposing action into a_r + a_e provide us a better exploration method (than others like PSRL, OFU...)? Since this is the main idea of this paper as an extension of BPO, I think this point is important. The experiments shows that it can work in some cases, but I do not see an explanation (the "residual learning" paragraph is high level and I do not get an insight from that.).
review_title: Official Blind Review #1

Meta-review: This paper constitutes interesting progress on an important topic; the reviewers identify certain improvements and directions for future work (see in particular the updates from AnonReviewer1), and I urge the authors to continue to develop refinements and extensions.
Decision: Reject