Bayesian Residual Policy Optimization: Scalable Bayesian Reinforcement Learning with Clairvoyant Experts
Authors: Gilwoo Lee
Brian Hou
Sanjiban Choudhury
Siddhartha S. Srinivasa
Venue:  (2020)
Keywords: Bayesian Residual Reinforcement Learning
Residual Reinforcement Learning
Bayes Policy Optimization

Abstract: Informed and robust decision making in the face of uncertainty is critical for robots that perform physical tasks alongside people. We formulate this as a Bayesian Reinforcement Learning problem over latent Markov Decision Processes (MDPs). While Bayes-optimality is theoretically the gold standard, existing algorithms do not scale well to continuous state and action spaces. We propose a scalable solution that builds on the following insight: in the absence of uncertainty, each latent MDP is easier to solve. We split the challenge into two simpler components. First, we obtain an ensemble of clairvoyant experts and fuse their advice to compute a baseline policy. Second, we train a Bayesian residual policy to improve upon the ensemble's recommendation and learn to reduce uncertainty. Our algorithm, Bayesian Residual Policy Optimization (BRPO), imports the scalability of policy gradient methods as well as the initialization from prior models. BRPO significantly improves the ensemble of experts and drastically outperforms existing adaptive RL methods.

Under review as a conference paper at ICLR 2020
BAYESIAN RESIDUAL POLICY OPTIMIZATION: SCALABLE BAYESIAN REINFORCEMENT LEARNING WITH CLAIRVOYANT EXPERTS
Anonymous authors Paper under double-blind review
ABSTRACT
Informed and robust decision making in the face of uncertainty is critical for robots that perform physical tasks alongside people. We formulate this as a Bayesian Reinforcement Learning problem over latent Markov Decision Processes (MDPs). While Bayes-optimality is theoretically the gold standard, existing algorithms do not scale well to continuous state and action spaces. We propose scalable solution that builds on the following insight: in the absence of uncertainty, each latent MDP is easier to solve. We split the challenge into two simpler components. First, we ob- tain an ensemble clairvoyant experts and fuse their advice to compute a baseline policy. Second, we train Bayesian residual policy to improve upon the ensemble's s recommendation and learn to reduce uncertainty. Our algorithm. Bayesian Residual Policy Optimization (BRPO), imports the scalability of policy gradient methods as well as the initialization from prior models. BRPO significantly improves the ensemble of experts and drastically outperforms existing adaptive RL methods.
INTRODUCTION
Robots operating in the real world must resolve uncertainty daily basis. Often times, a robot is uncertain about how the world around it evolves. For example, a self-driving car must drive safely around unpredictable actors like pedestrians and bicyclists. A robot arm must reason about occluded objects when reaching into a cluttered shelf. On other occasions, a robot is uncertain about the task needs to perform. An assistive home robot must infer human's intended goal by interacting with them. Both examples of uncertainty require simultaneous inference and decision making, which can be framed as Bayesian reinforcement learning (RL) over latent Markov Decision Processes (MDPs). Agents do not know which latent MDP they are interacting with, preventing them from acting optimally with respect to that MDP. Instead, Bayes optimality only requires that agents be optimal with respect to their current uncertainty over latent MDPs.
The Bayesian RL problem can be viewed as solving a large continuous belief MDP, which is computationally infeasible to solve directly (Ghavamzadeh et al., 2015). We build upon simple yet recurring observation (Osband et al., 2013; Kahn al., 2017: Choudhury al., 2018): while solving the belief MDP may be hard, solving individual latent MDPs is much more tractable. Given exact predictions for all actors, the self-driving car can invoke motion planner to find a collision- free path. The robot arm can employ optimal controller to dexterously retrieve object given exact knowledge of all objects. Once the human's 's intended goal discovered, the robot can provide assistance. Hence, the overall challenge boils down to solving two (perhaps) simpler sub-challenges: solving the latent MDPs and combining these solutions to solve the belief MDP.
Let's assume can approximately solve the latent MDPs to create ensemble of policies as shown in Figure 1. We can think of these policies as clairvoyant experts, i.e., experts that think they know the latent MDP and offer advice accordingly. A reasonable strategy is to weigh these policy proposals by the belief and combine them into a single recommendation to the agent. While this recommendation is good for some regimes, agent to disregard the recommendation and explore the space effectively to collapse uncertainty. This leads to our key insight.
be misleading when uncertainty is high. The onus then is the
Learning Bayesian corrections on top of clairvoyant experts is a scalable strategy for solving complex reinforcement learning problems.

---

Under review as a conference paper at ICLR 2020
Figure I:
An overview of Bayesian Residual Policy Optimization. (a) Cheese location unknown and expert proposals disagree about which direction to move in, (b) The Bayesian residual policy learns to smell for cheese, reducing uncertainty. (c) The experts' recommendation guides the agent the cheese! BRPO balances exploration (Bayesian residual policy) with exploitation (expert recommendations).
While learning corrections echoes the philosophy of boosting (Freund & Schapire, 1999), agent goes one step beyond: learns to take uncertainty-reducing actions that highlight which expert to boost.
Our algorithm, Bayesian Residual Policy Optimization (BRPO), augments a belief-space batch policy optimization algorithm (Lee al., 2019) with clairvoyant experts (Figure 1). The agent observes the experts' recommendation, belief over the latent MDPs, and state. It returns a correction over the expert proposal, including ancertainty-reducinn sensing actions that experts never need take.
Our key contribution is the following:
We propose scalable Bayesian RL algorithm solve problems with complex latent rewards and dynamics.
We experimentally demonstrate that BRPO outperforms both the ensemble experts and existing adaptive RI. algorithms.
RELATED WORK
Bayesian reinforeement learning formalizes RL where one has a prior distribution over possible MDPs (Ghavamzadeh et al., 2015; Shani et al., 2013). However, the Bayes-optimal policy, which is the best approximation is necessary (Hsu et al., 2008). One way is to approximate the value function, as done in SARSOP (Kurniawati al., 2008) and PBVI (Pineau al., 2003); however, they cannot deal with continous state actions. Another strategy is to resort to sampling, such as BAMCP (Guez et al., 2012), POMCP (Silver Veness, 2010), POMCPOW (Sunberg & Kochenderfer, 2018). However, these approaches require significant amount of online computation.
Belief-Space RL Methods
do under uncertainty, is intractable solve for and
Online approaches forgo acting Bayesloptimally right from the onset, and instead aim to eventually act optimally. The question then becomes: how do we efficiently gain information about the test time MDP act optimally? BEB (Kolter & Ng, 2009) and POMDP-lite (Chen al., 2016) introduce an auxiliary reward term to encourage exploration and prove optimality. This has inspired work on more general, non -an-Bayesian curiosity based heuristics for reward gathering (Achiam & Sastry, 2017; Burda et al., 2018; Pathak al., 2017; Houthooft et al., 2016). Online exploration is also well studied in the bandit literature, and techniques such as posterior sampling (Osband al., 2019) bound the leamnrr' 's regret. UP-OSI (Yu et al., 2017) predicts the most likely MDP and maps that to an action. Gimelfarb et al. (2018) learns a gating over multiple expert value functions. However, online methods over-explore and drive the agent unsafe regimes.
SawaalyyApppoxiimaeeyy--CCooo -Corrett (PAC)
2

---

Under review as a conference paper at ICLR 2020
Another alternative is treat belief MDP problems large state space that must be compressed. Peng et al. (2018) use Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) to encode a history of observations to generate an action. Methods like BPO (Lee et al., 2019) explicitly utilize the belief distribution and compress it to learn a policy. The key difference between BRPO and BPO is that BRPO uses expert, enabling it to scale to handle complex latent tasks that may require multimodal policies.
Meta-reinforrement Learning Meta-reinforcement learning (MRL) approaches train sample- efficient learners by exploiting structure common to distribution of MDPs. For example, MAML. (Finn et al., 2017) trains gradient-based learners while RI.2 (Duan et al., 2016) trains memory-based learners. While meta-supervised learning has well established Bayesian roots (Bax- ter, 1998; 2000), it wasn't until recently that meta-reinforcement learning was strongly tied to Bayesian Reinforcement Learning (BRL) (Ortega et al., 2019; Rabinowitz, 2019). Nevertheless, even non- Son-Bayesian MRL approaches address problems pertinent to BRL. MAESN (Gupta et al., 2018) learns structured noise for exploration. E-MAML (Stadie et al., 2018) adds an explicit exploration bonus to the MAML objective. GMPS (Mendonca et al., 2019) exploit availability of MDP experts partially reduce BRL to II.. Our work is more closely related Bayesian MRL approaches. MAML-HB (Grant et al., 2018) casts MAML as hierarchical Bayes and improves posterior estimates. BMAML (Yoon et al., 2018) uses non-parametric variational inference improve posterior esti- mates. PLATIPUS (Finn et al., 2018) learns a parameter distribution instead of a fixed parameter. PEARL (Rakelly et al., 2019) learns a data-driven Bayes filter across tasks. In contrast to these approaches, use experts at test time, learning only to optimally correct them.
Residual Learning Residual learning has its foundations in boosting (Freund & Schapire, 1999) where combination of weak learners, each learning on the failures of previous, make strong learner. It also allows for injecting priors in RL, by boosting off of hand-designed policies or models. Prior work has leveraged known but approximate models by learning the residual between the approximate dynamics and the discovered dynamics (Ostafew et al., 2014; 2015; Berkenkamp & Schoellig, 2015). There has also been work on learning residual policies over hand-defined ones for solving long horizon (Silver et al., 2018) and complex control tasks (Johannink et al., 2019). Similarly, our approach starts with useful initialization (via experts) and learns to improve via Bayesian reinforcement learning.
3 PRELIMINARIES: BAYESIAN REINFORCEMENT LEARNING
In Bayesian reinforcement learning, the agent does not know the reward and transition functions but knows that they are determined by latent variable @ E +. Formally, the problem is defined by a tuple (S, 2, A, T, R. Po. 7). where S is the observable state space of the underlying MDPs, & is the latent space, and is the action space. T and R are the transition and reward functions parameterized by ø. The initial distribution over (s, o) is given by Po S x 4 R+ and is the discount.
Bayesian RL considers the long-term expected reward with respect to the uncertainty over & rather than the true (unknown) value of é. Uncertainty is represented as a belief distribution e B over latent variables Φ. The Bayes-optimal action value function is given by the Bellman equation:
Q(s,b.a') R(s, a') + Σ P(s'|s,,,,'' (')') s, a') max ((s(,'('''
(1)
The Bayesian reward function is the expected reward R(s,b,a') = Eseo b(0) o, a'). The Bayesian transition function is P(s'|s, b, a.) Eego ((o) 2, a'). The posterior update P(b')s, b, a') is computed recursively, starting from initial belief by.
1' ((0'),,ba
17 Σ (o))(s SE#
where 17 is the normalizing constant, and the transition function defined as T(s, o, P(s', o' [s. a') P(s'|s, ss, ó, a' P('P is, o, a', s'). At timestep the belief by ((() is the posterior distribution over & given the history of states and actions, (sp, dj $1, 82). When @ corresponds to physical parameters for an autonomous system, often assume that the latent states are fixed.
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Bayesian Residual Policy Optimization
Require: Bayes filter v. belief bo. prior Po. residual policy Te expert Tee horizon H, Vitry Nsampl
1: for 1,2,
7hr do
2: for n 1....... Resample do
3: Sample latent MDP M: (so, 20) Po
4: Tu < Simul wiimllaee(ne a Te. bo, th, M, H)
5: Update policy: 0, - watchoolicyopttimlaatiiio
fr,," Trampl
6:
return TThest
7:
procedure SIMULATEE(TO, Te: bo, 1,, M, H)
8: for
do Oc, ~ -
9:
Expert recommendation
10:
ar f ar, + ae.. ar (1( b de,) Residual action
Execute at on M, observe re, St
12:
bi b, s 11 St)
T (sp. bu, Or, 81,81. ,rru Cr 8H, 8H.bH) Only residuals are recorded
14:
return T
Our algorithm utilizes black-box Bayes filter to produce a posterior distribution over the latent states. However, Bayes filter can also be interpreted as function that compresses the history of states and actions. Recent work suggests that Long Short- Term Memory (LSTM) cells (Hochreiter Schmidhuber, 1997) can be meta meta-trained to compress history and predict subsequent states (Ortega et al., 2019). Such learned representations can be substituted for the belief distribution that we have chosen here.
3.1 BAYESIAN REINFORCEMENT LEARNING AND POSTERIOR SAMPLING
Posterior Sampling Reinforcement Learning (PSRL) (Osband et al., 2013) is an online RL algorithm that maintains a posterior over latent MDP parameters d. However, the problem setting considers and how it uses this posterior are quite different than what we consider in this paper.
In this work, MDP for a single episode; latent parameters are resampled for each episode. The PSRL regret analysis assumes MDPs with finite horizons and repeated episodes with the same test MDP, i.c. the latent parameters are fixed for all episodes.
focused zero-shot scenarios where the agent can only interact with the test
Before each episode, PSRL samples an MDP from its posterior over MDPs, computes the optimal policy for the sampled MDP, and executes it on the fixed test MDP. Its posterior is updated after each episode, concentrating the distribution around the true latent parameters. During this exploration period, it can perform arbitrarily poorly (see Section 6. of the appendix for concrete example). Furthermore, sampling latent MDP from the posterior determinizes the parameters; as a result, there is no uncertainty in the sampled MDP, and the resulting optimal policies that are executed will never take sensing actions. In this work, we have focused on Bayesian RL problems where sensing is critical to performance. BRPO, like other Bayesian RL algorithms, focuses on learning the Bayes-optimal policy during training, which can used at test time to immediately explore and exploit in new environment.
4 BAYESIAN RESIDUAL POLICY OPTIMIZATION (BRPO)
In Bayesian Residual Policy Optimization, we first construct ensemble of clairvoyant experts where each approximately solves latent MDP. Expert proposals are gated with the belief over MDPs to compute a recommendation. We then train Bayesian residual policy to correct the recommendation, resulting in an elegant exploration-exploitttior tradeoff. The agent learns to produce smaller corrections when the recommendation is effective, i.c. when uncertainty is small or when all clairvoyant experts agree with a good recommendation. Otherwise, the agent overrides the recommendation and learns to explore effectively.
4

---

Under review as a conference paper at ICLR 2020
Expert 1
Belief
Residual Policy
State
Correction
Action
Expert k
Recommendation
Figure 2: Bayesian residual policy network architecture.
4.1 ENSEMBLE OF CLAIRVOYANT EXPERTS
For simplicity of exposition, assume the Bayesian RL problem consists of k underlying latent MDPs, $1...., Øk- Clairvoyant experts т; can be computed for each Φ; via single-MDD RL methods (or optimal control, if transition and reward functions are known). If b(te) is the posterior belief each MDP 6i, we want to combine experts to construct belief-aware recommendation that maps the state and belief to a distribution over actions ne S
P(A).
One choice for Te is to select the maximum a posteriori (MAP) action.
GMAP arg max (o))m((as))
However, computing the MAP estimate may require optimizing a non-convex function, e.g., when the distribution multimodal. We can instead maximize the lower bound using Jensen's inequality.
log Σ ((bs); N
Σ b( (als)
(4)
This is much easier to solve, especially log me (als) convex. each (a(s)) a Gaussian with mean μr and covariance E, c.g. from TRPO (Schulman et al., 2015), the resultant action is
a" arg max Σ log (als))
Σ (O
Σ
(o: pi
(5)
When the belief has collapsed to one di the resulting ensemble recommendation follows the corre- sponding n; exactly. Thus, as entropy reduces, the ensemble is more reliable.
There are other alternatives to consider. One choice for тe to directly use the mixture model Σ ((ls) This would be equivalent to posterior sampling (Osband et al., 2013).
While this belief-aware ensemble easy to attain, is not Bayes-optimal. In particular, since the clairvoyant experts do not take explicit uncertainty-reducinn actions, the ensemble will not recommend do so. Consider the Maze4 example: each clairvoyant expert knows its corresponding latent MDP's hidden goal position, and thus navigates optimally without sensing. A Bayes-optimal agent, on the other hand, would take sensing actions to identify latent goal.
4.2 RESIDUAL POLICY LEARNING
In each training iteration, BRPO collects trajectories by simulating the current policy several MDPs sampled from the prior distribution (Algorithm 1). every timestep of the simulation, the ensemble is queried for action recommendation, which is summed with the correction from the residual policy network (Figure 2) and executed. The Bayes filter updates the posterior after observing the resulting state. The collected trajectories (with only residual actions) are the input to policy optimization algorithm (Schulman et al., 2015; 2017) which updates the residual policy network.
The residual policy does not solve the original belief MDP. Since its corrective actions summed with the ensemble's recommendations, it in fact operates in residual belief MDP (with respect to the recommendations). Actions simply shifted by the recommendations. That is, for every residual
5

---

Under review as a conference paper at ICLR 2020
action a, and expert recommendation de' ~ b)), define new transition dynamics T from the original T":
P,(s, (,s,,,, JNJEN Pr(s' b b', an + ae a (Te((e))s)))
(6)
This new transition function T defines the residual belief MDP. Since the experts are fixed, the residual belief-MDP is also fixed during training and testing time. Thus, this strategy inherits all mathematical guarantees from the underlying policy optimization algorithm, such as monotonic improvement from the ensemble's baseline policy. This leads to the following theorem.
Theorem 1. BRPO inherits the mathematical properties of the underlying batch policy optimization algorithm.
Proof.
The proof directly follows from Equation 6.
EXPERIMENTAL RESULTS
We choose problems that highlight common challenges in robotics:
Explicit implicit sensing actions required to infer the latent MDP.
Sensing is costly, and different sensing actions may have different costs.
Solutions for each latent MDP are significantly different.
In all domains that we consider, BRPO improves on the ensemble's s recommendation, learning to sense in cost-effective manner. As seen in Door 4, BRPO also can develop more effecttive strategies for both sensing and control.
Latent Goal Mazes In the Maze4 and Mazel0 environments, the agent must identify which latent goal is active. This is an example of where the dynamics are the same across all latent MDPs, but the task must be inferred.
At the beginning of each episode, the latent goal is set to one of four goals (Maze4) or ten goals (Maze10). This problem requires explicit sensing distinguish the goal. Sensing can happen simultaneously as the agent moves, but costs -1; the agent receives noisy measurement of the distance to the goal, with noise proportional to the true distance. This motivates the agent minimize sensing and sense when closer to goals to obtain better measurements.
After each action, the agent observes its current position, velocity, and distance to all latent goals. sensing is invoked, it also observes the noisy distance to the goal. In addition, the agent observes the categorical belief distribution over the latent goals and the ensemble's recommendation. Each expert proposes action (computed via motion planning) that navigates to the corresponding goal. The experts are unaware of the penalty for reaching incorrect goals, which will demonstrate that BRPO can improve such suboptimal experts.
In Maze4, reaching the active goal provides a terminal reward of 500, while reaching an incorrect goal gives penalty of 5500 The task ends when the agent receives either the terminal reward or penalty, or after 500 timesteps. In Ma Mazel 0, the agent receives a penalty of -5 and continues to explore after reaching an incorrect goal.
Figure 3a demonstrates rollouts by the trained BRPO agents on Maze4 and Mazel0. In Maze 1 O, goals that are near each other have drastically different paths them, making task inference more important. For both Maze4 and Mazel0, O, we see that the agent reroutes itself (multiple times in Mazel0) 0) while it invokes sensing to get better belief.
In this more classical POMDP problem, there are 4 possible doors to the goal in the next room. At the beginning each episode, each door is opened or closed with 0.5 probability. To check the doors, the agent either sense (-1) 1) crash into them (-10), As with the mazes. the sense action can be taken simultaneously as the agent moves. Sensing returns a noisy binary vector for all four doors, with exponentially decreasing accuracy proportional to the distance each door.
Doors

---

Under review as a conference paper at ICLR 2020
(a) Latent goal mazes with four (Maze4) and (Maze10) possible goals. The agent navigates. changing its direction as goals are deemed less likely (more transparent). We have marked the true goal with red the last frame for clarity.
SENSE
(b) Door4. The agent senses only when it is near the wall with doors, where sensing is most accurate. The transparency of the red bars indicates the posterior probability that the door blocked. With sensing, the third door becomes more likely open while the others become more likely to closed.
Figure 3: BRPO policy keyframes. Best viewed in color.
Crashing returns an accurate indicator of the door crashed into. At every step, the agent observes its position, velocity, distance to goal, and whether it crashed passed through door. In addition. the agent observes the categorical distribution over the 2 = 16 possible door configurations (from the Bayes filter) and the ensemble's recommendation. The agent receives a terminal reward of 100 it reaches the goal within 300 timesteps.
We observe that BRPO's learned policy is quite different from any of the experts. Each expert navigates directly through the closest open door. BRPO gets very close to the wall (to minimize sensor noise) and senses while sliding along the wall, before identifying an open door and navigating through it.
5.1 BRPO OUTPERFORMS ADAPTIVE RL METHODS
We compare BRPO to adaptive RI. algorithms that consider the helief over latent states: BPO (Lee al., 2019) and UP-MLE, a modification Yu al. (2017) that uses the most likely estimate from the Bayes filter'. We also compare with the ensemble of experts baseline, with one key difference. The ensemble will not take any sensing actions (as discussed Section 4), so we strengthen by sensing with probability 0.5 at each timestep. This is equivalent to the initial BRPO policy, which adds random noise the ensemble recommendation. More sophisticated sampling strategies can be considered but require more task-specific knowledge to design; see the appendix for more discussion (Section 6.2).
Figure 4a compares the training performance of all algorithms across the three environments. (Note that OPTIMAL is unachievable, since it requires full knowledge of the latent MDP.) In RANDOM, the agent randomly chooses one of the clairvoyant experts to follow for the entire episode.
This originally introduced in Lee et al. (2019).
7

---

Under review as a conference paper at ICLR 2020
BRPO
UTMLE
BPO
EXSEMBLE
RLANOM
OPTIMAL
398
18
(a) Training curves. BRPO dramatically outperforms agents that not leverage expert knowledge, and siguificantly improves the ensemble of experts.
(b) Sensing locations. In Maze4 and Mazel0, sensing is dense around the starting regions (the bottom row in Maze4 and center in Mazel0) and in areas where multiple latent goals are nearby. The agent sometimes reroutes before reaching incorrect goal. In Door4, BRPO only senses when close the doors, where the sensor most accurate.
Figure 4: BRPO performance on Maze4, Mazel 0, and Door4 (left to right).
BRPO agents dramatically outperforms BPO and UP-MLE agents. fact, we have trained BPO and UP-MLE with an additional boost to encourage information information--athering (Section 5.3); without such bonuses, they did not learn to take any meaningful behavior. Even with the bonus, these agents learns to solve the task only partially. In Maze4 and Mazel0 0, they only reach some of the goals. In Door4, 4, they only learn to navigate through one of the first two doors and will occasionally crash.
Examining where sensing has happened (Figure 4b). when goals must be distinguished, and uses the belief to reroute itself Maze4 and Mazel0 Qualitatively, find that UP-MLE relies exclusively on crashing into doors to reduce uncertainty, which is extremely costly. The BRPO agent avoids crashing in almost all scenarios.
that the BRPO agent learns to sense
5.2 RESIDUAL POLICY INPUTS
The BRPO policy takes the belief distribution, state, and ensemble recommendation as inputs (Figure 2). However, since the ensemble recommendation implicitly includes the belief, the belief may not be necessary input to the policy the recommendation is already provided.
The results show that providing both belief and recommendation as inputs to the policy are important (Figure 5a). Although BRPO with only the recommendation performs comparably BRPO with both inputs on Ma z and Maze10 0, the one with both inputs produce faster learning on Door4.
5.3 INFORMATION-ATTIERR
REWARD BONUSES
Because BRPO maximizes the Bayesian Bellman equation (Equation 1), exploration incorporated into its long- tong-term objective. As result, auxiliary rewards encourage exploration unneeded. However, existing work that does not explicitly consider the belief has suggested various auxiliary reward terms that encourage exploration, such intrinsic rewards (Pathak al., 2017) surprisal
8

---

Under review as a conference paper at ICLR 2020
Belief +
Recommendation
Revonmenddation only
ENSEMBLE
OPTIAAL
398
Taloo
(a) Including both belief recommendation as policy imputs results faster learning in Door4.
ENSEMULL
OPTIMAL
500
5000
100
(b) information-gatheiie reward bonuses (Equation 7) unnecessary with BRPO. Large bonuses the agent to ignore penalties from the environment, resulting suboptimal performance (Door
Figure 5: Residual policy input and reward bonus experiments Maze Mazel0, 0, and Door (left to right).
rewards (Achiam & Sastry, 2017). To investigate whether such rewards benefit the BRPO agent, we augment the reward function with the following auxiliary bonus:
F(st, be, az) rst,, be, ar) + Es,. [l|b, b [l]]
where ||b, be+1llll 111 = Σ (br(0)) b.41 (d.)) rewards change in belief.?
Figure 5b summarizes the performance of BRPO when training with 0, 10, 100. Too much emphasis on information-gatheriin causes the agent over-explore and therefore underperform. In Door4 with e = 100, we qualitatively observe that the agent crashes into the doors more often. This because crashing significantly changes the belief for that door; the huge reward bonus outweighs the penalty of crashing from the environment.
We find that BPO and UP-MLE unable learn without an exploration bonus. We used for Maze4 and 4 and 100 for Mazel0, O. With the bonus, both BPO and UP-MLE learn to sense initially but struggle solve the challenging latent MDPs.
DISCUSSION AND FUTURE WORK
In the real world, robots must deal with uncertainty, either due to complex latent dynamics or task specifics. Because uncertainty is an inherent part of these tasks, we can at best aim for optimality under uncertainty, i.e., Bayes optimality. Existing BRL algorithms POMDP solvers do not scale well to problems with complex latent MDPs or a large (continuous) set of MDPs.
We decompose BRL problems into two parts: solving each latent MDP and being Bayesian the solutions. Our algorithm, Bayesian Residual Policy Optimization, operates on the residual belief-MDP space given an ensemble of experts. BRPO focuses learning explore, relying on the experts for exploitation. BRPO is capable of solving complex problems, outperforming existing BRL algorithms and improving on the original ensemble of experts.
An analogous term has heen introduced Chen al. (2016).

---

Under review as a conference paper at ICLR 2020
Although out of scope for this work, few key challenges remain. First is efficient construction of an ensemble of experts, which becomes particularly important for continuous latent spaces with infinitely many MDPs. Infinitely many MDPs do not necessarily require infinite experts, as many may converge to similar policies. An important future direction is subdividing the latent space and computing qualitatively diverse set of policies (Liu al., 2016). Another challenge is developing an efficient Bayes filter, which is an active research area. In certain occasions, the dynamics of the latent MDPs may not be accessible, which would require learned Bayes filter. Combined with tractable, efficient Bayes filter and an efficiently computed set of experts, we believe that BRPO will provide an even more scalable solution for BRL problems.
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprini arXiv: (xXiv:1703.01322 2017.
Jonathan Baxter. Theoretical models of learning learn. In Learning to learn, pp- 71-94. Springer, 1998.
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Felix Berkenkamp and Angela P Schoellig. Safe and robust learning control with gaussian processes. In 2015 European Control Conference (ECC), pp. 2496-2500 IEEE, 2015.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: / 12894, 2018.
Min Chen, Emilio Frazzoli, David Hsu, and Wee Sun Lee. POMDP-lite for robust robot planning under uncertainty. In IEEE International Conference on Robotics and Automation, 2016.
Sanjiban Choudhury, Mohak Bhardwaj. Sankalp Arora, Ashish Kapoor, Gireeja Ranade, Sebastian Scherer, and Debadeepta Dey. Data-driven planning via imitation learning. The International Journal of Robotics Research, 37(13-14):1632-1622 2018.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint 1XXi::661..027777 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- -oolue 70, pp. 1 1112--1132 135. JMLR. org, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. arXiv preprint 18X.021806.02817, 2018.
Yoav Freund and Robert Schapire. short introduction boosting. Journal-lapanese Society For Artificial Intelligence, 141771-780):611,, 1999.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al. Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8(5-6):359444 483, 2015.
Michael Gimelfarb, Scott Sanner, and Chi-Guhn Lee. Reinforcement learning with multiple experts: A bayesian model combination approach. In Advances in Neural Information Processing Systems, pp. 9028-9528333 9538, 2018
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hicrarchical bayes. arXiv preprint 180iv:1801 08930,0 2018.
Arthur Guez, David Silver, and Peter Dayan. Efficient adyes-adaptive reinforcement learning using sample-based search. In Advances in Neural Information Processing Systems, 2012.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pp. 5302-5311, 2018.
10

---

Under review as a conference paper at ICLR 2020
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1 100--1117, 2016.
David Hsu, Wee S Lee, and Nan Rong. What makes some pomdp problems easy to approximate? In Advances in neural information processing systems, pp. 689-696, 2008.
Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control. In 2019 International Conference on Robotics and Automation (ICRA), pp. 6023-6029. IEEE, 2019.
Gregory Kahn, Tianhao Zhang, Sergey Levine, and Pieter Abbeel. Plato: adaptive trajectory optimization. In IEEE International Conference Robotics and Automation, pp. 3342-3349, IEEE, 2017.
Policy learning using
Zico Kolter and Andrew Ng. Near-Bayer Baar-Bayesian exploration in polynomial time. In International Conference on Machine Learning. 2009.
Hanna Kurniawati, David Hsu, and Wee Sun Lee. SARSOP: by approximating optimally reachable belief spaces. In Robotics: Science and Systems, 2008.
Efficient point-based POMDP planning
Gilwoo Lee, Brian Hou, Aditya Mandalika, Jeongseok Lee, Sanjiban Choudhury, and Siddhartha Srinivasa. Bayesian policy optimization for model uncertainty. In International Conference on Learning Representations, 2019.
Yao Liu, Zhaohan Guo, and Emma Brunskill. Pac continuous state online multitask reinforcement learning with identification. In Proceedings of the 2016 International Conference Autonomous Agents & Multiagent Systems, pp. 438-444-44 446. International Foundation for Autonomous Agents and Multiagent Systems, 2016.
Russell Mendonea, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. arXiv preprint arXiv: (w1.1994440999 2019.
Pedro A. Ortega, Jane X. Wang. Mark Rowland, Tim Genewein, Zeb Kurth-Nelson Razvan Pascanu, Nicolas Heess, Joel Veness, Alexander Pritzel, Pablo Sprechmann, Siddhant M. Jayakumar, Tom McGrath, Kevin Miller, Mohammad Gheshlaghi Azar, Ian Oshand, Neil C. Rabinowitz, András György, Silvia Chiappa, Simon Osindero, Yee Whye Teh, Hado van Hasselt, Nando de Freitas, Matthew Botvinick, and Shane Legg. Meta-learning of sequential strategies. arXiv preprint arXiv:1905.03030, 2019.
Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. In Advances Neural Information Processing Systems, 2013.
Ian Osband, Benjamin Van Roy, Daniel J. Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20(124):1-62, 2019. URL http:
//jmlr. http//ppprrs//000 18800/18--0 339. .hml.
Chris J Ostafew, Angela P Schoellig, and Timothy D Barfoot. Learning-based nonlinear model predictive control to improve vision-based mobile robot path-tracking in challenging outdoor environments. In IEEE International Conference Robotics and Automation, pp. 4029-4036. IEEE, 2014.
Chris J Ostafew, Angela P Schoellig, and Timothy D Barfoot. Conservative to confident: treating uncertainty robustly within learning-based control. IEEE International Conference on Robotics and Automation, pp. 421-427. IEEF, 2015.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-drivan exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.

---

Under review as a conference paper at ICLR 2020
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In IEEE International Conference on Robotics and Automation, 2018.
Joelle Pineau, Geoff Gordon, Sebastian Thrun, al. Point-based value iteration: An anytime algorithm for POMDPs. In International Joint Conference on Artificial Intelligence, 2003.
Neil C. Rabinowitz. Meta-learners' learning dynamics are unlike learners'. arXiv preprint arXiv:1905.01320. 2019.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine Efficient off poolio meta-reinforreement learning via probabilistic context variables. arXiv preprint arXiv: 1903.0824, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Guy Shani, Joelle Pineau, and Robert Kaplow. A survey of point-based POMDP solvers. Journal on Autonomous Agents and Multiagent Systems, 27(1)11-,, 2013.
David Silver and Joel Veness. Monte-carlo planning in large POMDPs. In Advances in Neural Information Processing Systems, 2010.
Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual policy learning. arXiv preprint arXiv:18/2.006298, 2018
Bradly C. Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. arXiv preprint /803. 0/ / 18, 2018
Zachary Sunberg and Mykel Kochenderfer: Online algorithms for POMDPs with continuous state, ac- tion, and observation spaces. In International Conference on Automated Planning and Scheduling. 2018.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Informution Processing Systems, pp. 7332-7342, 2018
Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. In Robotics: Science and Systems, 2017.
12

---

Under review as a conference paper at ICLR 2020
APPENDIX
6.1 THE GAP BETWEEN BAYES OPTIMALITY AND POSTERIOR SAMPLING
$00
R
Depth d
Figure 6: A tree-like MDP that highlights the distinction between BRL and PSRL.
As discussed in Section 3.1, Bayesian reinforcement learning and posterior sampling address quite different problems. We present a toy problem to highlight the distinction between them.
Consider a deterministic tree-like MDP (Figure 6). Reward is received only at the terminal leaf states: one leaf contains pot of gold (Rt 100) and all others contain dangerous tiger (/t 10). All non-leaf states have two actions, go left (L) and go right (R). The start state additionally has sense action (S), which is costly (R = -0.1) but reveals the exact location of the pot of gold. Both algorithms are initialized with a uniform prior over the N = possible MDPs (one for each possible location of the pot of gold).
To contrast the performance of the Bayes-optimal policy and posterior sampling, we consider the multi-episode setting where the agent repeatedly interacts with the same MDP. The MDP is sampled once from the uniform prior, and agents interact with it for T episodes. This is the setting typically considered by posterior sampling (PSRL) (Osband al., 2013).
Before each episode, PSRL. samples an MDP from its posterior over MDPs, computes the optimal policy, and executes it. After each episode, it updates the posterior and repeats. Sampling from the posterior determinizes the underlying latent parameter. As a result, PSRL will never produce sensing actions to reduce uncertainty about that parameter because the sampled MDP has no uncertainty. More concretely, the optimal policy for each tree MDP is to navigate directly to the gold without sensing; PSRL will never take the sense action. Thus, PSRL makes an average of A-1 mistakes before sampling the correct pot of gold location and the cumulative reward over T episodes is
2
10
N
100
TT
N
mistakes
pot of gold
In the first episode, the Bayes-optimal first action is to sense. All subsequent actions in this first episode navigate toward the pot of gold, for an episode reward of -0.1 + 100. the subsequent T T-1 1 episodes, the Bayes-optimal policy navigates directly toward the goal without needing to sense, for a cumulative reward of 100T 0.1. The performance gap between the Bayes-optimal policy and posterior sampling grows exponentially with depth of the tree d.
Practically, naïve policy gradient algorithm (like BPO) would struggle learn the Bayes-optimal policy: it would need to learn to both sense and navigate the tree to the sensed goal. BRPO can take advantage of the set of experts, which each navigate to their designated leaf. During training, the BRPO agent only needs to learn balance sensing with navigation.
As mentioned in Section 3.1. PSRL an online learning algorithm and is designed to address domains where the posterior naturally updates result of multiple episodes of interactions with the latent MDP. PSRL is more focused on improving the performance over episodes, which is different from the average performance zero-shot performance that consider in this work.
13

---

Under review as a conference paper at ICLR 2020
6.2 EXPERIMENTS: BETTER SENSING ENSEMBLE
The ensemble we considered in Section 5 randomly senses with probability 0.5. A more effective sensing ensemble baseline policy could be designed manually, and used the initial policy for the BRPO agent to improve on. Designing such a policy can be challenging: it requires either task- specific knowledge, or solving an approximate Bayesian RL problem. We bypass these requirements by using BRPO.
On the Mazel0 environment, we have found via offline tuning that a more effective ensemble baseline agent senses only for the first 150 of 750 timesteps. The average return is 416.3 = 9.4, which outperforms the original ensemble baseline average return of 409.5 = 10.8. However, this still lower than the BRPO agent that starts with that original ensemble, which accumulated an average return of 465.7 4.7. This trained BRPO agent also achieves task completion rate of 100%, which better than the 96.3% completed by the improved ensemble baseline. The performance gap comes from the suboptimality of the ensemble recommendation, as experts are unaware of the penalty for reaching incorrect goals.
6.3 EXPERIMENTS: POSTERIOR SAMPLING
We have also evaluated the performance of PSRL on the Mazel0 10 environment. This has required slight modification PSRL to handle the zero-shot scenario: it now samples from the posterior at each timestep, and executes the corresponding optimal expert (which is aware of the penalties from reaching the wrong goal). However, PSRL never senses. As a result, this vanilla agent frequently incurs the penalty for reaching incorrect goals, achieving an average return of 1224.4 + 11.3. Augmenting the PSRL agent by sensing with probability 0.5 (as with ensemble method) results in average return of 464.1 + 5.5, and task completion rate of 94%. Failures occur when the posterior does not collapse to a single target for the cesterior-sampler experts to navigate toward.
14


Reviews:
[{"review_id": "HJxPLs_qcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a Bayesian residual policy which improves a ensemble of expert policies by learning to reduce uncertainty. The algorithm is designed for reducing uncertainty due to the occluded objects and uncertainty about tasks. It is verified on two problems, cheese finding and door findiing, and compared with several different baselines. \n\nThe idea of the paper is good and Algorithm 1 sets out to learn the exploration policy when the expert policies do not agree. The exposition and writing are clear. The experiments are details and convey that the proposed method outperforms the baselines.\n\nThat said, the formulation of the task is a bit unusual and too specific, making me wonder if the method works for other tasks. Some questions to clarify the task formulation:\n1. Do agent start locations and cheese locations change during the training and evaluation? The figures suggest they remain the same, in which case the generality is limited.\n\n2. When an agent senses for cheese, does it receive orientation or only the distance? If it receives the distances, will that not be a signal that matches the goals with some noise. In other words, why does the agent to sense several times at the beginning to associate which expert policy should be active, and then follow that policy.\n\n3. Why and how was the reward for the cheese finding task determined? It seems very specific.\n\n4. I would be helpful to provide some intuition about \\psi\n\nOverall an interesting paper, but not sure how well it would perform on a wider set of tasks.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BylbDBJaYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers Bayesian Reinforcement Learning problem over latent Markov Decision Processes (MDPs). The authors consider making decisions with experts, where each expert performs well under some latent MDPs. An ensemble of experts is constructed, and then a Bayesian residual policy is learned to balance exploration-exploitation tradeoff. Experiments on Maze and Door show the advantages of residual policy learning over some baselines.\n\n1. The Bayesian Reinforcement Learning problem this work considered is important. However, using experts immediately make the problem much easier. The original Bayesian Reinforcement Learning problem is then reduced to making decision with experts. Under this setting, there are many existing work with respect to exploration-exploitation tradeoff (OFU, Thompson Sampling) with theoretical guarantees. I did not see why using this residual policy learning (although as mentioned residual/boosting is useful under other settings) is reasonable here. There is not theoretical support showing that residual learning enjoys guaranteed performance. The motivation of introducing this heuristic is not clear.\n\n2. The comparisons with UPMLE and BPO seems not convincing. Both BPO and UPMLE do not use experts, and ensemble of experts outperforms them as shown in the experiments. And the ensemble baseline here is kind of weak (why sensing with probability 0.5 at each timestep?) Always 0.5 does not make sense (exploration should decrease as uncertainty reduced). Other exploration methods should be compared, to empirically show the advantages/necessities of residual policy learning.\n\nOverall, I consider the proposed BRPO a simple extension of BPO, with a heuristic of learning ensemble policy to make decisions. BRPO is lack of theoretical support, and it is not clear why residual policy learning here is necessary and what exactly the advantage is over other exploration methods. Comparisons with simple baseline like exploration with constant probability is not enough to justify the proposed method.\n\n=====Update=====\nThanks for the rebuttal. The comparison with PSRL improves the paper. However, I still think this paper needs more improvement as follows.\nTheorem 1 looks hasty to me. Batch policy optimization Alg is going to solve n_{sample} MDPs, which are generated from P_0. But Eq. (6) or Theorem 1 does not contain information about P_0, implying that P_0 has no impact, which is questionable (an uniform P_0 that can generate different MDPs and a deterministic P_0 can only generate one MDP should be very different). I suggest the authors do more detailed analysis.\nOn the other hand, I expected whether this special \"residual action\" heuristic has any guarantees in RL? Can decomposing action into a_r + a_e provide us a better exploration method (than others like PSRL, OFU...)? Since this is the main idea of this paper as an extension of BPO, I think this point is important. The experiments shows that it can work in some cases, but I do not see an explanation (the \"residual learning\" paragraph is high level and I do not get an insight from that.).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BygdF6XLqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors motivate and propose a learning algorithm, called Bayesian Residual Policy Optimization (BRPO), for Bayesian reinforcement learning problems. Experiment results are demonstrated in Section 5.\n\nThe paper is well written in general, and the proposed algorithm is also interesting. However, I think the paper suffers from the following limitations:\n\n1) This paper does not have any theoretical analysis or justification. It would be much better if the authors can rigorously prove the advantages of BRPO under some simplifying assumptions.\n\n2) It would be better if the authors can provide more experiment results, like experiment results in more games.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper constitutes interesting progress on an important topic; the reviewers identify certain improvements and directions for future work (see in particular the updates from AnonReviewer1), and I urge the authors to continue to develop refinements and extensions.
Decision: Reject