Bayesian Residual Policy Optimization: Scalable Bayesian Reinforcement Learning with Clairvoyant Experts

1 INTRODUCTION . Robots operating in the real world must resolve uncertainty on a daily basis . Often times , a robot is uncertain about how the world around it evolves . For example , a self-driving car must drive safely around unpredictable actors like pedestrians and bicyclists . A robot arm must reason about occluded objects when reaching into a cluttered shelf . On other occasions , a robot is uncertain about the task it needs to perform . An assistive home robot must infer a human ’ s intended goal by interacting with them . Both examples of uncertainty require simultaneous inference and decision making , which can be framed as Bayesian reinforcement learning ( RL ) over latent Markov Decision Processes ( MDPs ) . Agents do not know which latent MDP they are interacting with , preventing them from acting optimally with respect to that MDP . Instead , Bayes optimality only requires that agents be optimal with respect to their current uncertainty over latent MDPs . The Bayesian RL problem can be viewed as solving a large continuous belief MDP , which is computationally infeasible to solve directly ( Ghavamzadeh et al. , 2015 ) . We build upon a simple yet recurring observation ( Osband et al. , 2013 ; Kahn et al. , 2017 ; Choudhury et al. , 2018 ) : while solving the belief MDP may be hard , solving individual latent MDPs is much more tractable . Given exact predictions for all actors , the self-driving car can invoke a motion planner to find a collision-free path . The robot arm can employ an optimal controller to dexterously retrieve an object given exact knowledge of all objects . Once the human ’ s intended goal is discovered , the robot can provide assistance . Hence , the overall challenge boils down to solving two ( perhaps ) simpler sub-challenges : solving the latent MDPs and combining these solutions to solve the belief MDP . Let ’ s assume we can approximately solve the latent MDPs to create an ensemble of policies as shown in Figure 1 . We can think of these policies as clairvoyant experts , i.e. , experts that think they know the latent MDP and offer advice accordingly . A reasonable strategy is to weigh these policy proposals by the belief and combine them into a single recommendation to the agent . While this recommendation is good for some regimes , it can be misleading when uncertainty is high . The onus then is on the agent to disregard the recommendation and explore the space effectively to collapse uncertainty . This leads to our key insight . Learning Bayesian corrections on top of clairvoyant experts is a scalable strategy for solving complex reinforcement learning problems . Cheese location unknown Experts not in consensus on which direction to move Agent learns to move around and smell Uncertainty collapses ; expert recommendation converges Agent finds cheese ! Balanced exploration ( BRPO ) and exploitation ( experts ) While learning corrections echoes the philosophy of boosting ( Freund & Schapire , 1999 ) , our agent goes one step beyond : it learns to take uncertainty-reducing actions that highlight which expert to boost . Our algorithm , Bayesian Residual Policy Optimization ( BRPO ) , augments a belief-space batch policy optimization algorithm ( Lee et al. , 2019 ) with clairvoyant experts ( Figure 1 ) . The agent observes the experts ’ recommendation , belief over the latent MDPs , and state . It returns a correction over the expert proposal , including uncertainty-reducing sensing actions that experts never need to take . Our key contribution is the following : • We propose a scalable Bayesian RL algorithm to solve problems with complex latent rewards and dynamics . • We experimentally demonstrate that BRPO outperforms both the ensemble of experts and existing adaptive RL algorithms . 2 RELATED WORK . Belief-Space RL Methods Bayesian reinforcement learning formalizes RL where one has a prior distribution over possible MDPs ( Ghavamzadeh et al. , 2015 ; Shani et al. , 2013 ) . However , the Bayes-optimal policy , which is the best one can do under uncertainty , is intractable to solve for and approximation is necessary ( Hsu et al. , 2008 ) . One way is to approximate the value function , as done in SARSOP ( Kurniawati et al. , 2008 ) and PBVI ( Pineau et al. , 2003 ) ; however , they can not deal with continous state actions . Another strategy is to resort to sampling , such as BAMCP ( Guez et al. , 2012 ) , POMCP ( Silver & Veness , 2010 ) , POMCPOW ( Sunberg & Kochenderfer , 2018 ) . However , these approaches require a significant amount of online computation . Online approaches forgo acting Bayes-optimally right from the onset , and instead aim to eventually act optimally . The question then becomes : how do we efficiently gain information about the test time MDP to act optimally ? BEB ( Kolter & Ng , 2009 ) and POMDP-lite ( Chen et al. , 2016 ) introduce an auxiliary reward term to encourage exploration and prove Probably-Approximately-Correct ( PAC ) optimality . This has inspired work on more general , non-Bayesian curiosity based heuristics for reward gathering ( Achiam & Sastry , 2017 ; Burda et al. , 2018 ; Pathak et al. , 2017 ; Houthooft et al. , 2016 ) . Online exploration is also well studied in the bandit literature , and techniques such as posterior sampling ( Osband et al. , 2019 ) bound the learner ’ s regret . UP-OSI ( Yu et al. , 2017 ) predicts the most likely MDP and maps that to an action . Gimelfarb et al . ( 2018 ) learns a gating over multiple expert value functions . However , online methods can over-explore and drive the agent to unsafe regimes . Another alternative is to treat belief MDP problems as a large state space that must be compressed . Peng et al . ( 2018 ) use Long Short-Term Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) to encode a history of observations to generate an action . Methods like BPO ( Lee et al. , 2019 ) explicitly utilize the belief distribution and compress it to learn a policy . The key difference between BRPO and BPO is that BRPO uses an expert , enabling it to scale to handle complex latent tasks that may require multimodal policies . Meta-reinforcement Learning Meta-reinforcement learning ( MRL ) approaches train sampleefficient learners by exploiting structure common to a distribution of MDPs . For example , MAML ( Finn et al. , 2017 ) trains gradient-based learners while RL2 ( Duan et al. , 2016 ) trains memory-based learners . While meta-supervised learning has well established Bayesian roots ( Baxter , 1998 ; 2000 ) , it wasn ’ t until recently that meta-reinforcement learning was strongly tied to Bayesian Reinforcement Learning ( BRL ) ( Ortega et al. , 2019 ; Rabinowitz , 2019 ) . Nevertheless , even non-Bayesian MRL approaches address problems pertinent to BRL . MAESN ( Gupta et al. , 2018 ) learns structured noise for exploration . E-MAML ( Stadie et al. , 2018 ) adds an explicit exploration bonus to the MAML objective . GMPS ( Mendonca et al. , 2019 ) exploit availability of MDP experts to partially reduce BRL to IL . Our work is more closely related to Bayesian MRL approaches . MAML-HB ( Grant et al. , 2018 ) casts MAML as hierarchical Bayes and improves posterior estimates . BMAML ( Yoon et al. , 2018 ) uses non-parametric variational inference to improve posterior estimates . PLATIPUS ( Finn et al. , 2018 ) learns a parameter distribution instead of a fixed parameter . PEARL ( Rakelly et al. , 2019 ) learns a data-driven Bayes filter across tasks . In contrast to these approaches , we use experts at test time , learning only to optimally correct them . Residual Learning Residual learning has its foundations in boosting ( Freund & Schapire , 1999 ) where a combination of weak learners , each learning on the failures of previous , make a strong learner . It also allows for injecting priors in RL , by boosting off of hand-designed policies or models . Prior work has leveraged known but approximate models by learning the residual between the approximate dynamics and the discovered dynamics ( Ostafew et al. , 2014 ; 2015 ; Berkenkamp & Schoellig , 2015 ) . There has also been work on learning residual policies over hand-defined ones for solving long horizon ( Silver et al. , 2018 ) and complex control tasks ( Johannink et al. , 2019 ) . Similarly , our approach starts with a useful initialization ( via experts ) and learns to improve via Bayesian reinforcement learning . 3 PRELIMINARIES : BAYESIAN REINFORCEMENT LEARNING . In Bayesian reinforcement learning , the agent does not know the reward and transition functions but knows that they are determined by a latent variable φ ∈ Φ . Formally , the problem is defined by a tuple 〈S , Φ , A , T , R , P0 , γ〉 , where S is the observable state space of the underlying MDPs , Φ is the latent space , and A is the action space . T and R are the transition and reward functions parameterized by φ . The initial distribution over ( s , φ ) is given by P0 : S × Φ→ R+ , and γ is the discount . Bayesian RL considers the long-term expected reward with respect to the uncertainty over φ rather than the true ( unknown ) value of φ . Uncertainty is represented as a belief distribution b ∈ B over latent variables φ . The Bayes-optimal action value function is given by the Bellman equation : Q ( s , b , a′ ) = R ( s , b , a′ ) + γ ∑ s′ , b′ P ( s′|s , b , a′ ) P ( b′|s , b , a′ ) max a′′ Q ( s′ , b′ , a′′ ) ( 1 ) The Bayesian reward function is the expected reward R ( s , b , a′ ) = ∑ φ∈Φ b ( φ ) R ( s , φ , a ′ ) . The Bayesian transition function is P ( s′|s , b , a′ ) = ∑ φ∈Φ b ( φ ) P ( s ′|s , φ , a′ ) . The posterior update P ( b′|s , b , a′ ) is computed recursively , starting from initial belief b0 . b′ ( φ′|s , b , a′ , s′ ) = η ∑ φ∈Φ b ( φ ) T ( s , φ , a′ , s′ , φ′ ) ( 2 ) where η is the normalizing constant , and the transition function is defined as T ( s , φ , a′ , s′ , φ′ ) = P ( s′ , φ′|s , φ , a′ ) = P ( s′|s , φ , a′ ) P ( φ′|s , φ , a′ , s′ ) . At timestep t , the belief bt ( φt ) is the posterior distribution over Φ given the history of states and actions , ( s0 , a1 , s1 , ... , st ) . When φ corresponds to physical parameters for an autonomous system , we often assume that the latent states are fixed . Algorithm 1 Bayesian Residual Policy Optimization Require : Bayes filter ψ , belief b0 , prior P0 , residual policy πθ0 , expert πe , horizon H , nitr , nsample 1 : for i = 1 , 2 , · · · , nitr do 2 : for n = 1 , 2 , · · · , nsample do 3 : Sample latent MDP M : ( s0 , φ0 ) ∼ P0 4 : τn ← Simulate ( πθi−1 , πe , b0 , ψ , M , H ) 5 : Update policy : θi ← BatchPolicyOptimization ( θi−1 , { τ1 , · · · , τnsample } ) 6 : return πθbest 7 : procedure SIMULATE ( πθ , πe , b0 , ψ , M , H ) 8 : for t = 1 , · · · , H do 9 : aet ∼ πe ( ·|st−1 , bt−1 ) // Expert recommendation 10 : at ← art + aet , art ∼ πθ ( st−1 , bt−1 , aet ) // Residual action 11 : Execute at on M , observe rt , st 12 : bt ← ψ ( st−1 , bt−1 , at , st ) 13 : τ ← ( s0 , b0 , ar1 , r1 , s1 , b1 , · · · , arH , rH , sH , bH ) // Only residuals are recorded 14 : return τ Our algorithm utilizes a black-box Bayes filter to produce a posterior distribution over the latent states . However , a Bayes filter can also be interpreted as a function that compresses the history of states and actions . Recent work suggests that Long Short-Term Memory ( LSTM ) cells ( Hochreiter & Schmidhuber , 1997 ) can be meta-trained to compress history and predict subsequent states ( Ortega et al. , 2019 ) . Such learned representations can be substituted for the belief distribution that we have chosen here .

Summary: This paper considers Bayesian Reinforcement Learning problem over latent Markov Decision Processes (MDPs). The authors consider making decisions with experts, where each expert performs well under some latent MDPs. An ensemble of experts is constructed, and then a Bayesian residual policy is learned to balance exploration-exploitation tradeoff. Experiments on Maze and Door show the advantages of residual policy learning over some baselines.