Uncertainty - sensitive learning and planning with ensembles

1 Introduction . The model-free and model-based approaches to reinforcement learning ( RL ) have a complementary set of strengths and weaknesses . While the former offers good asymptotic performance , it suffers from poor sample complexity . In contrast , the latter usually needs significantly less training samples , but often fails to achieve state-of-the-art results on complex tasks ( which is primarily attributed to models ’ imperfections ) . The interplay of model-based and model-free approaches in RL has received a lot of research attention . This led , for example , to strong AI systems like Silver et al . ( 2017 ; 2018 ) or more recently to Lowrey et al . ( 2018 ) , which is closely related to our work . When dealing with challenging new RL domains it is helpful to develop tools for addressing strategic and tactical decision-making . These two perspectives complement each other : strategic perspective is global , static , and often imprecise , while tactical perspective is local , dynamic and exact . We argue that value function can be considered as an example of the former , while a planner as an example of the latter . Indeed , value function approximators provide noisy estimates ( imprecise ) of values ( static ) to every state ( global ) . Conversely , planning provides optimal control , which starting from a given state ( local ) generates actions ( dynamic ) that are temporally coherent and result in a better-executed trajectories ( exact ) . It is only reasonable to combine the two into one system . This is achieved by plugging the value function into the planner , to provide guided heuristics in the local search , shorten the search horizon , and make the search computationally efficient . For complex problems , the above setup has to be supplemented by an exploration strategy . Such a strategy might be based on modeling uncertainty of the value function approximation ( e.g. , using dropout sampling , variational inference , distributional RL , bootstrap ensemble , etc. ) . The uncertainty is quantified by a risk measure , which is then utilized by a planner to guide exploration . Consider a situation of an agent with limited memory and computational resources , being dropped into a complex and diverse environment . This setup excludes any hope for brute force strategy and implies that a perfect planner is out of reach . In this paper , we consider Sokoban , a classic logical puzzle known for its combinatorial complexity and a recent benchmark for RL , ChainEnv , a seemingly impossible task also known as a ’ hay in a needle- stack ’ problem ( see Osband et al . ( 2018 ) ) , and Toy Montezuma ’ s Revenge , environment notoriously known for its exploration difficulty ( see Guo et al . ( 2019 ) ) . Importantly , we focus on sparse reward variants of the aforementioned environments ( see Section 3 ) . Since of our method involves planning , an agent has to have an access to the world model . In this paper we assume that it is the perfect model . The reason is that challenges presented by the selected environments are the dominating factor driving difficulty of the task ( in fact , answering the question of whether a Sokoban level is solvable is NP-hard , see e.g . Dor & Zwick ( 1999 ) ) . It is also in part due to this fact that a significant body of work dealing with difficult environments makes a similar assumption ( see e.g . Silver et al . ( 2017 ; 2018 ) , Orseau et al . ( 2018b ) ) . The main contribution of this work is showing how recent progress in AI can be brought together to improve planning , value function learning , and exploration , in a way that together they form robust algorithms for solving challenging reinforcement learning environments . In particular : 1 . For uncertainty modeling , we assume the point of view Osband et al . ( 2018 ) , which uses ensembles to approximate posterior distribution . 2 . In the spirit of Lowrey et al . ( 2018 ) , we incorporate risk measures to guide exploration . 3 . For the planner , we base on AlphaZero Monte-Carlo Tree Search ( see Silver et al . ( 2017 ) ) , which we enhance with the possibility of exploiting the graph structure of environments and the properties of the optimal solutions . 4 . In the value function training protocol , we introduce several improvements , including a version of self imitation learning mechanism ( Oh et al . ( 2018a ) ) and hindsight ( Andrychowicz et al . ( 2017 ) ) . The rest of the paper is organized as follows . In the next subsection we provide an overview of related work . In Section 2 we present and discuss our method in details . This is followed by a section with experiments and passing to conclusions . We provide code to our work https : //github.com/learningandplanningICLR/learningandplanning and a dedicated website https : //sites.google.com/view/learn-and-plan-with-ensembles with more details and movies . 1.1 Related work . The ideas of mixing model-based and model-free learning were perhaps first stated explicitly in Sutton ( 1990 ) . Many approaches followed . More recently , in the groundbreaking series of papers Silver et al . ( 2017 ) , Silver et al . ( 2018 ) culminating in AlphaZero , the authors have developed an elaborate system that plans and performs model-free training to master the game of Go ( and others ) . Similar ideas were also studied in Anthony et al . ( 2017 ) . Perhaps , the work which is closest to ours is Lowrey et al . ( 2018 ) . It is argued in the paper that an agent with limited computational resources in a complex environment needs both to plan and learn from the incoming stream of experience . Importantly , the value function in Lowrey et al . ( 2018 ) is modeled by an ensemble of value functions . The risk measure used to combine them is given by the ’ log-sum-exp formula ’ ( Lowrey et al. , 2018 , 6 ) . The authors show experimentally that this approach leads to improvements in various continuous tasks , including humanoid training . In our work , we deal with a discrete setting , which enforces a different planning module ( here MCTS-based ) . Moreover , we treat more diverse class of risk measures ( see Section 2 ) . Constructing neural network models that would incorporate uncertainty in a principled Bayesian way has proven to be challenging and remains and open problem . A promising new results using ensembles include Osband et al . ( 2018 ; 2017 ) , Lakshminarayanan et al . ( 2017 ) . Ensembles of models were also successfully used to improve model-based RL training , see Kurutach et al . ( 2018 ) , Chua et al . ( 2018 ) , and the references therein . We note that unprincipled ensemble methods are willingly used by RL practitioners . For example , in a recent competition Kidzinski et al . ( 2019 ) aimed to train an agent able to use a prosthetic leg , four top-ten solutions used some sort of ensembles . Another work similar to ours is Guo et al . ( 2014 ) , in which the authors use MCTS in the role of an ‘ expert ’ from which a neural policy is learnt using the dagger algorithm , Ross & Bagnell ( 2014 ) . The basic difference is that Guo et al . ( 2014 ) uses a classical MCTS without value function nor ensembles . Many works aim to build planning and learning into neural network architectures , see e.g . Oh et al . ( 2017 ) or Farquhar et al . ( 2017 ) . Kaiser et al . ( 2019 ) , a recent work on model-based Atari , has shown the possibility of sample efficient reinforcement learning with an explicit visual model . Gu et al . ( 2016 ) uses model-based at the beginning phase of training and model-free methods in ‘ fine-tuning ’ . Furthermore , there is a body of work that attempts to learn the planning module , see Pascanu et al . ( 2017 ) , Racanière et al . ( 2017 ) , and Guez et al . ( 2019 ) . Finally , our paper is related to research focusing on study of exploration . Fundamental results in this area concern the multi-arm bandits problem , see Lattimore & Szepesvári ( 2018 ) and the references therein . Methods developed in this area have been successfully applied in planning algorithms , see Kocsis et al . ( 2006 ) and Silver et al . ( 2017 ; 2018 ) . Furthermore , a measure with a loading on variance ( defined in Section 2 ) is related to UCB-V algorithm developed in Audibert et al . ( 2007 ) . Another set of methods have been developed in an attempt to solve notoriously hard Montezuma ’ s Revenge , see for example Ecoffet et al . ( 2019 ) or Guo et al . ( 2019 ) . In our work we use also hindsight , see Andrychowicz et al . ( 2017 ) . Its primarily motivation is to enrich the learning signal and train generalized value function . However , in certain situation it can be considered as an exploration algorithm . 2 Method description . In this section , we describe our method : the planning and exploration components , as well as the training protocol . Algorithm 1 shows how the components are brought together in the training loop . The pseudo-code for planner.run episode is listed in Algorithm 2 , and the remainder of pseudo-code can be found in Appendix A. Algorithm 1 Learning and planning with ensembles Require : Environment env , Model model 1 : Initialize parameters of value function ensemble θ = ( θ1 , . . . , θK ) , Vθ = ( Vθ1 , . . . , VθK ) 2 : Initialize replay_buffer 3 : repeat 4 : s← env.reset ( ) 5 : episode , solved← planner.run episode ( s ; model , Vθ ) . see Algorithm 2 6 : values← evaluate episode ( episode ) . see Algorithm 8 7 : Optionally calculate a mask . see Appendix C 8 : replay buffer.add ( episode , values , solved , mask ) . see Algorithm 9 9 : B ← replay_buffer.batch ( ) . B = { ( sb , vb , mb ) } , see Algorithm 10 10 : Update Vθ by one step of gradient descent . e.g . RMSProp ∇θi 1 |B| ∑ ( s , v , m ) ∈B mi ( Vθi ( s ) − v ) 2 + ζθ2i , for i ∈ { 1 , . . . , K } 11 : until convergence For the planer component we develop an MCTS-inspired algorithm.1 It is novel in two aspects : it utilizes a risk-sensitive policy ( for tree traversal and action choice ) intended 1MCTS is a family of algorithms that iteratively build a search tree , alternating between the following stages : tree traversal , leaf expansion and evaluation , and backpropagation , see Browne et al . ( 2012 ) for a survey on the topic . to guide exploration , and it includes methods exploiting the graph structure and enables avoiding cycles.2 The loop avoidance is achieved in two ways : by backpropagation of some fixed negative value through the in-tree path ending with a leaf having no unvisited neightbours ( see the pseudo-code in Appendix A related to dead ends ) , and during tree traversal , when the agent encouraged to avoid actions leading to previously visited states on the path ( see pseudo-code related to penaltyp in Appendix A ) . To reinforce this effect , the agent is also encouraged to avoid actions leading to previously visited states on the episode level ( see Algorithm 2 , where penaltye ) . These enhancements combined , make it possible to learn even in sparse rewards scenarios , which is experimentally demonstrated in Section 3 ( all used environments have sparse rewards ) . We note that Algorithm 1 is not MCTS specific , other planners could be used as well . The logic of our MCTS is laid out in Algorithm 2 . We assume that each node of the search tree , say n , stores a visit count n.count , accumulated value n.value , accumulated reward n.reward ( a ) , and its children are denoted by n.child ( a ) for each action a ∈ A . We define Q̂θ ( n , a ) = n.reward ( a ) + γn.child ( a ) .value , where γ > 0 is a discount rate . Algorithm 2 planner.run episode ( ) Require : max_episode_len mcts_passes penaltye . Episode penalty Vθ . Value function Input : s . Starting state 1 : episode← ∅ 2 : for step = 1 to max episode len do 3 : root← s 4 : root.value← root.value−penaltye 5 : for 1 to num mcts passes do 6 : path , leaf← traversal ( root ) 7 : value← expand leaf ( leaf ; Vθ ) 8 : backpropagate ( value , path ) 9 : a← choose action ( root , { root } ) 10 : s , r , done← model.step ( a ) 11 : episode.append ( ( root , a , r ) ) 12 : if done then break 13 : return episode , done Algorithm 3 choose action ( ) Require : avoid loops . Bool Input : n , seen 1 : if avoid loops then 2 : A← { a ∈ A ( n ) : n.child ( a ) /∈ seen } 3 : if A = ∅ then . Terminal or dead-end 4 : return None 5 : else 6 : A← A ( n ) 7 : Choose action according to equation 1 a∗ ← arg max a∈A Eθ∼Θ [ φa ( Q̂θ ( n ) ) ] 8 : return a∗ Our proposed risk-sensitive exploration method is implemented at the level of MCTS traversal tree , see step 7 in Algorithm 3 . Its key defining elements are uncertainty modeling and risk measurement . A risk-sensitive tree traversal policy is defined as follows : a∗ ( n ) : = arg max a Eθ∼Θ [ φa ( Q̂θ ( n ) ) ] , Q̂θ ( n ) : = ( Q̂θ ( n , a′ ) : a′ ∈ A ) , ( 1 ) where A is the action space , φa : R|A| → R is a risk measure , and Q̂θ is an estimator of the Q-function . The posterior distribution Θ models uncertainty in the value estimation . We implement the posterior Θ using ensembles , namely Eθ∼Θ [ f ( θ ) ] is approximated by 1 K ∑K i=1 f ( θi ) , where K is the size of an ensemble . This approximation stems from Osband et al . ( 2018 , Lemma 3 ) , which shows that for Bayesian regression in a Gaussian linear 2This is useful for a broad class of environments ( including the ones we used in the experimental part ) for which the optimal trajectory does not have cycles . model , generating samples from posterior distribution is equivalent to solving a suitable optimization problem , see Appendix C. In our case we use an ensemble of value functions . In some cases , we ” sub-sample ” from Θ , which is inspired by Osband et al . ( 2016 ) and the classical Thomson sampling ( see Appendix C for details ) . Using risk measure is inspired by Lowrey et al . ( 2018 ) , who used φa ( x ) = eκxa for x ∈ R|A| and κ > 0 . In this paper we consider the following choices of φa : • A measure with a loading on variance , φa ( x ) = xa+κx2a , κ > 0 . This includes second moments and can be easily generalized to include variance , standard deviation and exploration bonuses . • A relative majority vote ( also known as plurality vote ) measure , φa ( x ) = 1 ( arg max a′ xa′ = a ) . ( 2 ) Contrary to the other cases , φa defined in equation 2 depends not only on marginal values of its input , but the whole input ( i.e . the estimator vector Q̂θ ) . It leads to a rule resembling optimal Bayes classifier form , i.e . the one which chooses a minimizing P ( a∗θ ( s ) 6= a ) . The intuitions behind the aforementioned choices of φa ’ s are as follows . One can think of φa ( x ) = eκxa used in Lowrey et al . ( 2018 ) as a measure capturing all moments of value ensemble . For small values of κ ≈ 0 , it behaves like the measure with a loading on variance ( via the Tylor expansion ) . The mean approximates the value of a given action , while the variance quantifies the epistemic uncertainty . Taking weighted sum of these terms ( as we do in ” mean+var ” experiments , see Table 3 ) has the aim of balancing exploitation and exploration . It is also related to UCB-V algorithm , see Audibert et al . ( 2007 ) . Voting , on the other hand , is a well established approach when combining ensembles , see e.g . Breiman ( 1996 ) or Rokach ( 2010 ) . A relative majority vote , in particular , is simple and it can lead to good performance , see e.g . Osband et al . ( 2016 ) . In the context of planning and RL , it has several interesting properties . In particular , the distribution of votes across ensembles encodes the uncertainty related to the optimal action in a given state . High uncertainty may result in stochastic movement ( caused by a uniform tie breaking ) , and consequently lead to higher exploration . On the other hand , low uncertainty may result in an exploitative behaviour of an agent and , as a result , lower exploration . Additionally , voting may improve decision making of an agent in the states , where some action can be dangerous ( e.g . lead to irreversible states ) . We conclude this section by describing the part of Algorithm 1 related to value functions update . The episodes are collected and stored in a replay buffer . The replay buffer performs some addition bookkeeping by storing , for each transition , information whether it comes from a solved episode or not ( in the case of environments that provide such information , like Sokoban ) . This mechanism allows sampling batches with a fixed ratio of solved to unsolved transitions ( see Algorithm 8 in Appendix A ) . Such a method resembles self-imitation techniques , see e.g . Oh et al . ( 2018b ) . The value functions are trained to minimize l2 distance from target values sampled from the buffer . The target values can be computed in two modes : bootstrap , which utilizes the values accumulated during the MCTS phase , and factual , corresponding to discounted rewards in an episode ( see Algorithm 8 in Appendix A ) . Additionally , some experiments use masks which , analogously to Osband et al . ( 2018 ) , form a mechanism of assigning a transition to a value function ( see Appendix C ) . Finally , we implement ensembles as a set of neural networks ( we also experiment with multi-headed architectures sharing some lower layers ) . We use both architectures with or without random priors , for details see Appendix B and Osband et al . ( 2018 ) .

Summary: The paper proposes an approach blending model-based, model-free methods and utilizing risk-sensitivity information in ensembles as part of the value estimation and exploration process. The exploration is based on risk- sensitivity measures such as moments and relative majority vote. There is a lot of work currently trying to marry the model free with model based approaches for integrated planning and learning as the authors have mentioned in the related work section of the paper and also called out similar methods and techniques. The authors have provided evidence via experiments in three environments and shown good results of using this blended approach. Code is also provided for others to further carry out explorations in this research area.