Width-based Lookaheads with Learnt Base Policies and Heuristics Over the Atari-2600 Benchmark

1 Introduction . The Atari-2600 games provide useful environments for benchmarking autonomous agents due to the diversity of behaviour required across the different games . The Atari-2600 games can be accessed through the Arcade Learning Environment ( ALE ) [ 1 ] which provides a typical Reinforcement Learning ( RL ) environment interface where given a state , the agent selects an action and receives a resulting state and reward . The two main approaches that have been used by autonomous agents applied to the Atari-2600 games have been RL methods [ 2 , 3 , 4 ] and Planning methods [ 5 , 6 ] . The RL approaches have had great success surpassing the performance of human players for many of the Atari-2600 games . However , RL approaches require long training times in order to train the Neural Networks ( NN ) used for policy and value functions . Planning agents do not require training time and instead use a bounded , fixed computational budget to decide which action to take at each time step of the game . The budget allowed for planning for each action is set as part of the experimental setting and can be set in such a way that the agent can play a game in real-time . Through the ALE interface , the agent is not provided a description of the transition or reward functions as is the case of models described through languages such as the Planning Domain Description Language ( PDDL ) [ 7 ] . Instead , planning agents applied to the Atari-2600 games are required to work with a simulator , treating the environment ’ s transition and reward functions as a black-box [ 5 ] . Width-based planning agents have been shown to be particularly successful on the Atari-2600 games when compared to other planning agents [ 5 , 6 ] . Width-based planners prioritise search effort on states deemed to be novel . The novelty of a state can be defined in a number of ways . Previously , novelty tests have been obtained from the RAM of the game [ 5 ] , handcrafted features computed from screen 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . pixels [ 6 ] and learnt features extracted from the screen pixels through a NN [ 8 , 9 , 10 ] . In this paper we consider planners with a novelty measure that does not require extensive feature engineering , or the internal state of the simulator , but is instead defined directly over the values of screen pixels . Recent approaches have combined the RL and planning methods into single agents that are applied to the Atari-2600 games [ 8 , 11 , 10 ] . Junyent et al . [ 8 ] combined a width-based planner with a learnt policy defined over a NN in order to guide the planner to promising areas of the search space . The learnt NN was also used to extract features from which the novelty of states were defined over . In this paper we introduce new width-based planning and learning methods that learn both policy and value networks using a methodical learning schedule . Through analysing previous width-based methods we construct and benchmark new width-based approaches for the Atari-2600 games . We also classify the Atari-2600 games according to their particular characteristics . The resulting game taxonomy helps us to gain insight into the performance of the algorithms we propose and benchmark . The paper contributions are : ( 1 ) an analysis of the previous width-based planning methods that have been applied to the Atari-2600 games , ( 2 ) introducing new width-based planning and learning approaches for playing the Atari-2600 games , ( 3 ) defining a methodical learning schedule for planning and learning methods , and ( 4 ) identifying characteristics of the Atari-2600 games that influence the performance of different planning approaches . 2 Background . 2.1 MDPs . We model the Atari games as Markov Decision Processes ( MDPs ) . We formalise MDPs , as described by Geffner and Bonet [ 12 ] , as the tuple of 𝑀 = ( S , 𝑠0 , 𝐴 , 𝑇 , 𝑅 ) , where S ⊆ R𝑑 , 𝑠0 ∈ S is the initial state , 𝐴 provides the sets of applicable actions such that 𝐴 ( 𝑠 ) is a set of actions applicable in 𝑠 ∈ S , 𝑇 is a set of distributions such that 𝑇 ( 𝑠 , 𝑎 , 𝑠′ ) gives the probability of the transition from state 𝑠 ∈ S to state 𝑠′ ∈ S given action 𝑎 ∈ 𝐴 ( 𝑠 ) , and 𝑅 is the reward function such that 𝑅 ( 𝑠 , 𝑎 ) returns the reward for performing action 𝑎 ∈ 𝐴 ( 𝑠 ) from state 𝑠 ∈ S. In this work we will be considering a special case , finite-horizon MDPs , where accumulated rewards need to be maximised over a given number of stages 𝑘 = 1 , . . . , 𝐻 , starting at a fixed initial state , 𝑠0 . Terminal states in finite-horizon MDPs are absorbing states . That is , if 𝑠 is a terminal state and we are at time step 𝑘 , every action 𝑎 will map ( 𝑠 , 𝑘 ) into ( 𝑠 , 𝑘 + 1 ) and will be reward-free i.e . 𝑅 ( 𝑠 , 𝑎 ) = 0 . Our goal is to produce a policy , 𝜋 , that maps any given state into an action , such that it maximises the expected accumulated reward received for an episode of the MDP , argmax𝜋𝐸 { 𝐻−1∑︁ 𝑘=0 𝑅 ( 𝑠𝑘 , 𝜋 ( 𝑠𝑘 ) ) } ( 1 ) where the expectation is over 𝑠𝑘+1 ∼ 𝑇 ( 𝑠𝑘 , 𝜋 ( 𝑠𝑘 ) , 𝑠𝑘+1 ) . We assume that we have access to a simulator of the environment that given any state-action pair ( 𝑠 , 𝑎 ) , where 𝑠 ∈ S and 𝑎 ∈ 𝐴 ( 𝑠 ) , the simulator returns the reward 𝑅 ( 𝑠 , 𝑎 ) , a resulting state 𝑠′ following the probability distribution 𝑇 ( 𝑠 , 𝑎 , 𝑠′ ) and whether 𝑠′ is a terminal state . In line with previous work [ 6 , 8 , 10 ] , we consider only discrete action sets and states that represent the internal state of the Atari environment such that action transitions are deterministic , that is 𝑇 ( 𝑠 , 𝑎 , 𝑠′ ) can only equal 1 for one state 𝑠′ and 0 for any other state 𝑠′′ , 𝑠 , 𝑠′ , 𝑠′′ ∈ S , 𝑠′ ≠ 𝑠′′ and 𝑎 ∈ 𝐴 ( 𝑠 ) . Note that while the agent can use the internal Atari game state to set the state of the simulator , it can only directly observe the Atari screen ’ s pixel value for any given state . 2.2 Online Planning over simulators . In this paper we explore online planning over simulators by considering width-based lookahead algorithms for the Atari-2600 games . Lookaheads use a simulator of the environment to consider rewards from different action trajectories from the current state into the future . An example of this is shown in Figure 1 where a lookahead is illustrated . We define the notion of lookahead as , Definition 1 ( Lookahead ) . A lookahead is defined as L = ( 𝑁 , 𝐶 , 𝑠𝑟 ) where 𝑁 is a set of nodes defined as state-action paths starting at the root state of the lookahead 𝑠𝑟 , and 𝐶 is a function that given a node 𝑛 ∈ 𝑁 and an action 𝑎 ∈ 𝐴 ( 𝑠 ) returns the children of 𝑛 , that is 𝑛𝑐 ∈ 𝐶 ( 𝑛 , 𝑎 ) and 𝑛𝑐 ∈ 𝑁 . Through backing up the rewards for each node in the lookahead , as shown in Figure 1 , an expected value can be found for each action applicable at the current state . That is , where 𝑛𝑠 is the last state along the state-action path of node 𝑛 , the operation of backing up the rewards and selecting which action to execute is , Definition 2 ( Action selection of lookahead ) . Given a lookahead L= ( 𝑁 , 𝐶 , 𝑠𝑟 ) ( Definition 1 ) the action to execute 𝑎 is selected at the root 𝑛𝑟 , where 𝑛𝑠𝑟 = 𝑠𝑟 , by argmax𝑎∈𝐴 ( 𝑛𝑠𝑟 ) { 𝑄 ( 𝑛𝑟 , 𝑎 ) } , where𝑄 ( 𝑛𝑟 , 𝑎 ) = 𝑅 ( 𝑛𝑠𝑟 , 𝑎 ) + ∑ 𝑛∈𝐶 ( 𝑛𝑟 , 𝑎 ) 𝑇 ( 𝑛𝑠𝑟 , 𝑎 , 𝑛𝑠 ) 𝑉 ( 𝑛 ) , and 𝑉 ( 𝑛 ) = 𝑉𝑇 ( 𝑛 ) , with 𝑉𝑇 ( 𝑛 ) being a termination cost , when ⋃ 𝑎∈𝐴 ( 𝑛𝑠 ) 𝐶 ( 𝑛 , 𝑎 ) = ∅ , otherwise 𝑉 ( 𝑛 ) = max𝑎 { 𝑅 ( 𝑛𝑠 , 𝑎 ) + ∑ 𝑛′∈𝐶 ( 𝑛 , 𝑎 ) 𝑇 ( 𝑛𝑠 , 𝑎 , 𝑛′𝑠 ) 𝑉 ( 𝑛′ ) } . Once an action is selected for execution , the lookahead is updated to have its root at the selected action ’ s resulting node and the lookahead continues being constructed from the new root node . We define the action selected by the agent as a part of its critical path . That is , Definition 3 ( Critical Path ) . Given the action selected 𝑎𝑡 at each time step 𝑡 = 0 , 1 , . . . , 𝑚 following Definition 2 , the critical path 𝜌 is the sequence of states and actions 𝜌 = ( 𝑠0 , 𝑎0 , 𝑠1 , 𝑎1 , . . . , 𝑎𝑚−1 , 𝑠𝑚 ) , such that 𝑠𝑖+1 ∼ 𝑇 ( 𝑠𝑖 , 𝑎𝑖 , · ) for 𝑖 = 0 , . . . , 𝑚 − 1 . 3 Related Work . 3.1 MuZero . Schrittwieser et al . [ 11 ] followed up on the AlphaGo [ 13 ] and AlphaZero [ 14 ] algorithms with MuZero . AlphaZero is a planning and learning agent that achieved state-of-the-art performance on the games of Go , Chess and Shogi [ 15 ] . AlphaZero learns a policy and value network that are used within a Monte Carlo Tree Search ( MCTS ) lookahead through sampling actions according to the policy network and evaluating the states within the lookahead with the value function . Unlike AlphaZero , MuZero does not require a simulator or model of the game environment but instead learns a model of the environment through interaction . MuZero achieved state-of-the-art performance in the Atari-2600 games when compared to existing model-free RL algorithm performances . We also explore using learnt value and policy networks within a lookahead but consider width-based methods as opposed to MCTS . MuZero ’ s experimental setting is different to the one considered in this paper as we require access to a simulator in the planning phase and use significantly less computing power . 3.2 Width-based Planners on Atari . Here we will provide an overview of the different width-based planners which have been applied to the Atari-2600 problems . In the next Section we go into the design and implementation details that each of the following planners use . Width-based planners [ 16 ] prioritise adding states to the lookahead with novel valuations of features that are defined over the states . IW ( 1 ) is a width-based breadth-first search that is guaranteed to run in linear time and space as it only expands novel states . IW ( 1 ) considers a state in the lookahead as novel if it is the first state within the lookahead to make a particular feature within the feature set true . Width-based planners were first applied to the Atari-2600 games by Lipovetzky et al . [ 5 ] , where they applied IW ( 1 ) over the RAM values of the game state as features . Lipovetzky et al. , and subsequent works that use IW ( 1 ) [ 17 , 18 ] , show that it outperforms breadth-first search and UCT [ 19 ] planners . Algorithm 1 : Overview of the RIW ( 1 ) Algorithm Input : A lookahead 𝐿 = ( 𝑁 , 𝐶 , 𝑠𝑟 ) , and a base policy 𝜋𝑏 Output : Updated lookahead 𝐿 1 while ¬ has_solved_label ( 𝑠𝑟 ) do 2 𝑠← 𝑠𝑟 // complete depth-first rollout from the root node ’ s state 3 while is_novel ( s ) ∧¬ is_terminal ( s ) do 4 𝑠′ , 𝑎 ←sample_unsolved_child ( 𝑠 , 𝜋𝑏 ) 5 𝐿 ← update_lookahead ( 𝐿 , 𝑠 , 𝑎 , 𝑠′ ) , 𝑠← 𝑠′ 6 end 7 update_solved_labels ( 𝑠 ) 8 end Bandres et al . [ 6 ] introduced a depth-first version of the IW ( 1 ) planner , named Rollout-IW ( 1 ) ( RIW ) . RIW ( 1 ) aims to contain the same nodes that are expanded by the IW ( 1 ) planner . As RIW ( 1 ) performs depth-first search it was argued by Bandres et al . that it has better any-time performance than IW ( 1 ) . The hypothesis for the better any-time performance of RIW ( 1 ) is that its search visits states that are further away from the initial state earlier in the search than its breadth-first search counter-part IW ( 1 ) . Algorithm 1 provides an overview of RIW ( 1 ) using the base policy 𝜋𝑏 . RIW ( 1 ) was originally defined to use a random uniform base policy 𝜋𝑏 , however any given policy can be used instead . The function sample_unsolved_child , samples an action 𝑎 ∼ 𝜋𝑏 ( 𝑠 ) and a transition 𝑠′ ∼ 𝑇 ( 𝑠 , 𝑎 , · ) provided that 𝑠′ has not been marked as solved . If the selected transition does not already exist in the lookahead update_lookahead adds it . The update_solved_labels function adds a solved label to the given state and back-propagates the solved label to its parents if the parent ’ s children have all been marked as solved . Bandres et al . results showed that RIW ( 1 ) outperformed IW ( 1 ) greatly when almost real-time budgets for planning were applied . Junyent et al . [ 8 ] follow up on Bandres et al. ’ s [ 6 ] use of a random policy for the base policy ( 𝜋𝑏 ) of RIW with 𝜋-IW ( 1 ) , an algorithm that replaces 𝜋𝑏 with a trained policy defined over a NN . The intended effect is to orient the lookahead to promising areas of the state space . The NN from the trained policy is also used to extract features from the screen pixels for the computation of state novelty . Recently Junyent et al . [ 10 ] introduced 𝜋-IW ( 1 ) + and 𝜋-HIW ( n , 1 ) as follow ups of 𝜋-IW ( 1 ) . 𝜋-IW ( 1 ) + modifies 𝜋-IW ( 1 ) ’ s random breaking of ties for the action selection ( Definition 2 ) to select the action with the branch of the lookahead that contains the most nodes . 𝜋-IW ( 1 ) + also adds a learnt value function , ? ̃ ? , which is used in the action selection ( Definition 2 ) by modifying 𝑉 ( 𝑠 ) to be max { ? ̃ ? ( 𝑠 ) , 𝑉∗ ( 𝑠 ) } , where 𝑉∗ ( 𝑠 ) is 𝑉 ( 𝑠 ) as described in Definition 2. 𝜋-HIW ( n , 1 ) is a hierarchical algorithm that has a high-level planner which uses a coarse down-sampling of the screen pixels as a feature set and a low-level planner which uses 𝜋-IW ( 1 ) + with the feature set defined through the policy network as previously described . The high-level planner uses a modified stochastic exploration policy , that selects actions with probability inversely proportional to state visitation counts .

Summary:   The paper proposes a new state of the art in planning-based approach   to Atari game.  The paper carefully analyses existing work, including   $pi$-IW+ which uses both value and policy functions, and fixes several   incorrect decisions that qualitatively does not make sense, such as   mixing up the estimate for the base and the critical path policy.    Perhaps what I would ask the authors is the way the paper is written:   The current text is not written in the style of "They do X which has   this issue; therefore we do Y" --- issue-focused description, but in   the style of "We do Y; X does this; X is problematic".  For example,   Sec 4.2 could be titled as "Addressing the complex novelty definition   in RIW", or "Simplifying the novelty; depth-based novelty is not   necessary".    The algorithm also has a mechanism inspired by AlphaZero, which   resembles active-learning (section 4.5), which compares the old and   the updated parameters.  Following up work should more carefully   evaluate this mechanism as an active-learning scheme.    Its second contribution is the proposal of a standard evaluation   scheme, which is now an adequate proposal due to the recent   improvement in the field.    The proposed algorithm is carefully evaluated empirically,   characterizing the sparseness and the branching factor.    This is a solid paper that I recommend for acceptance.