Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions

1 INTRODUCTION . Reinforcement Learning ( RL ) has made great progress in recent years , successfully being applied to settings such as board games ( Silver et al. , 2017 ) , video games ( Mnih et al. , 2015 ) and robot tasks ( OpenAI et al. , 2018 ) . Some of this advance is due to the use of deep learning techniques to avoid induction biases in the mapping of sensory information to states . However , widespread adoption of RL in real-world domains has remained limited primarily because of its poor sample efficiency , a dominant concern in RL ( Wu et al. , 2017 ) , and complexity of the training process that need to be managed by various heuristics . Hierarchical Reinforcement Learning ( HRL ) attempts to improve the sample efficiency of RL agents by making their policies to be hierarchical rather than single level . Using hierarchical policies can lead not only to faster learning , but also ease human understanding of the agent ’ s behaviour – this is because higher-level action representations are easier to understand than low-level ones ( Beyret et al. , 2019 ) . Identifying the right hierarchical policy structure is , however a non-trivial task ( Osa et al. , 2019 ) and so far progress in hierarchical RL has been slow and incomplete , as no truly scalable and successful hierarchical architectures exist ( Vezhnevets et al. , 2016 ) . Not surprisingly most stateof-the-art RL agents at the moment are not hierarchical . In contrast , humans , use hierarchical grammatical principles to communicate using spoken/written language ( Ding et al. , 2012 ) but also for forming meaningful abstractions when interacting with various entities . In the language case , for example , to construct a valid sentence we generally combine a noun phrase with a verb phrase ( Yule , 2015 ) . Action grammars are an analogous idea proposing there is an underlying set of rules for how we hierarchically combine actions over time to produce new actions . There is growing neuroscientific evidence that the brain uses similar processing strategies for both language and action , meaning that grammatical principles are used in both neural representations of language and action ( Faisal et al. , 2010 ; Hecht et al. , 2015 ; Pastra & Aloimonos , 2012 ) . We hypothesise that using action grammars would allow us to form a hierarchical action represen- tation that we could use to accelerate learning . Additionally , in light of the neuroscientific findings , hierarchical structures of action may also explain the interpretability of hierarchical RL agents , as their representations are structurally more similar to how humans structure tasks . In the following we will explore the use of grammar inference techniques to form a hierarchical representation of actions that agents can operate on . Just like much of Deep RL has focused on forming unbiased sensory representations from data-driven agent experience , here we explore how data-driven agent experience of actions can contribute to forming efficient representations for learning and controlling tasks . Action Grammar Reinforcement Learning ( AG-RL ) operates by using the observed actions of the agent within a time window to infer an action grammar . We use a grammar inference algorithm to substitute repeated patterns of primitive actions ( i.e . words ) into temporal abstractions ( rules , analogous to a sentence ) . Similarly , we then replace repeatedly occurring rules of temporal abstractions with higher-level rules of temporal abstractions ( analogous to paragraphs ) , and so forth . These extracted action grammars ( the set of rules , rules of rules , rules of rules of rules , etc ) is appended to the agent ’ s action set in the form of macro-actions so that agents can choose ( and evaluate ) primitive actions as well as any of the action grammar rules . We show that AG-RL is able to consistently and significantly improve sample efficiency across a wide range of Atari settings . 2 RELATED WORK . Our concept of action grammars that act as efficient representations of actions is both related to and different from existing work in the domain of Hierarchical Control . Hierarchical control of temporally-extended actions allows RL agents to constrain the dimensionality of the temporal credit assignment problem . Instead of having to make an action choice at every tick of the environment , the top-level policy selects a lower-level policy that executes actions for potentially multiple timesteps . Once the lower-level policy finishes execution , it returns control back to top-level policy . Identification of suitable low level sub-policies poses a key challenge to HRL . Current approaches can be grouped into three main pillars : Graph theoretic ( Hengst , 2002 ; Mannor et al. , 2004 ; Simsek et al. , 2004 ) and visitation-based ( Stolle & Precup , 2002 ; Simsek et al. , 2004 ) approaches aim to identify ” bottlenecks ” within the state space . Bottlenecks are regions in the state space which characterize successful trajectories . Our work , on the other hand , identifies patterns solely in the action space and does not rely on reward-less exploration of the state space . Furthermore , our action grammar framework defines a set of macro-actions as opposed to full option-specific sub-policies . Thereby , it is less expressive but more sample-efficient to infer . Gradient-based approaches , on the other hand , discover parametrized temporally-extended actions by iteratively optimizing an objective function such as the estimated expected value of the log likelihood of the observed data under the current policy with respect to the latent variables in a probabilistic setting ( Daniel et al. , 2016 ) or the expected cumulative reward in a policy gradient context ( Bacon et al. , 2017 ; Smith et al. , 2018 ) . Our action grammar induction , on the other hand , infers patterns without supervision solely based on a compression objective . The resulting parse tree provides an interpretable structure for the distilled skill set . Furthermore , recent approaches in macro-action discovery ( Vezhnevets et al. , 2017 ; Florensa et al. , 2017 ) attempt to split the goal declaration and goal achievement across different stages and layers of the learned architecture . Thus , while hierarchical construction of goals follows a set of coded rules , our action grammars are inferred entirely data-driven based on agent experience . Usually , the top level of the hierarchy specifies goals in the environment while the lower levels have to achieve them . Again , such architectures lack sample efficiency and easy interpretation . Our context-free grammar-based approach , on the other hand , is a symbolic method that requires few rollout traces and generalizes to more difficult task-settings . Finally , unlike recent work on unifying symbolic and connectionist methods , we do not aim to discover relationships between entities ( Garnelo et al. , 2016 ; Zambaldi et al. , 2018 ) . Instead our proposed action grammar framework achieves interpretability by extracting hierarchical subroutines associated with sub-goal achievements ( Beyret et al. , 2019 ) . 3 METHODOLOGY . The Action Grammar Reinforcement Learning framework operates by having the agent repeatedly iterate through two steps ( as laid out in Figure 1 ) : ( A ) Gather Experience : the base off-policy RL agent interacts with the environment and stores its experiences ( B ) Identify Action Grammar : the experiences are used to identify the agent ’ s action grammar which is then appended to the agent ’ s action set in the form of macro-actions Hierarchical RL with Action Grammars Algorithm During the first Gather Experience step of the game the base RL agent plays normally in the environment for some set number of episodes . The only difference is that during this time we occasionally run an episode of the game with all random exploration turned off and store these experiences separately . We do this because we will later use these experiences to identify the action grammar and so we do not want them to be influenced by noise . See left part of Figure 1 . After some set number of episodes we pause the agent ’ s interaction with the environment and enter the first Identify Action Grammar stage , see middle part of Figure 1 . This firstly involves collecting the actions used in the best performing of the no-exploration episodes mentioned above . We then feed these actions into a grammar calculator which identifies the action grammar . A simple choice for the grammar calculator is Sequitur ( Nevill-Manning & Witten , 1997 ) . Sequitur receives a sequence of actions as input and then iteratively creates new symbols to replace any repeating subsequences of actions . These newly created symbols then represent the macro-actions of the action grammar . To minimise the influence of noise on grammar generation , however , we need to regularise the process . A naive regulariser is k-Sequitur ( Stout et al. , 2018 ) which is a version of Sequitur that only creates a new symbol if a sub-sequence repeats at least k times ( instead of at least two times ) , where higher k corresponds to stronger regularisation . Here we use a more principled approach and regularise on the basis of an information theoretic criterion : we generate a new symbol if doing so reduces the total amount of information needed to encode the sequence . After we have identified the action grammar we enter our second Gather Experience step . This firstly involves appending the macro-actions in the action grammar to the agent ’ s action set . To do this without destroying what our q-network and/or policy has already learned we use transfer learning . For every new macro-action we add a new node to the final layer of the network , leaving all other nodes and weights unchanged . We also initialise the weights of each new node to the weights of their first primitive action ( as it is this action that is most likely to have a similar action value to the macro-action ) . e.g . if the primitive actions are { a , b } and we are adding macro-action abb then we initialise the weights of the new macro-action to those of a . Then our agent begins interacting with the environment as normal but with an action set that now includes macro-actions and four additional changes : i ) In order to maximise information efficiency , when storing experiences from this stage onwards we use a new technique we call Hindsight Action Replay ( HAR ) . It is related to Hindsight Experience Replay which creates new experiences by reimagining the goals the agent was trying to achieve . Instead of reimagining the goals , HAR creates new experiences by reimagining the actions . In particular it reimagines them in two ways : 1 . If we play a macro-action then we also store the experiences as if we had played the sequence of primitive actions individually 2 . If we play a sequence of primitive actions that matches an existing macro-action then we also store the experiences as if we had played the macro-action See Appendix A for an example of how HAR is able to more than double the number of collected experiences in some cases . ii ) To make sure that the longer macro-actions receive enough attention while learning , we sample experiences from an action balanced replay buffer . This acts as a normal replay buffer except it returns samples of experiences containing equal amounts of each action . iii ) To reduce the variance involved in using long macro-actions we develop a new technique called Abandon Ship . During every timestep of conducting a macro-action we calculate how much worse it is for the agent to continue executing its macro-action compared to abandoning the macro-action and picking the highest value primitive action instead . Formally we calculate this value as d = 1− exp ( qm ) exp ( qhighest ) where qm is the action value of the primitive action we are conducting as part of the macro-action and qhighest is the action value of the primitive action with the highest action value . We also store the moving average , m ( d ) , and moving standard deviation , std ( d ) , of d. Then each timestep we compare d to threshold t = m ( d ) +std ( d ) z where z is the abandon ship hyperparameter that determines how often we will abandon macro-actions . If d > t then we abandon our macroaction and return control back to the policy , otherwise we continue executing the macro-action . Algorithm 1 AG-RL 1 : Initialise environment env , base RL algorithm R , replay buffer D and action set A 2 : for each iteration do 3 : F ← GATHER EXPERIENCE ( A ) 4 : A← IDENTIFY ACTION GRAMMAR ( F ) 5 : 6 : procedure GATHER EXPERIENCE ( A ) 7 : transfer learning ( A ) . If action set changed do transfer learning 8 : F ← ∅ . Initialise F to store no-exploration episode experiences 9 : for each episode do 10 : if no exploration time then turn off exploration . Periodically turn off exploration 11 : E ← ∅ . Initialise E to store an episode ’ s experiences 12 : while not done do 13 : mat = R.pick action ( st ) . Pick next primitive action / macro-action 14 : for at in mat do . Iterate through each primitive action in the macro-action 15 : if abandon ship ( st , at ) then break . Abandon macro-action if required 16 : st+1 , rt+1 , dt+1 = env.step ( at ) . Play action in environment 17 : E ← E ∪ { ( st , at , rt+1 , st+1 , dt+1 ) } . Store the episode ’ s experiences 18 : R.learn ( D ) . Learning iteration for base RL algorithm 19 : D ← D ∪ HAR ( E ) . Use HAR when updating replay buffer 20 : if no exploration time then F ← F ∪ E . Store no-exploration experiences 21 : return F 22 : 23 : procedure IDENTIFY ACTION GRAMMAR ( F ) 24 : F ← extract best episodes ( F ) . Keep only the best performing no-exploration episodes 25 : action grammar← grammar algorithm ( F ) . Infer action grammar using experiences 26 : A← A ∪ action grammar . Update the action set with identified macro-actions 27 : return A iv ) When our agent is picking random exploration moves we bias its choices towards macro-actions . For example , when a DQN agent picks a random move ( which it does epsilon proportion of the time ) we set the probability that it will pick a macro-action , rather than a primitive action , to the higher probability given by the hyperparameter “ Macro Action Exploration Bonus ” . In these cases , we do not use Abandon Ship and instead let the macro-actions fully roll out . The second Gather Experience step then continues until it is time to do another Identify Action Grammar step or until the agent has been trained for long enough and the game ends . Algorithm 1 provides the full AG-RL algorithm .

Summary: The authors propose a method for learning macro-actions in a multi-step manner, where Sequitur, a grammar calculator, is leveraged together with an entropy-minimisation based strategy to find relevant macro-actions. The authors propose a system to bootstrap the weights of these macro-actions when increasing the policy's action space, and a system to increase the amount of data (and bias it towards macro-actions) used to learn a policy for when conditioned on this increased action-space. The authors test against a subset of the Arcade Learning Environment suite.