Better state exploration using action sequence equivalence

1 INTRODUCTION . Despite the rapidly improving performance of Reinforcement Learning ( RL ) agents on a variety of tasks ( Mnih et al. , 2015 ; Silver et al. , 2016 ) , they remain largely sample-inefficient learners compared to humans ( Toromanoff et al. , 2019 ) . Contributing to this is the vast amount of prior knowledge humans bring to the table before their first interaction with a new task , including an understanding of physics , semantics , and affordances ( Dubey et al. , 2018 ) . The considerable quantity of data necessary to train agents is becoming more problematic as RL is applied to ever more challenging and complex tasks . Much research aims at tackling this issue , for example through transfer learning ( Rusu et al. , 2016 ) , meta learning , and hierarchical learning , where agents are encouraged to use what they learn in one environment to solve a new task more quickly . Other approaches attempt to use the structure of Markov Decision Processes ( MDP ) to accelerate learning without resorting to pretraining . Mahajan & Tulabandhula ( 2017 ) and Biza & Jr. ( 2019 ) learn simpler representations of MDPs that exhibit symmetrical structure , while van der Pol et al . ( 2020 ) show that environment invariances can be hard-coded into equivariant neural networks . A fundamental challenge standing in the way of improved sample efficiency is exploration . We consider a situation where the exact transition function of a Markov Decision Process is unknown , but some knowledge of its local dynamics is available under the form of a prior expectation that given sequences of actions have identical results . This way of encoding prior knowledge is sufficiently flexible to describe many useful environment structures , particularly when actions correspond to agent movement . For example , in a gridworld ( called RotationGrid hereafter ) where the agent can move forward ( ↑ ) and rotate 90◦to the left ( x ) or to the right ( y ) , the latter two actions are the inverse of each other , in that performing one undoes the effect of the other . During exploration , to encourage the visitation of not yet seen states , it is natural to simply ban sequences of actions that revert to previously visited states , following the reasoning of Tabu search ( Glover , 1986 ) . We observe further that yy and xx both lead to the same state ( represented as state 4 in Figure 1 ) . If actions were uniformly sampled , the chances of visiting this state would be much higher than any of the others . Based on these observations , we introduce a new method taking advantage of Equivalent Action SEquences for Exploration ( EASEE ) , an overview of which can be found in Figure 1 . EASEE looks ahead several steps and calculates action sampling probabilities to explore as uniformly as possible new states conditionally on the action sequence equivalences given to it . It constructs a partial MDP which corresponds to a local representation of the true MDP around the current state . We then formulate the problem of determining the best distribution over action sequences as a linearly constrained convex optimization problem . Solving this optimization problem is computationally inexpensive and can be done once and for all before learning begins , providing a principled and tractable exploration policy that takes into account environment structure . This policy can easily be injected into existing reinforcement learning algorithms as a substitute for -greedy exploration . Our contribution is threefold . First , we formally introduce the notion of equivalent action sequences , a novel type of structure in Markov Decision Processes . Then , we show that priors on this type of structure can easily be exploited during offline exploration by solving a convex optimization problem . Finally , we provide experimental insights and show that incorporating EASEE into a DQN ( Mnih et al. , 2015 ) improves agent performance in several environments with various structures . Overview We assume that we have sets of equivalent action-sequences for the environment . Equivalent action sequences are sequences that lead to the same state . These sequences are used to build a DAG that models where the agent will end up after any sequence of actions of length d. Because some sequences are equivalent , several parent nodes may share a child node . A naive exploration scheme like -greedy would waste resources by over exploring such child nodes . Instead , we leverage this information using the DAG constructed above ; our method executes an exploratory action that maximizes the entropy of the future visited states . 2 RELATED WORK . Improved Exploration The problem of ensuring that agents see sufficiently diverse states has received a lot of attention from the RL community . Many methods rely on intrinsic rewards ( Schmidhuber , 1991 ; Chentanez et al. , 2005 ; Şimşek & Barto , 2006 ; Lopes et al. , 2012 ; Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Pathak et al. , 2017 ) to entice agents to unseen or misunderstood areas . In the tabular setting , these take the form of count-based exploration bonuses which guide the agent toward poorly visited states ( e.g . Strehl & Littman ( 2008 ) ) . Scaling this method requires the use of function approximators ( Burda et al. , 2019 ; Badia et al. , 2020 ; Flet-Berliac et al. , 2021 ) . Unlike EASEE , these methods necessitate the computation of non-stationary and vanishing novelty estimates , which require careful tuning to balance learning stability and exploration incentives . Moreover , because these bonuses are learned , and do not allow for the use of prior structure knowledge , they constitute an orthogonal approach to ours . In Gupta et al . ( 2018 ) exploration strategies are learned from prior experience . Unlike EASEE this requires meta-training over a distribution of tasks . Redundancies in Trajectories The idea that different trajectories can overlap and induce redundancies in state visitation is used in Leurent & Maillard ( 2020 ) and Czech et al . ( 2020 ) in the case of Monte-Carlo tree search . However , they require a generative model , and propose a new Bellman operator to update node values according to newly uncovered transitions rather than modifying exploration . Closer to our work , Caselles-Dupré et al . ( 2020 ) study structure in action sequences , but restrict themselves to commutative properties . Grinsztajn et al . ( 2021 ) quantifies the probability of cycling back to a previously visited state , motivated by the analysis of reversible actions . Tabu search ( Glover , 1986 ) is a meta-heuristic which uses knowledge of the past to escape local optima . It is popular for combinatorial optimization ( Hertz & Werra , 2005 ) . Like our approach , it relies on a local structure : actions which are known to cancel out recent moves are deemed tabu , and are forbidden for a short period of time . This prevents cycling around already found solutions , and thus encourages exploration . In Abramson & Wechsler ( 2003 ) , tabu search is combined with reinforcement learning , using action priors . However , their method can not make use of more complex action-sequence structure . Maximum State-Visitation Entropy Our goal to explore as uniformly as possible every nearby state can be seen as a local version of the Maximum State-Visitation Entropy problem ( MSVE ) ( de Farias & Van Roy , 2003 ; Hazan et al. , 2019 ; Lee et al. , 2019 ; Guo et al. , 2021 ) . MSVE formulates exploration as a policy optimization problem whose solution maximizes the entropy of the distribution of visited states . Although some of these works ( Hazan et al. , 2019 ; Lee et al. , 2019 ; Guo et al. , 2021 ) can make use of priors about state similarities , they learn a global policy and can not exploit structure in action sequences . Action Space Structure The idea of exploiting structure in action spaces is not new . Large discrete action spaces may be embedded in continuous action spaces either by leveraging prior information ( Dulac-Arnold et al. , 2016 ) or learning representations ( Chandak et al. , 2019 ) . Tavakoli et al . ( 2018 ) manage high-dimensional action spaces by assuming a degree of independence between each dimension . Farquhar et al . ( 2020 ) introduce a curriculum of progressively growing action spaces to accelerate learning . These methods aim to improve the generalization of policies to unseen actions in large action spaces rather than enhancing exploration . Leveraging previous trajectories to extract prior knowledge , Tennenholtz & Mannor ( 2019 ) provide an understanding of actions through their context in demonstrations . 3 FORMALISM . 3.1 EQUIVALENCE OVER ACTION SEQUENCES . We consider a Markov Decision Process ( MDP ) defined as a 5-tupleM = ( S , A , T , R , γ ) , with S the set of states , A the action set , T the transition function , R the reward function and the discount factor γ . The set of actions is assumed to be finite |A| < ∞ . We restrict ourselves to deterministic MDPs . A possible extension to MDPs with stochastic dynamics is discussed in Appendix A.6 . In the following , the notations are borrowed from formal language theory . Sequences of actions are analogous to strings over the set of symbols A ( possible actions ) . The set of all possible sequences of actions is denoted A ? = ⋃∞ k=0Ak where Ak is the set of all sequences of length k and A0 contains as single element the empty sequence Λ . We use . for the concatenation operator , such that for v1 ∈ Ah1 , v2 ∈ Ah2 , v1.v2 ∈ Ah1+h2 . The transition function T : S ×A → S gives the next state s′ when action a is taken in state s : T ( s , a ) = s′ . We recursively extend this operator to action sequences T : S ×A ? → S such that , ∀s ∈ S , ∀a ∈ A , ∀w ∈ A ? : T ( s , Λ ) = s T ( s , w.a ) = T ( T ( s , w ) , a ) Intuitively , this operator gives the new state of the MDP after a sequence of actions is performed from state s. Definition 1 ( Equivalent sequences ) . We say that two action sequences a1 . . . an and a′1 . . . a′m ∈ A ? are equivalent at state s ∈ S if T ( s , a1 . . . an ) = T ( s , a ′ 1 . . . a ′ m ) ( 1 ) Two sequences of actions are equivalent overM if they are equivalent at state s for all s in S. This is written : a1 . . . an ∼M a′1 . . . a′m ( 2 ) This means that we consider two sequences of actions to be equivalent when following one or the other will always lead to the same state . When the considered MDPM is unambiguous , we simplify the notation by writing ∼ instead of ∼M . We argue that some priors about the environments can be easily encoded as a small set of action sequence equivalences . For example , we may know that going left then right is the same thing as going right then left , that rotating two times to the left is the same thing as rotating two times to the right , or that opening a door twice is the same thing as opening the door once . All these priors can be encoded as a set of equivalences : Definition 2 ( Equivalence set ) . Given a MDP M and several equivalent sequence pairs v1 ∼ w1 , v2 ∼ w2 , . . . , vn ∼ wn , we say that Ω = { { v1 , w1 } , { v2 , w2 } , . . . { vn , wn } } is an equivalence set overM . Formally , Ω is a set of pairs of elements of A ? , such that Ω ⊂ ( A ? ) 2 . By abuse of notation , we write v ∼ w ∈ Ω if { v , w } ∈ Ω . Intuitively , it is clear that action sequence equivalences can be combined to form new , longer equivalences . For example , knowing that going left then right is the same thing as going right then left , we can deduce that going two times left then two times right is the same thing as going two times right then two times left . In the same fashion , if opening a door twice produces the same effect as opening it once , opening three times the door does the same . We formalize these notions in what follows . First , we note that equivalent sequences can be concatenated . Proposition 1 . If we have two pairs of equivalent sequences overM , i.e . w1 , w2 , w3 , w4 ∈ A ? such that w1 ∼ w2 w3 ∼ w4 then the concatenation of the sequences are also equivalent sequences : w1 · w3 ∼ w2 · w4 The proof is given in Appendix A.1 . We are now going to define formally the fact that the equivalence of two sequences can be deduced from an equivalence set Ω . We first consider the previous example where an action a has the effect of opening a door , such that a.a ∼ a . We can then write a.a.a ∼ ( a.a ) .a ∼ ( a ) .a ∼ a.a ∼ a by applying two times the equivalence a.a ∼ a and rearranging the parentheses . More generally and intuitively , the equivalence of two action sequences v and w can be deduced from Ω , which we denote v ∼Ω w , if v can be changed into w iteratively , chaining equivalences of Ω . More formally , we write v ∼1Ω w if v can be changed to w in one steps , meaning : ∃u1 , u2 , v1 , w1 ∈ A ? such that v = u1.v1.u2 w = u1.w1.u2 v1 ∼ w1 ∈ Ω ( 3 ) For n ≥ 2 , we say that v can be changed into w in n steps if there is a sequence v1 , . . . , vn ∈ A ? such that v ∼1Ω v1 ∼1Ω · · · ∼1Ω vn = w. Finally , we say that v ∼Ω w if there is n ∈ N such that v can be changed into w in n steps . The relation ∼Ω is thus a formal way of extending equivalences from a fixed equivalence set Ω , and at first glance not connected with ∼ , which deals with the equivalences of the MDP dynamics . We now show a connection between the two notions . Theorem 1 . Given an equivalence set Ω , ∼Ω is an equivalence relationship . Furthermore , for v , w ∈ A ? , v ∼Ω w ⇒ v ∼ w. The proof is given in Appendix A.2 . Given this relation between ∼ and ∼Ω , we will simplify the notation in what follows by writing ∼ instead of ∼Ω when the equivalence set considered is unambiguous . As ∼Ω is an equivalence relationship , it provides a partition over action sequences : two action sequences in the same set lead to the same final state from any given state .

Summary: In the context of reinforcement learning, authors propose an exploration strategy based on environment-specific prior knowledge of action equivalence. An example of such equivalence is rotating 180° twice in a grid world, as the agent comes back to the same original state: the action sequence forms an identity in this case. The proposed exploration, instead of picking a random action with probability (\epsilon)/(number of actions), builds an abstracted lookahead tree of specified depth that merges equivalent action sequences and adjusts the distribution of \epsilon over them. Authors demonstrate this i) increases the number of unique state visitations in grid-world examples, and ii) DQN with their exploration strategy achieves a higher reward faster than the baseline.