Better state exploration using action sequence equivalence

1 INTRODUCTION . Despite the rapidly improving performance of Reinforcement Learning ( RL ) agents on a variety of tasks ( Mnih et al. , 2015 ; Silver et al. , 2016 ) , they remain largely sample-inefficient learners compared to humans ( Toromanoff et al. , 2019 ) . Contributing to this is the vast amount of prior knowledge humans bring to the table before their first interaction with a new task , including an understanding of physics , semantics , and affordances ( Dubey et al. , 2018 ) . The considerable quantity of data necessary to train agents is becoming more problematic as RL is applied to ever more challenging and complex tasks . Much research aims at tackling this issue , for example through transfer learning ( Rusu et al. , 2016 ) , meta learning , and hierarchical learning , where agents are encouraged to use what they learn in one environment to solve a new task more quickly . Other approaches attempt to use the structure of Markov Decision Processes ( MDP ) to accelerate learning without resorting to pretraining . Mahajan & Tulabandhula ( 2017 ) and Biza & Jr. ( 2019 ) learn simpler representations of MDPs that exhibit symmetrical structure , while van der Pol et al . ( 2020 ) show that environment invariances can be hard-coded into equivariant neural networks . A fundamental challenge standing in the way of improved sample efficiency is exploration . We consider a situation where the exact transition function of a Markov Decision Process is unknown , but some knowledge of its local dynamics is available under the form of a prior expectation that given sequences of actions have identical results . This way of encoding prior knowledge is sufficiently flexible to describe many useful environment structures , particularly when actions correspond to agent movement . For example , in a gridworld ( called RotationGrid hereafter ) where the agent can move forward ( ↑ ) and rotate 90◦to the left ( x ) or to the right ( y ) , the latter two actions are the inverse of each other , in that performing one undoes the effect of the other . During exploration , to encourage the visitation of not yet seen states , it is natural to simply ban sequences of actions that revert to previously visited states , following the reasoning of Tabu search ( Glover , 1986 ) . We observe further that yy and xx both lead to the same state ( represented as state 4 in Figure 1 ) . If actions were uniformly sampled , the chances of visiting this state would be much higher than any of the others . Based on these observations , we introduce a new method taking advantage of Equivalent Action SEquences for Exploration ( EASEE ) , an overview of which can be found in Figure 1 . EASEE looks ahead several steps and calculates action sampling probabilities to explore as uniformly as possible new states conditionally on the action sequence equivalences given to it . It constructs a partial MDP which corresponds to a local representation of the true MDP around the current state . We then formulate the problem of determining the best distribution over action sequences as a linearly constrained convex optimization problem . Solving this optimization problem is computationally inexpensive and can be done once and for all before learning begins , providing a principled and tractable exploration policy that takes into account environment structure . This policy can easily be injected into existing reinforcement learning algorithms as a substitute for -greedy exploration . Our contribution is threefold . First , we formally introduce the notion of equivalent action sequences , a novel type of structure in Markov Decision Processes . Then , we show that priors on this type of structure can easily be exploited during offline exploration by solving a convex optimization problem . Finally , we provide experimental insights and show that incorporating EASEE into a DQN ( Mnih et al. , 2015 ) improves agent performance in several environments with various structures . Overview We assume that we have sets of equivalent action-sequences for the environment . Equivalent action sequences are sequences that lead to the same state . These sequences are used to build a DAG that models where the agent will end up after any sequence of actions of length d. Because some sequences are equivalent , several parent nodes may share a child node . A naive exploration scheme like -greedy would waste resources by over exploring such child nodes . Instead , we leverage this information using the DAG constructed above ; our method executes an exploratory action that maximizes the entropy of the future visited states . 2 RELATED WORK . Improved Exploration The problem of ensuring that agents see sufficiently diverse states has received a lot of attention from the RL community . Many methods rely on intrinsic rewards ( Schmidhuber , 1991 ; Chentanez et al. , 2005 ; Şimşek & Barto , 2006 ; Lopes et al. , 2012 ; Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ; Pathak et al. , 2017 ) to entice agents to unseen or misunderstood areas . In the tabular setting , these take the form of count-based exploration bonuses which guide the agent toward poorly visited states ( e.g . Strehl & Littman ( 2008 ) ) . Scaling this method requires the use of function approximators ( Burda et al. , 2019 ; Badia et al. , 2020 ; Flet-Berliac et al. , 2021 ) . Unlike EASEE , these methods necessitate the computation of non-stationary and vanishing novelty estimates , which require careful tuning to balance learning stability and exploration incentives . Moreover , because these bonuses are learned , and do not allow for the use of prior structure knowledge , they constitute an orthogonal approach to ours . In Gupta et al . ( 2018 ) exploration strategies are learned from prior experience . Unlike EASEE this requires meta-training over a distribution of tasks . Redundancies in Trajectories The idea that different trajectories can overlap and induce redundancies in state visitation is used in Leurent & Maillard ( 2020 ) and Czech et al . ( 2020 ) in the case of Monte-Carlo tree search . However , they require a generative model , and propose a new Bellman operator to update node values according to newly uncovered transitions rather than modifying exploration . Closer to our work , Caselles-Dupré et al . ( 2020 ) study structure in action sequences , but restrict themselves to commutative properties . Grinsztajn et al . ( 2021 ) quantifies the probability of cycling back to a previously visited state , motivated by the analysis of reversible actions . Tabu search ( Glover , 1986 ) is a meta-heuristic which uses knowledge of the past to escape local optima . It is popular for combinatorial optimization ( Hertz & Werra , 2005 ) . Like our approach , it relies on a local structure : actions which are known to cancel out recent moves are deemed tabu , and are forbidden for a short period of time . This prevents cycling around already found solutions , and thus encourages exploration . In Abramson & Wechsler ( 2003 ) , tabu search is combined with reinforcement learning , using action priors . However , their method can not make use of more complex action-sequence structure . Maximum State-Visitation Entropy Our goal to explore as uniformly as possible every nearby state can be seen as a local version of the Maximum State-Visitation Entropy problem ( MSVE ) ( de Farias & Van Roy , 2003 ; Hazan et al. , 2019 ; Lee et al. , 2019 ; Guo et al. , 2021 ) . MSVE formulates exploration as a policy optimization problem whose solution maximizes the entropy of the distribution of visited states . Although some of these works ( Hazan et al. , 2019 ; Lee et al. , 2019 ; Guo et al. , 2021 ) can make use of priors about state similarities , they learn a global policy and can not exploit structure in action sequences . Action Space Structure The idea of exploiting structure in action spaces is not new . Large discrete action spaces may be embedded in continuous action spaces either by leveraging prior information ( Dulac-Arnold et al. , 2016 ) or learning representations ( Chandak et al. , 2019 ) . Tavakoli et al . ( 2018 ) manage high-dimensional action spaces by assuming a degree of independence between each dimension . Farquhar et al . ( 2020 ) introduce a curriculum of progressively growing action spaces to accelerate learning . These methods aim to improve the generalization of policies to unseen actions in large action spaces rather than enhancing exploration . Leveraging previous trajectories to extract prior knowledge , Tennenholtz & Mannor ( 2019 ) provide an understanding of actions through their context in demonstrations . 3 FORMALISM . 3.1 EQUIVALENCE OVER ACTION SEQUENCES . We consider a Markov Decision Process ( MDP ) defined as a 5-tupleM = ( S , A , T , R , γ ) , with S the set of states , A the action set , T the transition function , R the reward function and the discount factor γ . The set of actions is assumed to be finite |A| < ∞ . We restrict ourselves to deterministic MDPs . A possible extension to MDPs with stochastic dynamics is discussed in Appendix A.6 . In the following , the notations are borrowed from formal language theory . Sequences of actions are analogous to strings over the set of symbols A ( possible actions ) . The set of all possible sequences of actions is denoted A ? = ⋃∞ k=0Ak where Ak is the set of all sequences of length k and A0 contains as single element the empty sequence Λ . We use . for the concatenation operator , such that for v1 ∈ Ah1 , v2 ∈ Ah2 , v1.v2 ∈ Ah1+h2 . The transition function T : S ×A → S gives the next state s′ when action a is taken in state s : T ( s , a ) = s′ . We recursively extend this operator to action sequences T : S ×A ? → S such that , ∀s ∈ S , ∀a ∈ A , ∀w ∈ A ? : T ( s , Λ ) = s T ( s , w.a ) = T ( T ( s , w ) , a ) Intuitively , this operator gives the new state of the MDP after a sequence of actions is performed from state s. Definition 1 ( Equivalent sequences ) . We say that two action sequences a1 . . . an and a′1 . . . a′m ∈ A ? are equivalent at state s ∈ S if T ( s , a1 . . . an ) = T ( s , a ′ 1 . . . a ′ m ) ( 1 ) Two sequences of actions are equivalent overM if they are equivalent at state s for all s in S. This is written : a1 . . . an ∼M a′1 . . . a′m ( 2 ) This means that we consider two sequences of actions to be equivalent when following one or the other will always lead to the same state . When the considered MDPM is unambiguous , we simplify the notation by writing ∼ instead of ∼M . We argue that some priors about the environments can be easily encoded as a small set of action sequence equivalences . For example , we may know that going left then right is the same thing as going right then left , that rotating two times to the left is the same thing as rotating two times to the right , or that opening a door twice is the same thing as opening the door once . All these priors can be encoded as a set of equivalences : Definition 2 ( Equivalence set ) . Given a MDP M and several equivalent sequence pairs v1 ∼ w1 , v2 ∼ w2 , . . . , vn ∼ wn , we say that Ω = { { v1 , w1 } , { v2 , w2 } , . . . { vn , wn } } is an equivalence set overM . Formally , Ω is a set of pairs of elements of A ? , such that Ω ⊂ ( A ? ) 2 . By abuse of notation , we write v ∼ w ∈ Ω if { v , w } ∈ Ω . Intuitively , it is clear that action sequence equivalences can be combined to form new , longer equivalences . For example , knowing that going left then right is the same thing as going right then left , we can deduce that going two times left then two times right is the same thing as going two times right then two times left . In the same fashion , if opening a door twice produces the same effect as opening it once , opening three times the door does the same . We formalize these notions in what follows . First , we note that equivalent sequences can be concatenated . Proposition 1 . If we have two pairs of equivalent sequences overM , i.e . w1 , w2 , w3 , w4 ∈ A ? such that w1 ∼ w2 w3 ∼ w4 then the concatenation of the sequences are also equivalent sequences : w1 · w3 ∼ w2 · w4 The proof is given in Appendix A.1 . We are now going to define formally the fact that the equivalence of two sequences can be deduced from an equivalence set Ω . We first consider the previous example where an action a has the effect of opening a door , such that a.a ∼ a . We can then write a.a.a ∼ ( a.a ) .a ∼ ( a ) .a ∼ a.a ∼ a by applying two times the equivalence a.a ∼ a and rearranging the parentheses . More generally and intuitively , the equivalence of two action sequences v and w can be deduced from Ω , which we denote v ∼Ω w , if v can be changed into w iteratively , chaining equivalences of Ω . More formally , we write v ∼1Ω w if v can be changed to w in one steps , meaning : ∃u1 , u2 , v1 , w1 ∈ A ? such that v = u1.v1.u2 w = u1.w1.u2 v1 ∼ w1 ∈ Ω ( 3 ) For n ≥ 2 , we say that v can be changed into w in n steps if there is a sequence v1 , . . . , vn ∈ A ? such that v ∼1Ω v1 ∼1Ω · · · ∼1Ω vn = w. Finally , we say that v ∼Ω w if there is n ∈ N such that v can be changed into w in n steps . The relation ∼Ω is thus a formal way of extending equivalences from a fixed equivalence set Ω , and at first glance not connected with ∼ , which deals with the equivalences of the MDP dynamics . We now show a connection between the two notions . Theorem 1 . Given an equivalence set Ω , ∼Ω is an equivalence relationship . Furthermore , for v , w ∈ A ? , v ∼Ω w ⇒ v ∼ w. The proof is given in Appendix A.2 . Given this relation between ∼ and ∼Ω , we will simplify the notation in what follows by writing ∼ instead of ∼Ω when the equivalence set considered is unambiguous . As ∼Ω is an equivalence relationship , it provides a partition over action sequences : two action sequences in the same set lead to the same final state from any given state .

Summary: The paper proposes a method that, from a simple encoding of sequences of actions that have equivalent outcome in an MDP, allows to compute a local policy for local high-quality exploration (it replaces the random action of $\varepsilon$-greedy with an action that maximizes the entropy of future visited states). Full algorithmic details about how to do that are given in the paper, and experiments on a few environments show that the method is promising. The Freeway experiment is particularly interesting, as it shows that a (small) benefit can be obtained from the method even on an Atari game, with minimal domain knowledge (only a few equivalent action sequences are encoded), and using a slightly modified DQN algorithm.