Backplay: 'Man muss immer umkehren'

1 INTRODUCTION . An important goal of AI research is to construct agents that can learn well in new environments ( Levine et al. , 2016 ) . An increasingly popular paradigm for this task is deep reinforcement learning ( deep RL , Silver et al . ( 2016 ) ; Moravcík et al . ( 2017 ) ; Silver et al . ( 2017 ) ) . However , training an RL agent can take a very long time , particularly in environments with sparse rewards . In these settings , the agent typically requires a large number of episodes to stumble upon positive rewards and learn even a moderately effective policy that can then be refined . This is often resolved via hand-engineering a dense reward function . Such reward shaping , while effective , can also change the set of optimal policies and have unintended side effects ( Ng et al. , 1999 ; Clark & Amodei , 2016 ) . We consider an alternative technique for accelerating RL in sparse reward settings . The idea is to create a curriculum for the agent via reversing a single trajectory ( i.e . state sequence ) of reasonably good , but not necessarily optimal , behavior . We start our agent at the end of a demonstration and let it learn a policy in this easier setup . We then move the starting point backward until the agent is training only on the initial state of the task . We call this technique Backplay . Our contributions are threefold : 1 . We characterize analytically and qualitatively which environments Backplay will aid . 2 . We demonstrate Backplay ’ s effectiveness on both a grid world task ( to gain intuition ) as well as the four player stochastic zero-sum game Pommerman ( MultiAgentLearning , 2018 ) . 3 . We empirically show that Backplay compares favorably to other methods that improve sample complexity . Besides requiring vastly fewer number of samples to learn a good policy , an agent trained with Backplay can outperform its demonstrator and even learn an optimal policy following a sub-optimal ∗These two authors contributed equally demonstration . Our experiments further show Backplay ’ s strong performance relative to reward shaping ( involves hand tuning reward functions ) , behavioral cloning ( not intended for use with sub-optimal experts ) , and other forms of automatic curriculum generation ( Florensa et al . ( 2017 ) , requires a reversable environment ) . 2 RELATED WORK The most related work to ours is a blog post describing a method similar to Backplay used to obtain state-of-theart performance on the challenging Atari game Montezuma ’ s Revenge ( Salimans & Chen , 2018 ) . This work was independent of and concurrent to our own . In addition to reporting results on a different , complex stochastic multi-agent environment , we provide an analytic characterization of the method as well as an in depth discussion of what kinds of environments a practitioner can expect Backplay to out or underperform other existing methods . A popular method for improving RL with access to expert demonstrations is behavioral cloning/imitation learning . These methods explicitly encourage the learned policy to mimic an expert policy ( Bain & Sommut , 1999 ; Ross et al. , 2011 ; Daumé et al. , 2009 ; Zhang & Cho , 2016 ; Laskey et al. , 2016 ; Nair et al. , 2017 ; Hester et al. , 2017 ; Ho & Ermon , 2016 ; Aytar et al. , 2018 ; Lerer & Peysakhovich , 2018 ; Peng et al. , 2018 ) . Imitation learning requires access to both state and expert actions ( whereas Backplay only requires states ) and is designed to copy an expert , thus it can not , without further adjustments ( e.g . as proposed by Gao et al . ( 2018 ) ) , surpass a suboptimal expert . We discuss the pros and cons of an imitation learning + adjustment vs. a Backplay-based approach in the main analysis section . Other algorithms ( Ranzato et al. , 2015 ; Li et al. , 2016 ; Das et al. , 2017a ; b ) , primarily in dialog , use a Backplay-like curriculum , albeit they utilize behavioral cloning for the first part of the trajectory . This is a major difference as we show that for many classes of problems , we only need to change the initial state distribution and do not see any gains from warm-starting with imitation learning . Backplay is more similar to Conservative Policy Iteration ( Kakade & Langford , 2002 ) , a theoretical paper which presents an algorithm designed to operate with an explicit restart distribution . Also related to Backplay is the method of automatic reverse curriculum generation Florensa et al . ( 2017 ) . These approaches assumes that the final goal state is known and that the environment is both resettable and reversible . The curricula are generated by taking random walks in the state space to generate starting states or by taking random actions starting at the goal state McAleer et al . ( 2018 ) . These methods do not require an explicit ‘ good enough ’ demonstration as Backplay does . However , they require the environment to be reversible , an assumption that doesn ’ t hold in many realistic tasks such as a robot manipulating breakable objects or complex video games such as Starcraft . In addition , they may fare poorly when random walks reach parts of the state space that are not actually relevant for learning a good policy . Thus , whether a practitioner wants to generate curricula from a trajectory or a random walk depends on the environment ’ s properties . We discuss this in more detail in our analysis section and show empirical results suggesting that Backplay is superior . Hosu & Rebedea ( 2016 ) use uniformly random states of an expert demonstration as starting states for a policy . Like Backplay , they show that using a single loss function to learn a policy from both demonstrations and rewards can outperform the demonstrator and is robust to sub-optimal demonstrations . However , they do not impose a curriculum over the demonstration and are equivalent to the Uniform baseline in our experiments . Zhu et al . ( 2018 ) use a curriculum but manually tune it for each ‘ stage ’ of the environment . Within each stage , they use what we call Uniform training , which fails in our most challenging environments . Goyal et al . ( 2018 ) and Edwards et al . ( 2018 ) simultaneously proposed the use of a learned backtracking model to generate traces that lead to high value states . Their methods rely on either having the agent visit high reward states or learning a model of the environment capable of generating the states . Both of these are challenging in environments in which the dynamics near starting states are very different from those near goal states . Finally , Ivanovic et al . ( 2018 ) use a known ( approximate ) dynamics model to create a backwards curriculum for continuous control tasks . Their approach requires a physical prior which is not always available and often not applicable in multi-agent scenarios . In contrast , Backplay automatically creates a curriculum fit for any resettable environment with accompanying demonstrations . 3 BACKPLAY . Consider the standard formalism of a single agent Markov Decision Process ( MDP ) defined by a set of states S , a set of actions A , and a transition function T : S × A → S which gives the probability distribution of the next state given a current state and action . If P ( A ) denotes the space of probability distributions over actions , the agent chooses actions by sampling from a stochastic policy π : S → P ( A ) , and receives reward r : S × A → R at every time step . The agent ’ s goal is to construct a policy which minimizes its discounted expected return Rt = E [ ∑∞ k=0 γ krt+k+1 ] where rt is the reward at time t and γ ∈ [ 0 , 1 ] is the discount factor , and the expectation is taken with respect to both the policy and the environment . The final component of an MDP is the distribution of initial starting states s0 . The key idea in Backplay is that we do not initialize the MDP in only a fixed s0 . Instead , we assume access to a demonstration which reaches a sequence of states { sd0 , sd1 , . . . , sdT } . For each training episode , we uniformly sample starting states from the sub-sequence { sdT−k , sdT−k+1 , . . . , sdT−j } for some window [ j , k ] . Note that this training regime requires the ability to reset the environment to any state . As training continues , we ‘ advance ’ the window according to a curriculum by increasing the values of j and k until we are training on the initial state in every episode ( j = k = T ) . In this manner , our hyperparameters for Backplay are the windows and the training epochs at which we advance them . 3.1 QUANTITATIVE ANALYSIS . Next , we consider a simple environment in which we can analytically show that Backplay will improve the sample-efficiency of RL training . Consider a connected , undirected graph G = ( V , E ) . An agent is placed at a fixed node v0 ∈ V and moves to neighboring nodes at a negative reward of −1 for each step . Its goal is to reach a target node v∗ ∈ V , terminating the episode . This corresponds to an MDPM = ( S , A , P , R ) with state space S ∼ V , action space A ∼ E , deterministic transition kernel corresponding to P ( st+1 = j | st = i , at = ( l , k ) ) = δ ( j = k ) δ ( l = i ) + δ ( j = i ) δ ( l 6= i ) and uniform reward R ( st , at ) = −1 for st+1 6= v∗ . Assume that π is a fixed policy onM , such that the underlying Markov chain is irreducible and aperiodic : Kπ ( s ′ , s ) = ∑ a∈A P ( st+1 = s ′ | st = s , at = a ) π ( a | s ) Kπ has a single absorbing state at s = s∗ : = v∗ . Our goal is to estimate the value function of this policy , equivalent to the expected first-passage time of the Markov chain Kπ from s0 = s to s∗ : Vπ ( s ) = Eτ ( s , s∗ ) = Emin { j ≥ 0 ; s.t . sj = s∗ , s0 = s , si+1 ∼ Kπ ( · , si ) } . We consider a special tractable case where the value function can be well approximated by looking at the distance of a state from the goal state . Formally : Assumption 1 . For all θ , Vθ ( s ) = Vθ ( s′ ) whenever distG ( s′ , s∗ ) = distG ( s , s∗ ) . We wish to analyze the process which is the projection of our Markov policy π only in terms of the distance zt . However , now the transition probabilities will not only be a function of only zt and so this projection will be non-Markovian . Its Markov approximation z̄t is defined as the Markov chain K given by the expected transition probabilities Pr ( z̄t+1 = l − 1|z̄t = l ) = αl : = Prµ ( zt+1 = l − 1 | zt = l ) , ( 1 ) Pr ( z̄t+1 = l|z̄t = l ) = βl : = Prµ ( zt+1 = l | zt = l ) , l = 0 . . .M ( 2 ) and thus Pr ( z̄t+1 = l + 1|z̄t = l ) = 1− αl − βl = Prµ ( zt+1 = l + 1 | zt = l ) under the stationary distribution µ of Kπ . Assumption 2 . The projected process is well described by its Markovian approximation . Though these assumptions are relatively strong , they makes the analysis of Backplay in this graph complex but analytically tractable . Given a demonstration d = ( d0 = s0 , . . . , dL = s∗ ) , dl ∈ S we will perform Backplay Theorem 1 . When assumptions 1 and 2 hold , the sample complexity gains from using Backplay rather than standard RL are exponential in the diameter of the graph . O ( M 2 m α −m ) vs Ω ( Mα−M/2 ) . Proof . we sample it using a step size m > 0 ( such that j = 0 mod m , where j is defined in Section 3 ) to obtain d̄l : = distG ( dL−ml , s∗ ) , l = 0 , 1 , . . . , Lm , which satisfies d̄l ≤ lm for all l. For fixed l , we initialize the chain K at d̄l : z̄0 = d̄l . Since Pr ( z̄m = 0 ) ≥ ∏m−1 j=0 αj : = γ0 , m , after O ( γ−10 , m ) trials of length ≤M , we will reach the absorbing state and finally have a signal-carrying update for the Q-function at the originating state . We can consequently merge that state into the absorbing state and reduce the length of the chain by one . Repeat the argument m times so that after O ( ∑m j=0 γ −1 j , m ) = O ( mγ −1 0 , m ) trials , the Q-function is updated at z̄0 . Repeat at Backplay steps lm , l = 1 , . . . Mm , and we reach a sample complexity of Tm = M m−1∑ k=0 O M m∑ j=0 γ−1km , ( k+1 ) m−j . In the case where αl = α for all l , we obtain γ−1km , ( k+1 ) m−j = γ0 , m−j = α −m+j and therefore Tm = O ( M2 ( 1−αm+1 ) m ( 1−α ) α −m ) , where the important term is the rate M 2 m α −m . On the other hand , Hong & Zhou ( 2017 ) shows that the first-passage time τ ( 0 , M ) in a skip-free finite Markov chain of M states with a single absorbing state is a random variable whose momentgenerating function ϕ ( s ) = Esτ ( s , s∗ ) is given by ϕ ( s ) = M∏ j=1 ( 1− λj ) s 1− λjs , ( 3 ) where λ1 , . . . , λM are the non-unit eigenvalues of the transition probability matrix . It follows that Eτ ( 0 , M ) = ϕ′ ( 1 ) = ∑M j=1 1 1−λj ≈ ( 1− λ1 ) −1 , which corresponds to the reciprocal spectral gap.1 Chen & Saloff-Coste ( 2013 ) further shows that this reciprocal spectral gap is Ω ( α−M/2 ) in our case , and therefore the model without Backplay will on average take TM = Ω ( α−M/2 ) trials to reach the absorbing state and receive information . We can analyze the uniform strategy similarly . The probability that a trajectory initialized at one of the uniform samples will reach the absorbing state is lower bounded by M∑ j=1 αjP ( z̄0 = j ) = 1 M M∑ j=1 αj = α− αM+1 M ( 1− α ) , ( 4 ) which is approximately αM when α is small , leading to a sample complexity of O ( M 2α−1 ) to update the value function at the originating state , and O ( M3α−1 ) at the starting state . Comparing this rate 1The spectral gap of a Markov Chain is the difference between the first and second largest eigenvalue magnitudes of its transition probability matrix . It controls its mixing time among many other properties . to Backplay with m = 1 , observe that the uniform strategy is slower by a factor of M ( and one can verify that the same is true for generic step size m by imagining that we first sampled a window of size m and then sub-sampled our state from that window ) , suggesting that it loses efficiency on environments with large diameter . The preceding analysis suggests that a full characterization of Backplay is a fruitful direction for reinforcement and imitation learning theory , albeit beyond the scope of this paper .

Summary: The paper presents a strategy for solving sparse reward tasks with RL by sampling initial states from demonstrations. The method, Backplay, starts by sampling states near the end of a demonstration trajectory, so that the agent will be initialized to states near the goal. As training progresses, the initial state distribution is incrementally shifted towards earlier steps in the demonstration, until the agent is trained starting from the original initial state. The authors further provide an analysis of the sample complexity of this method on a simple MDP. The method is demonstrated on a maze navigation task and a challenging game Pommerman.