Competitive experience replay

1 INTRODUCTION . Recent progress in deep reinforcement learning has achieved very impressive results in domains ranging from playing games ( Mnih et al. , 2015 ; Silver et al. , 2016 ; 2017 ; OpenAI , 2018 ) , to high dimensional continuous control ( Schulman et al. , 2016 ; 2017 ; 2015 ) , and robotics ( Levine et al. , 2018 ; Kalashnikov et al. , 2018 ; Andrychowicz et al. , 2018b ) . Despite these successes , in robotics control and many other areas , deep reinforcement learning still suffers from the need to engineer a proper reward function to guide policy optimization ( see e.g . Popov et al . 2017 , Ng et al . 2006 ) . In robotic control as stacking bricks , reward function need to be sharped to very complex which consists of multiple terms ( Popov et al. , 2017 ) . It is extremely hard and not applicable to engineer such reward function for each task in real world since both reinforcement learning expertise and domain-specific knowledge are required . Learning to perform well in environments with sparse rewards remains a major challenge . Therefore , it is of great practical importance to develop algorithms which can learn from binary signal indicating successful task completion or other unshaped reward signal . In environments where dense reward function is not available , only a small fraction of the agents ’ experiences will be useful to compute gradient to optimize policy , leading to substantial high sample complexity . Providing agents with useful signals to pursue in sparse reward environments becomes crucial in these scenarios . ∗This work was done when the author was an intern at Salesforce Research . In the domain of goal-directed RL , the recently proposed hindsight experience replay ( HER ) ( Andrychowicz et al. , 2017 ) addresses the challenge of learning from sparse rewards by re-labelling visited states as goal states during training . However , this technique continues to suffer from sample inefficiency , ostensibly due to difficulties related to exploration . In this work , we address these limitations by introducing a method called Competitive Experience Replay ( CER ) . This technique attempts to emphasize exploration by introducing a competition between two agents attempting to learn the same task . Intuitively , agent A ( the agent ultimately used for evaluation ) receives a penalty for visiting states that the competitor agent ( B ) also visits ; and B is rewarded for visiting states found by A . Our approach maintains the reward from the original task such that exploration is biased towards the behaviors best suited to accomplishing the task goals . We show that this competition between agents can automatically generate a curriculum of exploration and shape otherwise sparse reward . We jointly train both agents ’ policies by adopting methods from multi-agent RL . In addition , we propose two versions of CER , independent CER , and interact CER , which differ in the state initialization of agent B : whether it is sampled from the initial state distribution or sampled from off-policy samples of agent A , respectively . Whereas HER re-labels samples based on an agent ’ s individual rollout , our method re-labels samples based on intra-agent behavior ; as such , the two methods do not interfere with each other algorithmically and are easily combined during training . We evaluate our method both with and without HER on a variety of reinforcement learning tasks , including navigating an ant agent to reach a goal position and manipulating objects with a robotic arm . For each such task the default reward is sparse , corresponding to a binary indicator of goal completion . Ablation studies show that our method is important for achieving a high success rate and often demonstrates faster convergence . Interestingly , we find that CER and HER are complementary methods and employ both to reach peak efficiency and performance . Furthermore , we observe that , when combined with HER , CER outperforms curiosity-driven exploration . 2 BACKGROUND . Here , we provide an introduction to the relevant concepts for reinforcement learning with sparse reward ( Section 2.1 ) , Deep Deterministic Policy Gradient , the backbone algorithm we build off of , ( Section 2.2 ) , and Hindsight Experience Replay ( Section 2.3 ) . 2.1 SPARSE REWARD REINFORCEMENT LEARNING . Reinforcement learning considers the problem of finding an optimal policy for an agent that interacts with an uncertain environment and collects reward per action . The goal of the agent is to maximize its cumulative reward . Formally , this problem can be viewed as a Markov decision process over the environment states s ∈ S and agent actions a ∈ A , with the ( unknown ) environment dynamics defined by the transition probability T ( s′|s , a ) and reward function r ( st , at ) , which yields a reward immediately following the action at performed in state st. We consider goal-conditioned reinforcement learning from sparse rewards . This constitutes a modification to the reward function such that it depends on a goal g ∈ G , such that rg : S × A×G → R. Every episode starts with sampling a state-goal pair from some distribution p ( s0 , g ) . Unlike the state , the goal stays fixed for the whole episode . At every time step , an action is chosen according to some policy π , which is expressed as a function of the state and the goal , π : S ×G → A . For generality , our only restriction on G is that it is a subset of S. In other words , the goal describes a target state and the task of the agent is to reach that state . Therefore , we apply the following sparse reward function : rt = rg ( st , at , g ) = { 0 , if |st − g| < δ −1 , otherwise ( 1 ) where g is a goal , |st − g| is a distance measure , and δ is a predefined threshold that controls when the goal is considered completed . Following policy gradient methods , we model the policy as a conditional probability distribution over states , πθ ( a| [ s , g ] ) , where [ s , g ] denotes concatenation of state s and goal g , and θ are the learnable parameters . Our objective is to optimize θ with respect to the expected cumulative reward , given by : J ( θ ) = Es∼ρπ , a∼π ( a|s , g ) , g∼G [ rg ( s , a , g ) ] , where ρπ ( s ) = ∑∞ t=1 γ t−1Pr ( st = s ) is the normalized discounted state visitation distribution with discount factor γ ∈ [ 0 , 1 ) . To simplify the notation , we denote Es∼ρπ , a∼π ( a|s , g ) , g∼G [ · ] by simply Eπ [ · ] in the rest of paper . According to the policy gradient theorem ( Sutton et al. , 1998 ) , the gradient of J ( θ ) can be written as ∇θJ ( θ ) = Eπ [ ∇θ log π ( a|s , g ) Qπ ( s , a , g ) ] , ( 2 ) where Qπ ( s , a , g ) = Eπ [ ∑∞ t=1 γ t−1rg ( st , at , g ) |s1 = s , a1 = a ] , called the critic , denotes the expected return under policy π after taking an action a in state s , with goal g . 2.2 DDPG ALGORITHM . Here , we introduce Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) , a model-free RL algorithm for continuous action spaces that serves as our backbone algorithm . Our proposed modifications need not be restricted to DDPG ; however , we leave experimentation with other continuous control algorithms to future work . In DDPG , we maintain a deterministic target policy µ ( s , g ) and a critic Q ( s , a , g ) , both implemented as deep neural networks ( note : we modify the standard notation to accommodate goal-conditioned tasks ) . To train these networks , episodes are generated by sampling actions from the policy plus some noise , a ∼ µ ( s , g ) +N ( 0 , 1 ) . The transition tuple associated with each action ( st , at , gt , rt , st+1 ) is stored in the so-called replay buffer . During training , transition tuples are sampled from the buffer to perform mini-batch gradient descent on the loss L which encourages the approximated Q-function to satisfy the Bellman equation L = E ( Q ( st , at , gt ) −yt ) 2 , where yt = rt+γQ ( st+1 , µ ( st+1 , gt ) , gt ) . Similarly , the actor can be updated by training with mini-batch gradient descent on the loss J ( θ ) = −EsQ ( s , µ ( s , g ) , g ) through the deterministic policy gradient algorithm ( Silver et al. , 2014 ) , ∇θJ ( θ ) = E [ ∇θµ ( s , g ) ∇aQ ( s , a , g ) |a=µ ( s , g ) ] . ( 3 ) To make training more stable , the targets yt are typically computed using a separate target network , whose weights are periodically updated to the current weights of the main network ( Lillicrap et al. , 2015 ; Mnih et al. , 2013 ; 2015 ) . 2.3 HINDSIGHT EXPERIENCE REPLAY . Despite numerous advances in the application of deep learning to RL challenges , learning in the presence of sparse rewards remains a major challenge . Intuitively , these algorithms depend on sufficient variability within the encountered rewards and , in many cases , random exploration is unlikely to uncover this variability if goals are difficult to reach . Recently , Andrychowicz et al . ( 2017 ) proposed Hindsight Experience Replay ( HER ) as a technique to address this challenge . The key insight of HER is that failed rollouts ( where no task reward was obtained ) can be treated as successful by assuming that a visited state was the actual goal . Basically , HER amounts to a relabelling strategy . For every episode the agent experiences , it gets stored in the replay buffer twice : once with the original goal pursued in the episode and once with the goal replaced with a future state achieved in the episode , as if the agent were instructed to reach this state from the beginning . Formally , HER randomly samples a mini-batch of episodes in buffer , for each episode ( { si } Ti=1 , { gi } Ti=1 , { ai } Ti=1 , { ri } Ti=1 , { s′i } Ti=1 ) , for each state st , where 1 ≤ t ≤ T − 1 in an episode , we randomly choose sk where t+ 1 ≤ k ≤ T and relabel transition ( st , at , gt , rt , st+1 ) to ( st , at , sk , r′t , st+1 ) and recalculate reward r′t , r′t = rg ( st , sk ) = { 0 , if |st − sk| < δ , −1 , otherwise . ( 4 ) 3 METHOD . In this section , we present Competitive Experience Replay ( CER ) for policy gradient methods ( Section 3.1 ) and describe the application of multi-agent DDPG to enable this technique ( Section 3.2 ) .

Summary: The authors propose a new method for learning from sparse rewards in model-free reinforcement learning settings. This is a challenging and important problem in model-free RL, mainly due to the lack of effective exploration. They propose a new way of densifying the reward by encouraging a pair of agents to explore different states (using competitive self-play) while trying to learn the same task. One of the agents (A) receives a penalty for visiting states that the other agent (B) also visits, while B is rewarded for visiting states found by A. They evaluate their method on a few tasks with continuous action spaces such as ant navigation in a maze and object manipulation by a simulated robotic arm.  Their method shows faster convergence (in some cases) and better performance than comparable algorithms.