Hindsight Curriculum Generation Based Multi-Goal Experience Replay

1 INTRODUCTION . Multi-goal tasks with sparse rewards present a big challenge for training a reliable RL agent . In multi-goal tasks ( Plappert et al. , 2018 ) , an agent learns to achieve multiple different goals and receives no positive feedback until it reaches the position defined by the desired goal . Such a sparse rewards problem makes it difficult to reuse past experiences for that the positive feedback to reinforce policy is rare . It ’ s impractical to carefully engineer a shaped reward function ( Ng et al. , 1999 ; Popov et al. , 2017 ) that aligns with each task or assign a set of general auxiliary tasks ( Riedmiller et al. , 2018 ) , which relies on expert knowledge . To enrich the positive feedback , Andrychowicz et al . ( 2017 ) provides HER , a novel multi-goal experience replay strategy , which enables an agent to learn from unshaped reward , e.g . a binary signal indicating successful task solving . Specifically , HER replaces the desired goals with the achieved ones then recalculates the rewards of sampled experiences . By relabeling experiences with pseudo goals , it expands experiences without further exploration and is likely to turn failed experiences into successful ones with positive feedback . Experience relabeling makes better use of failed experiences . However , not all the achieved goals lead the origin experience to a reliable state-action visitation under the current policy . ( For simplicity , we denote state-goal pairs as augmented states . ) A Value function of a policy for a specific state-action pair can generalize to similar ones , in return the estimate of the value function may get worse without sufficient visitation near the state-action pair . For HER , it almost replays uniform samples of past experiences whilst goals for exploring are finite . When performing a value update , the current policy could not give a credible estimate of the universal value function ( Schaul et al. , 2015 ) , if the state-action pair is not well-explored . In other words , the current policy may have difficulty in generalizing to the state-action pair . Recent works focus on improving HER by evaluating the past experiences where we sample pseudo goals from . HER with Energy-Based Prioritization ( EBP ) ( Zhao & Tresp , 2018 ) defines a trajectory energy function as the sum of the transition energy of the target object over the trajectory . Curriculum-guided HER ( CHER ) ( Fang et al. , 2019b ) adaptively selects the failed experiences according to the proximity to the true goals and the curiosity of exploration over diverse pseudo goals with gradually changed proportion . Though these variants select valuable goals for replay , it remains a challenge that the agent will not further explore most of the pseudo goals then it is risky to directly generalize over pseudo goals . Humans show great capability in abstracting and generalizing knowledge but it takes a large number of experiences to learn to represent similar states with similar features . Fortunately , states with different achieved goals and desired goals may be similar intrinsically if the states and distance between them are similar . During an episode , the distance varies widely for a fixed desired goal , which has the potential for generalization . Therefore , we take advantage of relative goals-distances between the achieved goals and the desired goals-to transform data . The relative goals strategy alleviates the challenge of generalization without sufficient data . By explicitly discovering similar states in the replay buffer , it enables us to see the density of state-action visitations for unexplored goals more conveniently . The density reflects the likelihood of the corresponding state-action pair , indicating whether it is well explored . Besides , the generalization over relative goals is feasible only if the explored relative goals are widely distributed ( Schaul et al. , 2015 ) . In a word , it is significant to ensure sufficient state-action visitations and maintain a balanced distribution over valid goals . In this paper , we present to resample hindsight experiences with a relative goals strategy . The main criterion to sample experiences for a reliable generalization is based on 1 ) the likelihood of the corresponding state-action pair under the current policy ; 2 ) the overall distribution of the relative goals . By constantly adjusting the distribution of goals , we propose Hindsight Curriculum Generation ( HCG ) , which generates a replay curriculum to progressively expand the range of experiences for training . The main advantage is that it makes efficient use of hindsight experiences as well as tries to ensure the generalization over state-action pairs . From the perspective of curriculum learning , the generated curriculum can be seen as a sequence of weights on the training experiences , which guide the learning by automatically generating suitable replay goals . Furthermore , we implemented HCG with the vanilla Deep Deterministic Policy Gradient ( DDPG ) ( P. et al. , 2015 ) on various Robotic Control problems . The robot , a 7-DOF Fetch Robotics arm which has a two-fingered parallel gripper or an anthropomorphic robotic hand with 24 degrees of freedom , performs the training procedure using the MuJoCo simulated physics engine ( Todorov et al. , 2012 ) . During the training procedure , our method extracts and leverages information from hindsight experiences with various state-goal pairs . We experimentally demonstrated that our method improves the sample efficiency of the vanilla HER in solving multi-goal tasks with sparse rewards . Ablation studies show that our method is robust on the major hyperparameters . 2 BACKGROUND . In this section we briefly introduce the multi-goal RL framework , universal value function approximators and hindsight experience replay strategy used in the paper . 2.1 MULTI-GOAL RL . Consider an infinite-horizon discounted Markov decision process ( MDP ) , defined by the tuple ( S , A , G , P , r , γ ) , where S is a set of states , A is a set of actions , G is a set of goals , P : S×A×S → R is the transition probability distribution , r : S ×A×G → R is the reward function , and γ ∈ ( 0 , 1 ) is the discount factor . In multi-goal RL , an agent interacts with its discounted MDP environment in a sequence of episodes . At the beginning of each episode , the agent receives a goal state g ∈ G. In this paper we set that each g ∈ G corresponds to a goal state sg ∈ S . Moreover , we assume that given a state s we can easily find a goal g which is satisfied in this state . At each timestep t , the agent observes a state st ∈ S , chooses and executes an action at ∈ A . And the agent will receive a resulting reward r ( st , at , g ) at the next timestep t + 1 . ( For simplicity , we denote rt = r ( st , at , g ) . ) In multi-goal RL , the reward function r is a binary sparse signal indicating whether the agent achieves the desired goal state : rt = { 1 , ||ϕ ( st+1 ) − g||2 ≤ δg 0 , otherwise where ϕ : S → G , a known and tractable mapping , defines the corresponding goal representation of each state , and δg is a task-specific tolerance threshold defined in Plappert et al . ( 2018 ) . 2.2 UNIVERSAL VALUE FUNCTION APPROXIMATORS . Problems following the multi-goal RL framework ( Plappert et al. , 2018 ) tell RL agent what to do using an additional goal as input . The goal is fixed in each episode whilst there is more than one goal to achieve . In the continuous control problems , the agent could not afford to learn a policy for each goal . Instead , the policy should generalize not just over states but also over goals via deep neural networks . Formally , Universal Value Function Approximators ( UVFA ) ( Schaul et al. , 2015 ) factor observed values into separate embedding vectors for states and goals , then learn a mapping from ( s , g ) pairs to factored embedding vectors . Let τ = s1 , a1 , s2 , a2 , . . . , sT−1 , aT−1 , sT denote a trajectory , which is also an episode , Rt = ∑T i=t γ i−tri denote its discounted return at every timestep t ∈ [ 1 , T ] . Let π : S × G → A denote a universal policy , V π : S × G → R denote its value function . The objective of the agent is to learn a general value function parameterized by θ that represents the expected discounted return , i.e . V π ( st , g ) : = E [ Rt|θ ] , or to learn a policy π that maximizes expected discounted return . The Q-function Qπ : S×G×A → R also depends on goals . Notice that the transition probability distribution is independent of goals , it is possible to train an approximator to the Q-function using direct bootstrapping from the Bellman equation Qπ ( st , g , at ) : = Est+1 [ rt + γV π ( st+1 , g ) ] . ( 1 ) 2.3 HINDSIGHT EXPERIENCE REPLAY . Experience replay is the key strategy of off-policy RL to remember and reuse past experiences , which has shown great power in solving large sequential decision-making problems , such as Atari Games ( Mnih et al. , 2015 ) and Robotic Control ( P. et al. , 2015 ) . By resampling experiences for agent training , it makes better use of experiences than on-policy RL . Let µ : S × G → A denote a universal behavior policy and π denote the target policy . In off-policy RL , the target policy can learn from experiences generated by any behavior policy as long as if P ( a = π ( st , g ) ) > 0 , we have P ( a = µ ( st , g ) ) > 0 at each t. If any state-action pair ( st , g , a ) is unavailable for behabior policy , there will be approximation error in the estimation of Qπ ( st , g , at ) . For any off-policy RL algorithm , HER modifies the desired goals g in the replay transitions to some achieved goals g′ sampled from failed episodes . Specifically , it stores transitions not only with the original goal used for its episode but also with a subset of other goals . Notice that for UVFAs , Eq . ( 1 ) holds with relabeled experiences , which makes it possible to relabel past experiences with additional goals . HER generates additional goals using hand-crafted heuristics . It proposes various goals generation strategies , e.g . future strategy that replays with m random states which come from the same episode as the transition being replayed and were observed after it . The hyperparameter m controls the ratio of relabeled experiences to those coming from normal experience replay . The unexplored goal g′ affects the estimation of Qπ ( st , g′ , at ) , which is performed by a variant of Eq . ( 1 ) Qπ ( st , g ′ , at ) = Est+1 [ rt + γQπ ( st+1 , g′ , a ) ] , ( 2 ) where a is sampled from π . The target policy may select an unfamiliar action a at the next stategoal pair in the backed-up value estimate . More generally , the estimate , Qπ ( st+1 , g′ , a ) , will be unreliable in lack of sufficient visitations near ( st+1 , g′ , a ) . Especially when relabeling experiences without further exploration , the universal value function could not generalize well over pseudo goals .

Summary: This paper focuses on the problem of goal conditioned reinforcement learning. The authors propose an alternative way of performing Bellman updates for goal conditioned value function. Specifically, the proposed method first partitions the space of state goal pairs by performing a K-means clustering, and then estimates the visitation frequency of a state goal pair to be inversely proportional to the maximum difference of Q values within the cluster. For state goal pairs with low visitation frequency, the authors construct a lower bound of the Bellman target value by finding the Q value of a near state goal pair and subtracting the Lipchitz value multiplied by the distance. The authors then use this lower bound as target value to update the Q function. To generate goals for hindsight replay, the authors adopted a skew-fit type method to the empirical distribution of the K-means clusters to sample goals for replay.