Hindsight Curriculum Generation Based Multi-Goal Experience Replay
Authors: Xiaoyun Feng
Venue:  (2021)
Keywords: reinforcement learning
multi-goal task
experience replay

Abstract: In multi-goal tasks with sparse rewards, it is challenging to learn from tons of experiences with zero rewards. Hindsight experience replay (HER), which replays past experiences with additional heuristic goals, has shown it possible for off-policy reinforcement learning (RL) to make use of failed experiences. However, the replayed experiences may not lead to well-explored state-action pairs, especially for a pseudo goal, which instead results in a poor estimate of the value function. To tackle the problem, we propose to resample hindsight experiences based on their likelihood under the current policy and the overall distribution. Based on the hindsight strategy, we introduce a novel multi-goal experience replay method that automatically generates a training curriculum, namely Hindsight Curriculum Generation (HCG). As the range of experiences expands, the generated curriculum strikes a dynamic balance between exploiting and exploring. We implement HCG with the vanilla Deep Deterministic Policy Gradient(DDPG), and experiments on several tasks with sparse binary rewards demonstrate that HCG improves sample efficiency of the state of the art.

Under review as a conference paper at ICLR 2021
HINDSIGHT CURRICULUM GENERATION BASED MULTI-GOAL EXPERIENCE REPLAY
Anonymous authors Paper under double-blind review
ABSTRACT
In multi-goal tasks with sparse rewards, it challenging to learn from tons of expe- riences with zero rewards. Hindsight experience replay (HER), which replays past experiences with additional heuristic goals, has shown it possible for off off-policy reinforcement learning (RL) to make use of failed experiences. However, the re- played experiences may not lead to well-explored state-action pairs, especially for pseudo goal, which instead results in a poor estimate of the value function. To tackle the problem, we propose to resample hindsight experiences based on their likelihood under the current policy and the overall distribution. Based the hindsight strategy, we introduce a novel multi-goal experience replay method that automatically generates training curriculum, namely Hindsight Curriculum Gen- eration (HCG). As the range of experiences expands, the generated curriculum strikes a dynamic balance between exploiting and exploring. We implement HCG with the vanilla Deep Deterministic Policy Gradien(DDDPG), and experiments on several tasks with sparse binary rewards demonstrate that HCG improves sample efficiency of the state of the art.
INTRODUCTION
Multi-goal tasks with sparse rewards present a big challenge for training a reliable RL agent. In multi-goal tasks (Plappert arl 2018), agent learns to achieve multiple different goals and re- ceives no positive feedback until it reaches the position defined by the desired goal. Such a sparse rewards problem makes difficult to reuse past experiences for that the positive feedback rein- force policy is rare. It's impractical to carefully engineer a shaped reward function (Ng et all [999 Popov et all 2017) that aligns with each task or assign set of general auxiliary tasks (Riedmiller et all 2018 g). which relies on expert knowledge. To enrich the positive feedback, /Andyychowizz
all (2O)V) provides HER, a novel multi-goal experience replay strategy, which enables an agent to learn from unshaped reward, e.g. binary signal indicating successful task solving. Specifically, HER replaces the desired goals with the achieved ones then recalculates the rewards of sampled experiences. By relabeling experiences with pseudo goals, it expands experiences without further exploration and is likely to turn failed experiences into successful ones with positive feedback.
Experience relabeling makes better of failed experiences. However, not all the achieved goals lead the origin experience to a reliable state-action visitation under the current policy. (For sim- plicity, we denote state-goal pairs augmented states.) A Value function of a policy for specific state-action pair can generalize to similar ones, in return the estimate of the value function may get worse without sufficient visitation near the state-action pair. For HER, it almost replays uniform samples of past experiences whilst goals for exploring are finite. When performing a value update, the current policy could not give a credible estimate of the universal value function (Schaul all 2015). if the state-action pair is not well-exploree In other words, the current policy may have dif- ficulty in generalizing to the state-action pair. Recent works focus on improving HER by evaluating the past experiences where we sample pseudo goals from. HER with Energy-Basse (EBP) (Zha) & Tresp, 2018) defines a trajectory energy function as the sum of the transition energy of the target object over the trajectory. Curriculum-guider HER (CHER) rnn et all 2019)) adap- tively selects the failed experiences according to the proximity to the true goals and the curiosity of exploration over diverse pseudo goals with gradually changed proportion. Though these variants select valuable goals for replay, remains a challenge that the agent will not further explore most of the pseudo goals then it is risky to directly generalize over pseudo goals.
Prioritization

---

Under review as a conference paper at ICLR 2021
Pushing
Sliding
Pick-and-Ploce
ManipulateHlnck
Manipulanegg
ManipulatePan
Figure I: Tasks in the open-ended environment with sparse binary rewards. They include Pushing, Sliding and Pick-and-Place with Fetch robotic tions with a Shadow Dexterous Hand. The agent obtains a reward of if achieves the desired goal within some task-specific tolerance and otherwise.
well different in-hand object manipula-
Humans show great capability in abstracting and generalizing knowledge but takes a large number of experiences to learn to represent similar states with similar features. Fortunately, states with different achieved goals and desired goals may be similar intrinsically if the states and distance between them are similar. During episode, the distance varies widely for a fixed desired goal, which has the potential for generalization. Therefore, we take advantage of relative goals-distances between the achieved goals and the desired goals-to transform data. The relative goals strategy alleviates the challenge of generalization without sufficient data. By explicitly discovering similar the density of state-action visitations for unexplored
states in the replay buffer, it enables goals more conveniently. The density reflects the likelihood of the corresponding state-action pair, indicating whether it is well explored. Besides, the generalization over relative goals is feasible only if the explored relative goals are widely distributed (Schaml et all 2015). In word, it is significant ensure sufficient state-action visitations and maintain balanced distribution over valid goals.
In this paper, we present to resample hindsight experiences with relative goals strategy. The main criterion to sample experiences for reliable generalization is based on 1) the likelihood of the corresponding state-action pair under the current policy; 2) the overall distribution of the relative goals. By constantly adjusting the distribution of goals, we propose Hindsight Curriculum Genera- tion (HCG), which generates replay curriculum to progressively expand the range of experiences for training. The main advantage is that it makes efficient use of hindsight experiences as well as tries to ensure the generalization over state-action pairs. From the perspective of curriculum learning, the generated curriculum can be seen as sequence of weights on the training experiences, which guide the learning by automatically generating suitable replay goals.
Furthermore, we implemented HCG with the vanilla Deep Deterministic Policy Gradient(DDPG) (P et all 2015) various Robotic Control problems. The robot, 7-DOF Fetch Roboties arm which has a two-fingered parallel gripper or an anthropomorphic robotic hand with 24 degrees of freedom, performs the training procedure using the MuJoCo simulated physics engine (Lodorov at 2012). During the training procedure, our method extracts and leverages information from hindsight expe- riences with various state-goal pairs. We experimentally demonstrated that our method improves the sample efficiency of the vanilla HER in solving multi-goal tasks with sparse rewards. Ablation studies show that our method is robust on the major hyperparameters.
2 BACKGROUND
In this section briefly introduce the multi-goal RL framework, universal value function approxi- mators and hindsight experience replay strategy used in the paper.
2.1 MULTI-GOAL RL
Consider an infinite-horizon discounted Markov decision process(MDP), defined by the tuple where S is set states, a set of actions, is a set of goals, P: S A R is the transition probability distribution, r S x A G - IR is the reward function, and y E (0, 1) the discount factor. multi-goal RL, an agent interacts with its discounted MDP environment in sequence of episodes. At the beginning of each episode, the agent receives goal state G. In this paper we set that each g E 9 corresponds to goal state $g E S. Moreover, we assume that given state easily find goal g which is satisfied in this state. At each timestep (, the agent
2

---

Under review as a conference paper at ICLR 2021
observes state St E S, chooses and executes action d, A. And the agent will receive result- ing reward r(8t, dz. g) at the next timestep + 1. (For simplicity, we denote 7, = r(st, at at,g). .) In multi-goal RL, the reward function r is binary sparse signal indicating whether the agent achieves the desired goal state:
1, [[o(s++1) alla S 6g
0,
otherwise
where Φ : S - G, a known and tractable mapping, defines the corresponding goal representation of each state, and 6 is task-specific tolerance threshold defined in Plappert al. (2018).
2.2 UNIVERSAL VALUE FUNCTION APPROXIMATORS
Problems following the multi-goal RL framework (Plappert et all 2018) tell RL agent what to do using an additional goal imput. The goal is fixed in each episode whilst there is more than one goal to achieve. the continuous control problems, the agent could not afford to learn a policy for each goal. Instead, the policy should generalize not just over states but also over goals via deep neural networks. Formally, Universal Value Function Approximators (UVFA) (Schaul et all 2015) factor observed values into separate embedding vectors for states and goals, then learn mapping from (s, g) pairs to factored embedding vectors. Let T = 81, 41, 82; ,2,..., ST-1, -1- ST denote trajectory. which is also episode, Rs > K=t 'r, denote its discounted return at every timestep & E [1. Let π : S G A denote universal policy, V : S G + R denote its value function. The objective of the agent is to learn general value function parameterized by that represents the expected discounted return, i.e.
V (st, g) = E [R (
or to learn policy that maximizes expected discounted return. The Q-function Q S x R also depends on goals. Notice that the transition probability distribution is independent of goals, it is possible to train approximator to the Q-function using direct bootstrapping from the Bellman equation
Q" (Sts I, a.) = Ee,, In + >V (8(+1,99))]
2.3 HINDSIGHT EXPERIENCE REPLAY
Experience replay is the key strategy of off-policy RL to remember and reuse past experiences, which has shown great power in solving large sequential pocision-maaing problems, such as Atari Games (Mmih et all 2013) and Robotic Control(P. et all 2015). By resampling experiences for agent training, it makes better use of experiences than on-policy RI. Let S x g + A denote universal behavior policy and denote the target policy. In off-policy RI, the target policy can learn from experiences generated by any behavior policy as long as if (P(a = π (St, g)) > O, we have P(a- p(5t, g)) > 0 at each If any state-action pair (St, g, a) is unavailable for behabior policy, there will be approximation error in the estimation of Q" (st, 4:).
For any off-policy RL algorithm, HER modifies the desired goals g the replay transitions some achieved goals g' sampled from failed episodes. Specifically, it stores transitions not only with the original goal used for its episode but also with a subset of other goals. Notice that for UVFAs, (MI holds with relabeled experiences, which makes it possible to relabel past experiences with additional goals. HER generates additional goals using hand-crafted heuristies. It proposes various goals generation strategies, e.g. future strategy that replays with random states which come from the same episode as the transition being replayed and were observed after The hyperparameter m controls the ratio of relabeled experiences those coming from normal experience replay.
The unexplored goal g' affects the estimation of QT g', az), which is performed by a variant of (II
Q (st. g', ax) Ess4i [re + (S a)))
where a is sampled from т. The target policy may select an unfamiliar action a at the next state- goal pair in the backed-up value estimate. More generally, the estimate, Q ("((s+1.gf g', a). will be unreliable in lack of sufficient visitations near ($++1. a). Especially when relabeling experiences without further exploration, the universal value function could not generalize well over pseudo goals.
3

---

Under review as a conference paper at ICLR 2021
3 HINDSIGHT CURRICULUM GENERATION
In this section, we introduce a hindsight experience selection method to sample valuable experiences for replay, as well maintain the generalization of the policy. The hindsight strategy operates preprocessed experiences with relative goals. First, we introduce the preprocessing and the general- izability assumption. Then propose practical clustering method to discover similar experiences and take the density of state-action visitations into account to give a more reliable value estimate. Fi- nally, we control the range of replayed goals by maintaining a balanced distribution, which generates a hindsight curriculum to guide the learning.
3.1 PREPROCESSING
The previous works (Andrychowicz al, 2017: Zhao & Tresp, 2018: Fang et all 2019R) simply concatenate state and a goal together as joint input for value function, so do we. As the deep neural network has a strong generalization ability, it should learn a similar representation for similar augmented states, i.c. state-goal pairs. In particular, it relies the generalizability assumption.
Assumption 1. A value function of a policy т for a specific state-goal pair (s, g) has some general- izability to another state-goal pair (s', 9") close to (s,
Yet generalizing the policy to unseen state-action pairs requires a large number of samples, given the huge state and action spaces that characterize continuous robotic control problems. As described in Section 233 replaying with uniform samples of past experiences without fully exploring pseudo goals leads to unreliable estimation. To better generalize not just over goals but also states, preprocess the experiences with a relative goals strategy. Specifically, we denote as the achieved goal and ý the relative goal where ğ g 9s- As described in Section 211 we obtain the achieved goal from a known and tractable mapping. When it comes to relatives goals space, Assumption 1 turns to more common assumption: A value function of a policy ग for specific relative goal has some generalizability to another relative goal ğ' close to ğ.
In practice, it is more convenient to use relative goals when evaluating the similarity of state-goal pairs, especially when neither states nor goals are close. The strategy provides various samples for generalization over relative goals. After the preprocessing, we manipulate data of the form (s, 9) to help estimate value function and resample hindsight experiences.
3.2 CLUSTERING AND EVALUATION
VALID EXPERIENCES ACQUIRING
To acquire sufficient data near state-action pair, we first perform clustering on augmented states in replay buffer, then estimate the likelihood of state-action visitation under the current policy.
Firstly, we perform K-means clustering over experiences with a distance function, which defines how close of (s, g) and (s', g'). Specifically, we adopt the Euclidean distance in the definition of d(+, +) since it is iask-irreeevant, i.c.
d((s,)) ()..)).C (s', (s' - d(s, +(g.)))))
where the hyperparameter I ensures that the influence of each item won't be omitted. It partitions augmented states into clusters in which each state belongs to the cluster with the nearest mean. easy sample similar states for new state after finding the nearest clustering center. Formally, for each (s, g). there is {(*", J" a² (d((d)(( 9), (s',
(,, (6",7") g²)) < e1, =
n. where e is small.
Secondly, we estimate the likelihood of state-action visitation with batch of similar states. If state-action pair is well-explored by a policy, the policy will give reliable and stable value estimates for similar states. Hence we define that the likelihood F(s, g, क (s, ğ)) is inversely proportional to the max estimate difference, i.e.
F(s, 9. ((. x maxisiso ||Q (x, Ø, ((, F)) (s "', (
(4)
Thus the estimated value more reliable for the state-action pair with a high likelihood score, and vice versa. (In practice, it takes desired goal g instead of input, i.e. Q" (8 (+.
4

---

Under review as a conference paper at ICLR 2021
EVALUATION
Approximating the value of state-action pair from scratch can be difficult with the sparse rewards. Besides evaluating the likelihood of a state-action visitation, we modify the Eq,(2) to give the value a bound related similar states.
Assume that we have (s.g) g) ~ (s',g') (q.(3) evaluats the similarity between them and 61 0 small. Inspired by Assumption (s. g) and (s', g') are similar, and as excuting the identical the value function can generalize from one to another. Formally, the generalizability can be specifically expressed the Lipschitz continuity of state-action value function Ren (2I11):
) ~ I' and d( d((s.g), (s',g') < c1, where d(*, *) defined in
(Q" (s, Ø)) - (s', T'' (s),,'')) Ld((s, g), (s', ('))
where L > is the Lipschitz coefficient and d(x;, x) defined in Eq.(E) evaluats the similarity between (8,g) g) and (8', "") In the open-ended environment, it's reasonable to claim that the bound (q.
holds for most of the state-goal pairs when d((s; .), (s', ,'))
)) is small.
As described Section 2 unsufficient visitations lead to an unreliable estimate. When the like- lihood of state-action visitation is low, it is alternative to sample action from the excuted ones at the similar state. Therefore, we take the likelihood into account to sample the action the next state for the critic. After acquiring a batch of experiences with similar states for the we modify the approximated
(s1+1, 9) pair, {(s', g", aa')ld((s, ())(ss (')) < G1. = value Q" (st+1, 9, x (8++1,9) in Eq.
Q" (* (.
F(S ((141, > 62
("(st+.,0,a)
( ('', aa) Ld((s++1,)) (s', (s',)"),, Isish
max
otherwise
where E2 > 0 is the threshold of the likelihood. It gives a reliable bound of Q value when the action generated by the policy is unfamiliar. Consequently, perform the value update in Eq.(() via
Q (8t 9, az) F.,,, [re + 7Q" (81+1-0,
Theorem Consider the vector space V over state-goal pairs. For any U.V E V, we define the metric as I|U - Vo = max(s.g)eSxc, (s(,)) [U(s,))-V(s, (s,,, g) Based on (B) we define the Bellman expectation backup operator T for policy π. For each (st, g).
T (V = Earees [r+ + 7V (8++1,9))
where
V*((L+1, Eu [Q ($141,
The value iteration formulates as performing 7 on vector space V. Then the Bellman expectation backup operator T with the modified value function is 7/- contraction, i.e.
[TT T (V)Il I ~>1 - Vll
The proof is trivial. There is doubt that it gives the value reliable bound well maintains the convergence of the value update, for that the 7 is a contraction.
3.3 GOALS REPLAY
We have given brief description of how HER and its variants sample additional goals from past experiences for replay in Section M In short, it is common to prioritize experiences that lead to valuable goals, whilst each method evaluates goals against different set of criteria. For our method (in Section (.2),, we propose to give reliable estimate with similar states, which are generated by clustering past experiences. is feasible only if samples are widely distributed, especially for relative goals. Thus, it is crucial to maintain diverse relative goals in the experiences.
In Robotic Control problems, G is a set of goals with finite volume. After the preprocessing, each g G leads to various relative goals. Maintaining a broad distribution of relative goals turns to mathematically maximize the entropy of the distribution. Since each relative goal is feasible, max- imizing the entropy of the distribution may seem to maintain uniform distribution. Inspired by the skew-fit strategy in (Pong et al, 2019) that iteratively increases the entropy of a generative model, propose heuristic operator to manipulate balanced distribution over valid relative goals.
5

---

Under review as a conference paper at ICLR 2021
CURRICULUM GENERATION
Let (pi, 2, be the distributions of relative goals resample the hindsight experiences. At i-th iteration, we sample N augmented states from valid experiences with Since we have parti- tioned M states into clusters (Cx: k =,2, ....... } it is trivial construct empirical distributions of clusters for past valid experiences and sampled hindsight experiences. Formally, by classifying state the nearest cluster, we get
M
Poe; (Ck)
Σ M
1
((+,)) E C.).
Phos (Cx)
Σ (((,0) Ca). (s,9) P..
where Pve. for valid experiences and Phe. for hindsight experiences.
Then we firstly skew the Pee. with the Phe. via
Pakewed, (Cx)
Pue, Za Pve. (Ck)Phe, (Cx), [-1,0],
where Zo is the normalization factor. Secondly, we estimate the Gaussian densities of clusters by calculating the mean l4g and the covariance of of goals for each cluster. We expand Pskeoed, distribution goals and fit pi+1 to it according to
P+t1 + Σ Pskewed, (C pk,
(8)
Repeat the process until we maintain uniform distribution over clusters broad distribution of relative goals. The Gaussian mixture model can trained reasonably fast for RL agents and we take lazy update on the clustering.
4 MULTI-GOAL EXPERIENCE REPLAY
The core idea behind Hindsight experience is that after experiencing some trajectory т, it stores in the replay buffer every transition 8, - St+1 not only with the original goal used for this episode but also with a subset of other goals. To give a reliable estimate for unseen goals, we make use of hindsight experiences well as try ensure the generalization over state-action pairs. particular, we present to resample hindsight experiences with a relative goals strategy (in Section B). It samples goals from experiences with high likelihood of the corresponding state-action pair under the current policy and maintains broad overall distribution of the relative goals.
Here we incorporate our accelerating the learning in multi-goal tasks with sparse binary rewards. During the process, the up- dated distributions (ps. i = 1, 2.2.... automatically form curriculum as the range of sampled goals progressively expand. Gradually changed probability also strikes balance between exploitation and exploration.
Curriculum Generation with vanilla HER, aiming at
The detailed algorithm is shown in Algorithm MI namely Multi-Goul Experience Replay. (For sim- plicity, we hide the relative goals strategy.)
EXPERIENMENTS
We employ environments for multi-goal RI. introduced in Plappert et all (2018) and some manipu- lation tasks are shown in Figure II This section is organized as follows. First, we compare the per- formance between vanilla HER, HER with evergy-Based Prioritization, Curriculum-guider HER and our methods. All the variants are implanted with the DDPG algorithm. Policies are represented as Multi-Layer Perceptrons (MLPs) with Rectified Linear Unit (ReLU) activation functions as in AAdrychowicc al] (2017). Then construct ablation experiments major hyper-parameters.
The whole training procedure is performed in the simulation. For improved efficiency, we use 20 workers each with rollouts which average the parameters after every update.

---

Under review as a conference paper at ICLR 2021
Algorithm I Multi-Goal Experience Replay
Require:
off-policy RL algorithm A
distribution p for resampling goals for replay
a reward function S x A x
Initialize & Initialize Replay buffer I Initialize K,1, 7 for episode
M do
Sample goal and an initial state
for f T T-1 do
Sample an action using the behavioral policy from A:
Gr f (selgg) (l) denotes concatenation)
Execute the action de and observe new state 8c+-1
Ti :=:= T(2,, 4z.
Store the transition (sullg, ae Te, $141ll9) in
end for
Perform the K-means clustering in R as describe in Section 2
for t= +
do
Sample set of additional goal for replay G with p in R for g' E G do
: r( Gt.g')
Store the transition (stllg or, str1llg)) in R
end for end for for / / N do
Sample minibatch B from the replay buffer R Perform one step of optimization using å and minibatch with the modified value func- tion via (G)
end for
Update with via (S) in Section 5.K end for
If the agent achieved the desired goal in an episode, we consider it success. We show the learning cures the median success rate of tasks different settings. The results averages across 5 random seeds and are with shaded areas represent one standard deviation. Experiments demonstrate that our method learns efficiently across different episodes and is robust on the major hyperparameters.
LEARNING CURVE IN MULTI-GOAL SETTING
For all Experience Replay we store each transition in the replay buffer twice: once with the goal used for the generation of the episode and once with the goal corresponding to the future state sampled from the episode. The major difference between the variants is how to sample replay goals, which state in Section 0 and E Notice that there is modified value iteration for our variant only. All the variant shares the same environment setting and experimental configuration, except for their private controlling parameters. The policies and parameters will be listed in the Appendix.
From the learning curve in Figure E. we can see the success rates during test rollouts increase with the training process. Our method outperforms all the variant improves the learning efficiency. It also maintain stable high success rate. which is comparable with the vanilla HER algorithm. The learning efficiency indicates that the modified value function matches well with the distribution of tasks. More results will be shown in the Appendix.
early stage, which demonstrates that
ABLATION STUDIES
In this section, we discuss key design choices for our Hindsight Curriculum Generation method that provide substantially improved performance. The major hyperparameter the Lipschitz coefficient
7

---

Under review as a conference paper at ICLR 2021
2011
Pushing
Pick-and- Placc
HandManipulatclooc
Figure 2: Learning curves for variants with multiple goals. All curves shown are trained with default DDPG hyper- parameerass shows that at an early stage, our method (labeled MultiGoalER) learns more efficiently and can maintain stable high success rate.
that controls the value generalization. Notice that the number of clustering points K is also essential. But find that the performance is not sensitive when it varies. Thus keep it constant during all the training episodes. Results in Figure E indicates that the choice of L is robust. More results will be shown in the Appendix.
Pushing
Sliding
Pick-and-Placee
Figure 3: Ablation studies on hyper-parameter L. All curves shown trained with default DDPG hyper-parameters.
6 RELATED WORKS
Hindsights in RL: HER introduces hindsight relabelling scheme to extract more information from failures. Temporal Difference Model(TDM) (Pong et all 2018) generalizes policy not only unseen goals but also a multi-step temporal scale by relabelling. Hindsight Policy Gradient (HPG))(Raubed stall 2017) adopts the potential for goal-conditional policies enable higher-level planning based on subgoals in policy gradient methods. Generalized Hindsight (GH)(L all 2020) converts the data generated from the policy under one task to a different task. Morever, Exploration via Hindsight Goal Generation (HGG (RGn) et all 2019) constructs a curriculum on goals guiding the exploration of the environment. Dynamic HER (DHER) ([ (Fang et all 20197) assembles successful experiences from two relevant failures with dynamic goals.
CONCLUSION
In this paper, propose Hindsight Curriculum Generation (HCG) sample replay experiences that better generalize to unfamiliar goals. It analyzes the likelihood the corresponding state- action pair under the current policy and the overall distribution of the relative goals. Combing with an policy RL trainer, we introduce a multi-goal experience replay method that automatically generates replay curriculum to progressively expand the range of experiences for training. We have evaluated the method on several Robotic Control problems in the multi-goal tasks suffering from sparse rewards, and experiments demonstrate that it outperforms the state-of-the of-te------httttttt methods on the sample efficiency.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems 30, pp.
2017.
Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang. DHER: Hindsight experience replay for dynamic goals. In International Conference on Learning Representations, 2019a.
Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curricllun-guided hindsight experience replay. In Advances in Neural Information Processing Systems 32, pp. 12602-12613. 2019b.
Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. ArXiv:abs/002..1 1708, 2020.
V. Mnih, K. Kavukcuoglu, and D. Silver. Human-level control through deep reinforcement learning. Nature 518, pp. 529-533, 2015.
A. Y. Ng. D. Harada, and S Russell. Policy invariance under reward transformations: Theory and application to reward shaping. ICML, 99:278 287, 1999.
Lillicrap T. Hunt J., Pritzel A., T. Heess, N.and Erez, Yand Silver D. Tassa, and Wierstra D. Continuous control with deep reinforcement learning. In arXiv preprint 1509.0297 2015.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Macick Chociej, Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. CoRR, abs/ 1obp/880..0944, 2018. URL
Vitchyr Pong. Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model- free deep RL for model-based control. CoRR, abs/1802.0908/ 2018. URL http: orc/ abs 1802. 09081.
Vitchyr H. Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- State-covering self-supervised reinforcement learning. CoRR, abs/1903.03698, 2019. URL
http: htrgg/ab//9990 03698.
Ivaylo Popov, Nicolas Heess, Timothy P. Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Vecerík, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin A. Riedmiller. Data-efficient deep reinforcement learning for dexterous manipulation. CoRR, abs/1704.03073, 2017. URL http:
//axxiv.org/abs/11 03073.
Paulo Rauber, Filipe Mutz, and Jürgen Schmidhuber. Hindsight policy gradients. CoRR, abs/1711.06006 2017. URL http: wwwpxii.org/aae... abs http//bbss/lll..eee
Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. CoRR, Nbs/1990.04279,, 2019. URL http:://aras www.wx.www.//b///000 042279
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch, 2018.
T. Schaul, D. Horgan, K. Gregor, and D Silver. Universal value function approximators. ICML, pp. 1312 1320, 2015.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Rui Zhao and Volker Tresp. Ensrgy-based hindsight experience prioritization. abs/1810.0136. 2018. URL httl:http///xeee wrg./iioorrr////oooooo abs abs01810.00163.
CoRR,


Reviews:
[{"review_id": "uSBk8alBhp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper focuses on the problem of goal conditioned reinforcement learning. The authors propose an alternative way of performing Bellman updates for goal conditioned value function. Specifically, the proposed method first partitions the space of state goal pairs by performing a K-means clustering, and then estimates the visitation frequency of a state goal pair to be inversely proportional to the maximum difference of Q values within the cluster. For state goal pairs with low visitation frequency, the authors construct a lower bound of the Bellman target value by finding the Q value of a near state goal pair and subtracting the Lipchitz value multiplied by the distance. The authors then use this lower bound as target value to update the Q function. To generate goals for hindsight replay, the authors adopted a skew-fit type method to the empirical distribution of the K-means clusters to sample goals for replay.\n\nThe authors evaluate the proposed method on 3 simulated robotics manipulation tasks and compare against HER-EBP and CHER as baselines. The results indicate that the proposed method outperforms prior methods in terms of sample efficiency. The authors also provide ablation studies for the Lipschitz constant hyperparameter.\n\n\n### Comments\nThe paper is well written and the idea proposed in this paper is easy to follow. The authors clearly demonstrate the advantage of the proposed method in the 3 robotic manipulation tasks. However, I do have some concerns about the proposed method and results.\n\nFirst of all, the proposed method seems to be based on some heuristics which have neither been proven to be correct nor been verified empirically. For example, the authors estimate the visitation frequency of a state goal pair to be inverse of max Q function variation within the cluster (equation 4). It is not clear why this is a good estimate, since it is possible for an entire cluster to have low visitation frequency and also low Q value variations because the Q function has not been trained much in the region of the cluster. It would be important to provide either a proof or an empirical study.\n\nMoreover, the proposed method relies on some assumptions that might not hold true for many goal conditioned environments. For example, one assumption lies in the use of L2 distance in equation 3. In many goal conditioned tasks such as maze navigation, the L2 distance might not be a good metric between state goal pairs since two states close in L2 distance could be on two sides of a wall. The paper does not include discussion or empirical evaluations for such tasks.\n\nThirdly, it is not clear to me the Bellman iteration in equation 6 would converge to the optimal value. The authors only prove that it is a contraction and therefore will converge to some fixed point. However it is unclear to me whether the fixed point would be the same optimal Q value as the unmodified Bellman iteration.\n\nFinally, the proposed method introduces a few extra hyperparameters, such as the cluster K, the Lipchitz constraint of Q function L and the visitation frequency threshold. From the ablation study, we know that the proposed method is sensitive to the choice of L. Therefore, the natural question is whether the observed performance improvement is due to the tuning of these extra hyperparameters. Therefore, it would be important to perform more experiments on a wider range of tasks such as those in [1] and [2].\n\nDue to these limitations, I would not recommend acceptance for this paper before they are addressed.\n\n\n\nReferences\n\n[1] Pong, Vitchyr, et al. \"Temporal difference models: Model-free deep rl for model-based control.\" arXiv preprint arXiv:1802.09081 (2018).\n[2] Eysenbach, Ben, Russ R. Salakhutdinov, and Sergey Levine. \"Search on the replay buffer: Bridging planning and reinforcement learning.\" Advances in Neural Information Processing Systems. 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fPgrizr4Ycs", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a wealth of changes to the standard HER agent and obtain a performance improvement in 3 multi-goal tasks. The main observation made by the paper is that relabeled experiences may be very off-policy / out-of-distribution, and so value estimates for such experiences will be bad. \n\nTo help with this the authors propose the following:\n- apply K-means to cluster real (state, goal) tuples. \n- estimate the likelihood of a given (state, goal) tuple X by finding the closest cluster center in real experience, sampling n real experiences Y^i in that cluster, and taking the minimum of 1/d(X, Y^i). \n- they change the Bellman targets in case the likelihood estimate is low. In particular they change it to a lower bound based on some nearby real experience minus the distance times Lipschitz constant.\n- they use relative goals (g_original - g_current) instead of absolute goals (g_original).\n- there is some kind of curriculum on goal relabeling\n\nThis paper is hard to follow. The word usage and sentence structure is unnatural, and I find myself guessing at what exactly the authors mean. This carries through to the math. I think I understand what the modified Bellman backup above equation (6) is doing, but I'm still not entirely sure. I'm also not really following the Section that includes equation (8). As a result, the contributions are a bit unclear. \n\nTheorem 1 is not trivial and there is indeed doubt in my mind. A proof should be provided upon revision. \n\nThe related works section can be greatly improved. You should be relating the related work to your own.\n\nAn appendix was not provided, despite being referenced, and so the paper is missing additional results (the 3 environments are insufficient), implementation details, and hyperparameter details. Without these, this paper cannot be reimplemented and is not in a publishable state.\n\nNits:\n- Isn't Equation (5) just the definition of Lipschitz continuity, so I'm confused by what is meant by \"it's reasonable to claim that [it] holds\".\n- ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y_jx9d3MH9X", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper developed methods for resampling from the hindsight experience replay buffer. The resampling strategy was developed based on the current policy, and the overall distribution of the relative goals. As the distribution over goals evolves over time, the multi-goal agent's replay curriculum is adjusted throughout the learning process. The developed approach, called hindsight curriculum generation (HCG), was applied to DDPG, and evaluated using a set of four robot control problems. Results show that HCG performed better than a few baseline methods, and its performance was claimed to be insensitive to the choice of hyper-parameters. \n\nThe paper has a few issues. The sampling from hindsight experience is partially based on the likelihood of the corresponding state-action pair under the current policy. The reviewer is not sure that this strategy makes sense when the developed approach was applied to off-policy RL methods (DDPG in this case). It seems to be suggesting that, without exploration (completely following current policy), off-policy RL methods get the best results. Using only recent policies limits the variance over the collected experience. Some more discussions and justifications are needed for \"the likelihood of the corresponding state-action pair under the current policy.\"  \n\nThe two baselines of CHER and HER-EBP were not mentioned in the experiment section. The reviewer had to search the whole paper, and found the acronyms mentioned in the introduction section. The results are suspicious: how come CHER and EBP performed even worse than naive HER in Figure 2? The results are inconsistent to those reported in the CHER and EBP papers. \n\nIt's stated that \"Results in Figure 3 indicates that the choice of L is robust.\" This is apparently not the case from Figure 3. In the pick-and-place task, when L=5, it reached 0.8 success rate in 15 epochs, whereas the agent couldn't succeed at all when L=1 or L=0.5. The conclusion was not supported by evidence or experimental results. \n\nThe paper mentioned \"Appendix\" in a few places, but there is no appendix in this submission. \n\nIt's suggested to experiment with RL methods other than DDPG. There's the potential of applying the developed approach to on-policy methods that have been evident to performing better than DDPG in robot control tasks. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "AAoZKG_Mn8W", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces an extension of Hindsight Experience Replay (HER) called Hindsight Curriculum Generation (HCG) which is demonstrated to learn faster in multi-goal RL benchmarks.\n\nThe approach consists of two distinct contributions: First, is the idea of learning Q-values as a function of relative goals, instead of absolute goals. Second, is the idea of constructing a distribution over the relative goals from the replay buffer using a simple clustering algorithm and defining a sampling distribution over them. \n\nThis approach is then evaluated on three multi-goal RL environments that were previously open-sourced by OpenAI. Learning curves from these domains show that their approach HCG can produce faster learning when compared to the baselines.\n\nPros: \nThe approach seems to produce faster learning compared to the baselines considered.\n\nCons:\nThe paper seems to be rushed, making it hard to follow the ideas presented in the paper. It also has a few typos.\n\nThe motivation for the introduced approach is hard to understand, which makes it difficult to understand why/when the approach works for a given domain. The goal-sampling strategy seems to be arbitrary and no motivation is presented here to justify such an approach.\n\nThe related work section in the paper is not detailed. There are many approaches (listed below) that look at discovering curriculums for improving the speed of learning, and these can be considered to be orthogonal to HER, making it applicable to the current setup the authors have considered. The authors need to discuss how their approach of producing a curriculum relates/differs with the curriculum-based approaches.\n\nThe algorithm section in the main text does not have the necessary details to help understand the approach. In the pseudocode, many notations are used to present the approach, but I do not see the definitions for them in the main text. \n\nFrom the experiments, it is not possible to tell whether the improvement in observed performance is due to the idea of using relative goals or the goal-sampling strategy. I would suggest introducing a baseline HER agent that operates on the relative goals, similar to the HCG agent. This should help inform which part of the idea is important.\n\nTheorem 1 in the paper does not seem relevant to the approach and seems to be arbitrarily presented. It would be better if the authors could clarify how this theorem connects to their approach.\n\nForestier, S., Portelas, R., Mollard, Y., & Oudeyer, P. Y. (2017). Intrinsically motivated goal exploration processes with automatic curriculum learning. arXiv preprint arXiv:1708.02190.\n\nVeeriah, V., Oh, J., & Singh, S. (2018). Many-goals reinforcement learning. arXiv preprint arXiv:1806.09605.\n\nFlorensa, C., Held, D., Geng, X., & Abbeel, P. (2018, July). Automatic goal generation for reinforcement learning agents. In International conference on machine learning (pp. 1515-1528).\n\nGraves, A., Bellemare, M. G., Menick, J., Munos, R., & Kavukcuoglu, K. (2017). Automated curriculum learning for neural networks. arXiv preprint arXiv:1704.03003.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the idea of hindsight experience replay (HER) to learn Q functions with relative goals by constructing a distribution over relative goals sampled from a replay buffer using a clustering algorithm. This approach is evaluated on three multi-goal RL environments and is shown to learn faster than baselines.

${\bf Pros}$:
1. Faster convergence as compared to baselines
2. Interesting use of clustering in the context of HER but this choice is made without strong justifications or formal arguments

${\bf Cons}$:
1. Some of the key choices made in this paper are not justified or explained property, e.g. - the goal sampling strategy, choices made in the clustering algorithm and associated heuristics, implicit assumptions (e.g. R1 raised the question of using L2 distance in measuring metrics between two states) 
2. There are several choices made without sufficient formal arguments, verification or guarantees. 

The paper studies an interesting problem but could be made stronger by incorporating feedback received during the discussion period. 
Decision: Reject