Bootstrapped Hindsight Experience replay with Counterintuitive Prioritization
Authors: Jiawei Xu
Shuxing Li
Chun Yuan
Zhengyou Zhang
Lei Han
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement learning
hindsight experience replay
counterintuitive prioritization

Abstract: Goal-conditioned environments are known as sparse rewards tasks, in which the agent gains a positive reward only when it achieves the goal. Such an setting results in much difficulty for the agent to explore successful trajectories. Hindsight experience replay (HER) replaces the goal in failed experiences with any practically achieved one, so that the agent has a much higher chance to see successful trajectories even if they are fake. Comprehensive results have demonstrated the effectiveness of HER in the literature. However, the importance of the fake trajectories differs in terms of exploration and exploitation, and it is usually inefficient to learn with a fixed proportion of fake and original data as HER did. In this paper, inspired by Bootstrapped DQN, we use multiple heads in DDPG and take advantage of the diversity and uncertainty among multiple heads to improve the data efficiency with relabeled goals. The method is referred to as Bootstrapped HER (BHER). Specifically, in addition to the benefit from the Bootstrapped version, we explicitly leverage the uncertainty measured by the variance of estimated Q-values from multiple heads. A common knowledge is that higher uncertainty will promote exploration and hence maximizing the uncertainty via a bonus term will induce better performance in Q-learning. However, in this paper, we reveal a counterintuitive conclusion that for hindsight experiences, exploiting lower uncertainty data samples will significantly improve the performance. The explanation behind this fact is that hindsight relabeling largely promotes exploration, and then exploiting lower uncertainty data (whose goals are generated by hindsight relabeling) provides a good trade-off between exploration and exploitation, resulting in further improved data efficiency. Comprehensive experiments demonstrate that our method can achieve state-of-the-art results in many goal-conditioned tasks.

Under review as a conference paper at ICLR 2022
BOOTSTRAPPED HINDSIGHT EXPERIENCE REPLAY WITH COUNTERINTUITIVE PRIORITIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Goal-conditioned environments are known as sparse rewards tasks, in which the agent gains a positive reward only when it achieves the goal. Such an setting re- sults in much difficulty for the agent to explore successful trajectories. Hindsight experience replay (HER) replaces the goal in failed experiences with any practi- cally achieved one, so that the agent has a much higher chance to see successful trajectories even if they are fake. Comprehensive results have demonstrated the effectiveness of HER in the literature. However, the importance of the fake trajec- tories differs in terms of exploration and exploitation, and it is usually inefficient to learn with a fixed proportion of fake and original data as HER did. this paper, inspired by Bootstrapped DQN, we use multiple heads in DDPG and take advantage of the diversity and uncertainty among multiple heads to improve the data efficiency with relabeled goals. The method is referred as Bootstrapped HER (BHER). Specifically, in addition to the benefit from the Bootstrapped ver- sion, we explicitly leverage the uncertainty measured by the variance of estimated Q-values from multiple heads. A common knowledge is that higher uncertainty will promote exploration and hence maximizing the uncertainty via bonus term will induce better performance in Q-learning. However, in this paper, we reveal a counterintuitive fact that for hindsight experiences, exploiting lower uncertainty data samples will significantly improve the performance. The explanation behind this fact that hindsight relabeling itself largely promotes exploration, and then exploiting lower uncertainty data (whose goals are generated by hindsight rela- beling) provides a good trade-off between exploration and exploitation, resulting in further improved data efficiency. Comprehensive experiments demonstrate that our method can achieve state-of-the-art results in many goal-conditioned tasks.
INTRODUCTION
Deep reinforcement learning (DRL) has gained significant achievements solving games (Zha et al. 2019 Crespo & Wichert 2020; Oh et al. 2021) and robotics problems (Rahimi et al.] 2018 Nguyen & La 20199] Zhu et al. 2021 3. Despite these successes, sparse rewards problems are still challenging. In sparse reward tasks, the agent gets a non-negative reward only when it achieves some goal. As a consequence, is very difficult for an agent to see successful trajectories. Indeed, many state-of-the-art RL algorithms, such as TRPO (Schulman et al. 2015). PPO (Schulman al., 2017). DQN (Mnih et al. 2015), and DDPG (Lillicrap et al. 2019), etc., often fail to perform well in sparse rewards environments.
To solve sparse rewards problems, there have been rich literature devoted to reward engineering. Potential Based Reward Shaping (PBRS) (Ng et al. 1999) uses difference between two potential functions to design a shaping function, where the potential function comes from the expertise knowl- edge and evaluates the value of state. Potential Based state-action Advice (PBA) (Wiewiora et al. 2003) extend PBRS to use potential function to evaluates the value of state-action pair. Dynamic Potential Based (DPB) (Devlin & Kudenko 2012) approach takes time as one of the inputs of the potential function. However, reward engineering usually requires domain-specific knowledge and it often suffers from sub-optimal performance. A brilliant idea for solving coal-coondiioned tasks is using hindsight experiences. HER (Andrychowicz et al. 2017), which replaces the desired goal in failed experiences with some achieved state in the failed trajectory to pretend that the agent obtains positive reward, has been demonstrated its effectiveness in wide range of goal-conditioned appli-

---

Under review as a conference paper at ICLR 2022
cations. However, HER has its own limitations that when relabeling the goal in one failed trajectory, there exists many choices of the pseudo goals that any achieved state can be selected. Generally, uniformly sampling one of them, as HER did, is not the most effective way from the perspective of importance sampling.
In this paper, we focus on improving the data efficiency in HER via the following directions: 1) inspired by Bootstrapped DQN (Osband et al.) 2016), we propose a bootstrapped version of HER that employs multiple Q-value heads to increase the diversity in choosing the pseudo goals for deep exploration. In this paper, we use DDPG as the basic RL algorithm, while the proposed techniques can be integrated by any off the-lff RL algorithms: 2) in addition to the benefit on exploration by using the bootstrapped Q-learning, we further explicitly leverage the uncertainty among the multi- ple heads by sampling goals conditioning on the variance of the estimated Q-values. Many previous approaches have shown that data with higher value uncertainty can promote exploration, and max- imizing the uncertainty by incorporating bonus term in the reward has been widely considered, such as bandit algorithms (Lattimore & Szepesvári 2020). To our surprise, this is totally not the case in HER, On the contrast, sampling goals with lower value uncertainty first is demonstrated to be significantly better than uniform sampling, and sampling goals with high uncertainty in prior deteriorates the performance of HER a lot. We thus call this empirical fact as counterintuitive prior- itization. Indeed, this can be explained by the trade-off between exploration and exploitation when importing HER, bootstrapped Q-learning and importance sampling. We refer our algorithm to as the Bootstrapped HER with counterintuitive prioritization (BHER). We experiment with a number of control tasks with continuous action space. The experimental results verify the superiority of BHER over many saate-of-the-ane baseline methods.
2 RELATED WORK
Sparse rewards in goal-conditioned environment is one of the major challenges of RL at present. The earlier work is UVF (Schaul et al. 2013) algorithm, which uses state and goal together as the conditions for agent take action. Bootstrapped DQN (Osband et al.] 2016 uses the bootstrap principle to expand network structure. It uses multiple randomized value functions to estimate the Q-value of the state-action pair. This is equivalent to agent having multiple policies to interact with the environment, which effectively enhances agent's 's exploration. But these methods don' 't work well in the environment of sparse rewards. Hindsight Experience Replay (HER) (Andrychowicz et al.) 2017) algorithm recalculates reward by randomly replacing the desired goal in the failed transition with the achieved goal. In this way, pseudo successful data can be obtained, which can improve data efficiency. HER algorithm better solves the problem of sparse rewards.
Since then, there have been many studies related HER algorithm. Dynamic Hindsight Experience Replay (DHER) (Fang et al. 2019a) utilizes the HER algorithm to process dynamic goals in the environment. Competitive Experience Replay (CER) (Liu et al. 2019) uses the competition between two agents to obtain better exploration. Coal-Conditioned Supervised Learning (GCSL) (Ghosh ct al., 2020 is a kind of imitation learning. It uses successful trajectories generated by agent as expert demonstrates, and supervised learning to optimize lower bound the goal-oriented RL objective. PlanGAN (Charlesworth & Montana, 2020) is a model-based RL algorithm that uses GANs (Creswell al., 2018) to learn the model of the environment and make planning. But this method requires a huge amount of computation to select a single action.
Recently, researchers have realized that not all data have the same value to learn (Schulman al. 2017). More of the work is to further improve data efficiency by adding priority replay on HER algorithm. Energy-Bast Basedd HER (Zhao & Tresp 2018) calculates the kinetic energy, potential energy and rotational energy as the total energy of the achieved goal according to the energy principle in physics. Then it gives higher priority to transitions with larger energy. Maximum Entropy- based Prioritization (MEP) (Zhao et al. 2020) calculates the entropy of a trajectory which is used as priority. Curriculum-guide between the achieved goal and the desired goal by designing two functions, which are added together with a weight to balance exploration and exploitation. These methods can be regarded as a form of curriculum learning (Elman 1993 Bengio et al., 2009 Zaremba & Sutskever] 2014 Graves et al.) 2017). But these algorithms either make some assumptions about the environment or design some specific functions calculate the priority. Unlike the above works, BHER algorithm does not
HER (CHER) (Fang et al. 201 calculates the proximity and diversity
2

---

Under review as a conference paper at ICLR 2022
have any requirements the environment when calculating the priority, and it is easy to generalize to other goal-conditioned environments.
PRELIMINARIES
In this section, we introduce the preliminaries, such as the challenge of goal-conditioned environ- ment, HER (Andrychowicz et al. (0177) and Bootstrapped DQN (Osband et al. 2016) which moti- vate us to propose our BHER algorithm.
3.1 GODL-COND ENVIRONMENT
In RL, the problem considered usually composed of an agent and an environment. At time step t, the agent takes action a conditioned on the state 9; of the environment, After the environment receives the action at transfers the state 8t+1 and feeds back the reward rt. The agent learns policy π (a]s) to maximize expected discounted return:
G = E,
Σ 2
where T is trajectory (so, do, To...., -1- and is the discount factor.
In goal-conditioned environment, the agent takes an action by taking an additional input, i.e., a goal g. and the policy is hence function of (s, that goal-conditioned environments are sparse, and only when the agent reaches the goal, it gets a non- negative feedback. Therefore, it is difficult for the agent explore successful trajectories when its policy starts from scratch.
<--ls, Generally, the rewards in
3.2
HER AND BOOTSTRAPPED DQN
HER replaces the desired goal g the failed transition with some sampled achieved state g' in the current trajectory, and then recalculates the reward based on the pseudo goal g'. In this way, the agent can see successful trajectories much more frequently, even the goal fake. It has been demonstrated in rich literature that HER is very effective for solving goal-conditioned tasks and it can adopted by any off-policy algorithms, such as DQN (Mnih et al., 2015). DDPG (Lillicrap ct al. 2019 TD3 (Fujimoto et al. 2018), etc.
Bootstrapped DQN designs multiple value function heads which are initialized randomly in the network structure of the DQN algorithm, and all heads share the underlying network. Each time agent interacts with the environment, one of the value function heads is randomly selected as policy generate trajectory. Each value function head is updated according its own target network for temporally extended exploration, and each value function head only uses data generated by itself for updates. Bootstrapped DQN subtly applies the Thompson sampling (Russo et al. 2017) heuristic to RL which allows for deep exploration.
4
METHODOLOGY
In this section, we introduce the bootstrapped version of HER with multiple Q-value heads and the counterintuitive prioritization.
4.1 BOOTSTRAPPED HER
We use DDPG as the basic RL algorithm. In goal-conditioned tasks, the policy and Q-value take both the state and goal as input. A bootstrapped version of DDPG is dipicted in Fig. [ where both the actor network and critic network implemented with multiple heads. All critic heads share an underlying embedding network, similar to what was used in Bootstrapped DQN (Osband) et al. 2016), and actor heads share underlying network for feature extraction as well. Each actor head is updated according to its corresponding critic head, for which its pscudo goals are sampled independently and it is updated by according to its own recomputed rewards.
3

---

Under review as a conference paper at ICLR 2022
Q-value
Q-value
Action
Action
Head
Head
Head 1
Head
Critic Shared Network
Actor Shared Network
(State, Goal)
Figure 1: The network structure of Bootstrapped HER.
As has been demonstrated in Bootstrapped DQN, random initialization of the parameters of each head is sufficient to generate diversity for deep exploration. This is also verified in our experiments for BHER. this way, we can benefit lot terms of exploration with the cost increasing only small amount of the network parameters from multiple heads, since they share a large proportion of parameters in the embedding network.
In Appendix D provide illustration of the distribution of goals the agent can arrive in toy environment by using multiple heads. Compared to the standard HER, we can verify that the bootstrapped version of HER is much more efficient in terms of exploration.
4.2 COUNTERINTUITIVE PRIORITIZATION
As we have shown, employing multiple Q-value heads for HER can effectively promote explo- ration, since each head is updated by sampling its own trajectories and pseudo goals. Moreover, with multiple critic heads, it is natural to compute the variance among the estimated Q-values. In this section, we propose to explicitly leverage this information. Specifically, given a transition, we inference all the Q-values and calculate the variance, which is stored in the replay buffer. Then, at the back Back-propagation stage, we sample the mini-batch conditioning on the stored variance. A com- mon experience is that sampling the data with higher variance will further encourage exploration these transitions that the policy rarely see. Previous results (Pathak et al. 2017 Zhelo et al. 2018 Li al. 2020) have demonstrated this principle result in much better performance in RI. However, as we will explain below and verify in the experiments, this principle is never true for BHER.
the Reacher environment (for more details, please refer Appendix AA as a toy example to visualize the variance of the Q-values stored in data. Fig. showws that during the training process at different training epochs, the variances of the hindsight transitions are consistently smaller than the variances of the original transitions. This is explainable, because in hindsight transitions, the agent always receives positive (pseudo) rewards, while for the original transitions without goal relabeling, the agent fails to reach the goal at most of the time at the early training epochs, unless the policy good enough. Indeed, Fig. 2(c) when we have trained over more training epochs, the variance in the original transitions decrease. According to RL. the policy will be updated more aggressively over the transitions whereas the values are higher. Therefore, the hindsight experiences deserve lower variance of the Q-values in BHER.
To verify this,
Now, becomes straightforward that if we give high priority in sampling transitions with larger still sampling original transitions without goal relabeling. That is,
Q-value variances, not performing HER any more. Therefore, we should assign higher priority to hindsight transitions with smaller Q-value variances in the training process. That is, enhance the exploitation on the

---

Under review as a conference paper at ICLR 2022
hindsight transision
0.3
0.3
original transition
(a) epoch=1
hindsight transition
original traisattion
0.6
0.4
0.0
(b) epoch=10
hindsight tranatonn
original transs transsition
0.15
0.15
0.10
0.10
0.05
0.03
0.00
0.00
(c) epoch=30
Figure 2: Variance of the Q-values estimated from multiple heads in BHER Reacher. The left side denotes the variances the hindsight transitions (with modified goals) in the replay buffer, while the right side plots the variances of the original transitions. In this illustration experiment, the ratio of sampled original data is set to 50%, and another half of data composed by the hindsight samples (in the original work of HER, this ratio is set 20% by default). This is eliminate the impact of data imbalance on evaluating the variances in the data. In our formal experiments, this ratio keeps the value as the same HER.
hindsight experiences by sampling these data more aggressively. So, we refer this sampling principle as Counterintuitive Prioritization.
Indeed, BHER with Counterintuitive Prioritization provides trade-off between exploration and exploitation that the bootstrapped Q-learning encourages deep exploration and more diversity, while counterintuitive prioritization promotes exploitation in the data samples which are selected by the bootstrapped progress and are worthy exploration. As we will demonstrate in our experiments on wide range of environemnts, the above methods can improve the standard HER method by large margin performance.
4.3 BOOTSTRAPPED HER WITH COUNTERINTUITIVE PRIORITIZATION
In this section, we provide detailed deseription of the proposed algorithm. In the following context, the abbreviation BHER denotes the bootstrapped HER with counterintuitive prioritization by default.
Algorithm 1 Calculate Priority
1: Requirement: original buffer B3, and HER buffer B
2: while training is not ending do
3: Sample minibatch transitions B, from B,
4: Replace desired goals with achieved goals transitions B, with HER
5: Calculate variance of Q-value for each transition in B,
6: Store transitions B, in B, with variances
7: Update priority for each transition in Ba (see Equation 2b
8: end while

---

Under review as a conference paper at ICLR 2022
Similar to Bootstrapped DQN (Osband et al., 2016), in our implementation of BHER, all heads share most of the parameters, and each head is updated according to its own target. However, an important difference is that each head in Bootstrapped DQN has maintain its own replay buffer to distinguish the data (diversity) generated by different heads (policies), while in BHER this is not necessary. hindsight experiences, which not exactly generated by
Note that in BHER, most transitions the training policies, so there is no need to distinguish which head generated the transitions, and all the heads can share the same replay buffer.
To implement counterintuitive prioritization, we create two replay buffers. One replay buffer has larger size and it stores the original interactions between the agent and the environment, and we call it the original buffer B., Another one is a smaller buffer, which stores a proportion of sampled hindsight and original data, and we call this buffer the HER buffer B₁. During training, we first sample in the original buffer according to the original HER algorithm, replace the goals according certain proportion, and then store them the HER buffer. We then calculate the priority of each transition É in the HER buffer according to the formula:
T
O Maa of + ErEB, (o 0² +
pt =
(2)
where of is the variance of Q-values of transition t, and of is the largest variance in the current HER buffer; is a very small number for numerical stability, and we set it 1e in our experiments; T is temperature coefficient used to adjust the sharpness of the priority distribution in the HER buffer. Then, the transitions the HER buffer are sampled according to the calculated priority. Algorithm brriefl illustrates the implementation of priority calculation process in our algorithm.
For data generation, each time we use all the actor heads to generate multiple trajectories and store them in the original buffer to ensure the diversity of the original data. When sampling data in the original buffer, we use the 'future' method in the HER algorithm, and set replay times to That is, 20% of the data in the HER buffer is original data. Algorithm 2 of our algorithm. In our experiments, we set the number of heads to 8, which is sufficient to allow the agent explore well. The detailed settings of other hyperparameters reported in Section
brieffy illustrates the implementation
Algorithm Bootstrapped Hindsight Experience replay with Counterintuitive Prioritization
Initiate: an off-policy RL Algorithm A, original buffer B, and HER buffer B,
2: for training epoch = 0, 2, ... M - do
3: for cycle = 0, 1, - N do
4: for head & = 0, 1, ... K
do
5: Sample an initial goal as desired goal and initial state so
6: for timestep / 0, I... , do
7:
Task an action 0₂ from the behavioral policy of A, i.e., a ~ mx(-186,9)
8:
Observe new state 8t+1 and achieved goal gf
9:
Calculate reward == r(st, (t.9)
Store the transition (st, Tes Tt. 9() in B,
11:
end for end for
13:
Sample transitions B, from B, and store them in B, by Algorithm 1 i.e.,
14:
for head & = 0, 1,... K do
15:
Sample a minibatch transitions B2, according to priority in B, Optimize k-th head of A using minibatch transitions Bn end for
16:
17:
18:
end for
19:
Evaluate the average performance of all heads through 100 rollouts
20:
end for
5 EXPERIMENT
We use a number of goal-reaching and robotics control environments to evaluate our BHER al- gorithm and compare with state-of-the-art baselines. We first introduce the environment and

---

Under review as a conference paper at ICLR 2022
(a) PoineDDLargeEnv
(b) Reacher
(c) FetchRcach
(d) SawyerReach
SawyerDoor
(f) HandReach
(g)
HandManipullteeen
Figure 3: Environment.
comparison results. In the end, we adjust and compare our own algorithms and show the results of ablation experiments.
5.1 ENVIRONMENTS
Fig. 3 shows the goal-conditioned environments we have evaluated in experiment. Goal-reaching environments include Point2DLargeeFnv, Reacher, FetchReach and SawyerReach, and robotics con- trol tasks include SawyerDoor, HandReach and HandManipulatePPn.. The state space and action space of all environments are continuous, and the reward is sparse and non-negative. Only when agent reaches desired goal, it gets reward of 0, otherwise it receives reward of -1. The max- imum number of steps in each environment is set to 50. For a more detailed description of the environments, please refer Appendix
5.2 BASELINES
The implementation of all algorithms is based on the DDPG algorithm. We compare our algorithm with the following state-of-the-ard HER based methods:
HER (Andrychowicz et al. 2017), which does not use prioritization and randomly samples transitions.
HEREBP (Zhao & Tresp 2018 which calculates the energy of transitions and use as the sample priority.
CHER (Fang et al. 2019b), which calculates the proximity and diversity of transitions as the sample priority.
We compare our algorithms with these baselines which are implemented by their open-sourced codes all the environments described above.
5.3 COMPARISON RESULTS ON ALL ENVIRONMENTS
The common hyperparameters of all algorithms are set as the same. When experiment in the goal-reaching environments, we set the epoch to 30 and the batch size to 64. In the robotics control tasks, the epoch is set 100 and the batch size is 256. We do not MPI to parallelize the program on multiple workers. In order to ensure that all algorithms have the same amount of data during training, algorithm uses heads generate multiple trajectories each time, while other

---

Under review as a conference paper at ICLR 2022
1,0
Lo-
10
1.0
Success Rate
0.0
Success Rate:
0.8-
044
20
Gpach
(a) PoineDDa
(b) Reacher
(c) FetchReach
(d) SawyerReach
Success Rate
Success Rate
2.00
0.5
0.0
1.0
25
75
Epoch
Epoch
(c) SawyerDoor
(f) HandReach
gg) HandManipulatePen
BHER
CHER
HEREBP
Figure Performance for allenvironments: success rate (line) with standard deviation range (shaded
aeea) on all environments acorss 10 random seeds.
algorithms use Dummy to run environments in parallel. After each epoch, we perform 100 rollouts for evaluation, and calculate the average success rate of the 100 rollouts. In our algorithm, each head generates a trajectory during each rollout progress when evaluation.
Fig. 4 shows the comparison results of all algorithms. For each algorithm, to repeat the training and testing and plot the mean curve and the standard derivation confidence. It can be observed that our algorithm outperforms all other baselines. In some environments, the performance of HEREBP and CHER not as good as HER, which may be because the method of calculating priority in these two algorithms is not suitable for the current environment. For example, in SawyerDoor environment, the goal has no potential energy or rotational energy, and HEREBP algorithm can only kinetic energy as the priority of transition.
10 random seeds
For BHER, we fine tune different temperature coefficients in calculating the priority for each en- vironment in our algorithm, as shown in Appendix B This hyperparameter can slightly adjust the performance of our algorithm. We choose the best temperature coefficient for each environment for BHER algorithm.
5.4 ABLATION EXPERIMENTS RESULTS
We conducted ablation experiments confirm the effectiveness of counterintuitive prioritization. Section 4.2 illustrates that counterintuitive priority replay method (giving higher priority to smaller variances of Q-value) can improve the exploitation of algorithm. Here compare it with a variant of our BHER algorithm that only uses Bootstrapped DDPG and do not use counterintuitive prioriti- zation, referred to BHER w/o prioritization. In addition, also compared the way of prioritizing transitions based on higher variance of Q-value which we call BHER w/ higher variance first.
The result in Fig. 5 verifies the effectiveness of counterintuitive prioritization. It improves the ploitation of algorithm by prioritizing transitions with small variance of the Q-value, and further improves data efficiency.
8

---

Under review as a conference paper at ICLR 2022
1.0
Lo-
10-
1.0
Success Rate
0.0
Success Ratte
0.8-
0.4
04
20
Gpach
(a) PoineDDarrgeEny
(b) Reacher
(c) FetchReach
(d) SawyerReach
0.7
Success Rate
Success Rate
2.30
0.3
0.0
2.0
25
75
75
Epoch
Epoch
(c) SawyerDoor
(f) HandReach
(g) HandManipullteeen
BHER
BHER w/ higher variance first
BHER w/o prioritization
Figure 5: Ablation experiments: success (line) with standard deviation range (shaded area) all environments acorss 10 random seeds.
We also conducted an experimental comparison on the adjustment of the priority temperature co- efficient. Appendix B shows the results. It can be seen that different environments have different optimal priority temperature coefficients.
In addition, Section 4.2 illustrates that the variances of the hindsight transitions are smaller than the variances the original transitions, and we give higher priority in transitions with smaller vari- ance. In this way, hindsight transitions have a higher probability of being sampled by the algorithm. Intuitively, we will think about impact not using original data on the performance gorithm. To see this, we conduct ablation experiments to evaluate the influence of the usage of the original data. We show our ablation experimental results in Appendix [ It illustrates that in some algorithm. But in com-
our
simple environments, the usage of original data plex environments, original data is necessary. The ratio of the original data not the focus of our algorithm, just it 20% according the original HER algorithm.
has little effect
CONCLUSION
The main contributions of the BHER algorithm we proposed are follows: The way the bootstrapped principle to design the network has greatly improved HER algorithm's 's exploration in environment of sparse rewards. And when we update the network, we do not distinguish the source of the data, which greatly improves the sample efficiency. 2) We designed counterintuitive prioritized replay method to improve the exploitation of algorithms, which provides a new criterion for priority replay in RL. And our priority calculation method will not vary depending the envi- ronment, which makes our algorithm have better generalization. 3) combined the bootstrapped principle and counterintuitive prioritization, which makes excellent trade-off between exploration and exploitation in RL. 4) Whether in a simple goal-reaching environment or a difficult robotics controll environment, our algorithm has achieved ofte--of-the-ar preformance. 5) Our algorithm does not have any assumptions or requirements on the environment, and can easily extended to any goal-conditioned environment.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
We provide Reproducibility Statement below. For the algorithms, we have attached the main codes of BHER the submitted zipp supplementary material.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. arXiv preprint arXiv: /000:00797999 2017.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmual international conference on machine learning. pp. 41 41-48, 2009.
Henry Charlesworth and Giovanni Montana. Plangan: Model-based planning with sparse rewards and multiple goals, 2020.
João Crespo and Andreas Wichert. Reinforcement learning applied to games. SN Applied Sciences, 2(5):1-16, 2020.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35 (1):53-65, 2018.
Sam Michael Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In Proceedings of the 1 11th International Conference on Autonomous Agents and Multiagent Systems, pp. 433- 440. IFAAMAS, 2012.
Jeffrey L Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):7 1):71-99, 1993.
Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang. DHER: Hindsight experience replay for dynamic goals. In International Conference Learning Representations, 2019a. URL
net Loee
Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guide hindsight experience replay. Advances in Neural Information Processing Systems, 2019b.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. 2020.
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference on machine learning, pp- 1311-1320. PMLR, 2017.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Jing Li, Xinxin Shi, Jiehao Li, Xin Zhang, and Junzheng Wang. Random curiosity-driven explo- ration in deep reinforcement learning. Neurocomputing. 418: 418:139-188 139-147, 2020.
Timothy Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2019.
Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (1775))))599-333 2015.
10

---

Under review as a conference paper at ICLR 2022
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Ieml, volume 99, pp. 278-287, 1999.
Hai Nguyen and Hung La. Review of deep reinforcement learning for robot manipulation. In 2019 Third IEEE International Conference on Robotic Computing (IRC), pp. 590-595. IEEE, 2019.
Inseok Oh, Seungeun Rho, Sanghin Moon, Seongho Son, Hyoil Lee, and Jinyun Chung. Creating pro-level ai for real-time fighting game using deep reinforcement learning. IEEE Transactions on Games, 2021.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn, 2016.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervvsed prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Matthias Plappert, Marein Andrychowicz, Alex Ray. Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.
Mehdi Rahimi, Spencer Gibb, Yantao Shen, and Hung Manh La. A comparison of various ap- proaches to reinforcement learning algorithms for multi-robot box pushing. In International Con- ference Engineering Research and Applications, pp. 16-30. Springer, 2018.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on 2017.
thompson sampling. arXiv preprint
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In Proceedings of the 32nd International Conference on Machine Learning. PMLR, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising reinforce- ment learning agents. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 792-799, 2003.
Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv: 2014.
Daochen Zha, Kwei-Hemg Lai, Yuanpu Cao, Songyi Huang, Ruzhe Wei, Junyu Guo, and Xia Hu. Rlcard: A toolkit for reinforcement learning in card games. arXiv preprint arX:4:1910.04376, 2019.
Rui Zhao and Volker Tresp. Energy-based hindsight experience prioritization. arXiv preprint 1rX1v:1870.01363, 2018
Rui Zhao, Xudong Sun, and Volker Tresp. Maximum entropy-regularized multi-goal reinforcement learning, 2020.
Oleksii Zhelo, Jingwei Zhang. Lei Tai, Ming Liu, and Wolfram Burgard. Curiosity-driven explo- ration for mapless navigation with deep reinforcement learning. arXiv preprint 17.Xiv:8804.000 2018.
Yuanyang Zhu, Zhi Wang, Chunlin Chen, and Daoyi Dong. Rule-based reinforcement learning for efficient robot navigation with space reduction. JEEE/ASME Transactions on Mechatronics, 2021.
11

---

Under review as a conference paper at ICLR 2022
APPENDIX
A ENVIRONMENTS
Fig. 3(a) 3(b) 3 ( shows the goal-reaching environment we have used, including four environ- Reacher, FetchReach and SawyerReach. Point2DLargeEnn is point- based environment, in which the blue point aims to reach the green circle. The blue point can reach any position in the environment. The state and goal of this environment tors, which respectively represent the position of the blue point and the green circle, and the action also 2-dimensional vector, which represents the moving distance of the green circle each time. In Reacher, 2-DoF (degrees of freedom) robotic arm whose task is to reach a particular target in the field. Its state is an 11-dimensional vector, which contains the angle of the two joints and the position of the fingertip relative to the target, and goal is a 2-dimensional vector that represents the position of the target, and action is also a 2-dimensional vector, representing the movement of two joints. FetchReach and SawyerReach are two very similar environments. Both have 7-DoF robotic arm pushing a box until it reaches the target position. In FetchReach, robotics arm has a two-fingered parallel gripper. The action is a 3-dimensional vector representing the desired gripper' movement. The goal is 3-dimensional vector describing the position of the target and the achieved goal is the position of the gripper. In SawyerReach, the arm's end-effector (EE) is constrained to 2-dimensional rectangle parallel to a table and constrained to only move in the 2-dimensional plane. The action controls EE position through the use of a mocap. The state is the 2-dimensional position of the EE and the goal is an 2-dimensional position of the EE.
ments: Point2DLarged
both 2-dimensional vec-
Fig. 3()) 3(t) 3(g) shows the robotics controll environment we have used, including three environ- ments: SawyerDoor, HandReach and HandMaanppllteePeaRotttt In SawyerDoor, these is a robotics arm which aims open small cabinet door, initially shut closed, sitting on a table angle. The state is a 4-dimensional vector consisting of the Cartesian coordinates of the arm's 's end- effector and the door's angle. The action a 3-dimensional vector controlling the position of the end-effector. The goal is the desired angle at which the door is opened. Hand environments has 24- DoF anthropomorphic robotic hand, whose 20 joints can be can be controlled independently whereas the remaining ones are coupled joints (Plappert et al. 2018). So the action is 20-dimensional vec- tor containing the absolute position control for all non-coupled joints of the hand. In HandReach, the goal is 15-dimensional vector and contains the target Cartesian position of each fingertip of the hand and is considered achieved the mean distance between fingertips and their desired position is less than cm. In VandManiipulatePenRnzzaa
specified
a pen is placed on the palm of the hand. The task is to then manipulate the pen such that target pose is achieved. The goal is 7-dimensional vector including the target position in Cartesian coordinates and target rotation in quaternions. We use the rotate mode which includes the target rotation æ and y axes of the pen and without target rotation around the z axis or target position.
B OPTIMAL PRIORITY TEMPERATURE COEFFICIENTS
We evaluate the impact of different priority temperature coefficients on our BHER algorithm.
Table Priority Temperature Coefficients
Point2DLargeEny
Reacher
FetchReach
SawyerReach
9.0
9.0
9.0
7.0
HandMaaipullateeeRotttte
HandReach
SawyerDoor
7.0
7.0
1.0
12

---

Under review as a conference paper at ICLR 2022
1.0
Lo-
1.-
1.0
Success Rate
0.8
Success Rate
0.8-
D4
20
Gpach
(a) PoireDDa
(b) Reacher
(c) FetchReach
(d) SawyerReach
Success Rate
00-
Success Date
2.30
0.5-
0.4
0.3
0.0
2.0
25
75
Epoch
Epoch
(c) SawyerDoor
(f) HandReach
(g) HandManipullteeenn
BHER (T= 1.0)
BHER (T 3.0)
BHER = 5.0)
BHER (T 7.0)
BHER (T 9.0)
Figure 6: Performance of BHER with different priority temperature coefficient.
We experimented with five priority temperature coefficients of 51.0,,30,550,,7.0,,01, in all envi- ronments. Fig. 6 shows the results our comparative experiment. can be seen that different environments have different optimal priority temperature coefficients. We select an optimal priority temperature coefficient for each environment, shown in the Table
C ABLATION EXPERIMENT FOR ORIGINAL DATA
Section 4.2 shows that hindsight transitions have smaller variance, and learning small variance tran- sitions is more conducive to improving the exploitation of agent. In other words, is better to let the agent learn the hindsight transitions first.
Therefore, we design an ablation experiment to evaluate the influence of the original data on our algorithm. We modify algorithm to not the original data which we call BHER w/o original data. Fig. o shows that in some relatively simple environments, such as Point2DLargeEnv and FetchReach, using original data has little effect the performance our algorithm. However, in other more complex environments, original data will seriously affect the performance of our algorithm. The ratio of the original data the focus of our paper, we this ratio just like HER algorithm.
Success Rate
0.8
Success Rate
0.8
0.0
2.0
0.0
0.4
Epoch
Epoch
Epoch
Epoch
(a) Loine2DLargeEnn
(b) Reacher
(c) FetchReach
(d) SawyerReach
13

---

Under review as a conference paper at ICLR 2022
Surcess Patee
Surceess Ratte
0.3
0.4
2.0
0.0
3.0
(e) SawyerDoor
(I)
HandReach
(g) HandMManipulateenn
BHER
BHER w/o original data
Figure 6: Ablation experiments results for original data.
We also evaluate the intuitive priority replay (higher priority for higher variance) without using the original data which we call BHER w/o original data w/ higher variance first. We don't I evaluate this setting in all environments, because think these results shows that even under the setting that does not use the original data, the counterintuitive priority replay method has better performance. This also illustrates the effectiveness of our counterintuitive prioritization, which improves the exploitation agent.
already prove method. Fig.
Success Ratee
0.8
Success Rate
0.8
0.0
2.5
02
Epoch
Epoch
Epoch
Epech
(i) LoireEDa
(i) Reacher
(k) FetchReach
SawyerReach
BHER
BHER w/o original data
BHER w/o original data w/ higher variance first
Figure 7: Ablation experiments results for original data and higher variance first.
D TRADE-OFF OF EXPLORATION AND EXPLOITATION IN TOY ENVIRONMENT
We use Reacher (for details, see Appendix A as the toy environment to illustrate the trade-off between exploration and exploitation in our BHER algorithm. Fig. S(a) and 8bbsshow
shows that agent
epoch=5
epoch=10
epoch=30
0.1
0.1
0.0
0.0
0.0
0.2
0.2
0.2
-0.2
-0.1
0.0
0.2
-0.2
-0.1
0.0
0.1
0.2
-0.2
-0.1
0.0
0.1
0.2
(a) HER
14

---

Under review as a conference paper at ICLR 2022
epoch=5
epoch=10
epoch=30
0.2
0.2
0.2
0.0
0.0
0.0
0.2
0.2
0.2
0.0
0.1
(b) BHER w/o prioritization
epoch=
epoch=10
epoch=30
0.2
0.2
0.2
0.0
0.0
0.0
0.2
0.2
0.2
-0.2
-0.1
0.0
-0.2
0.0
-0.2
-0.1
0.0
0.1
(c) BHER
Successful goal
*
Failed goal
Figure 7: Distribution of the goals in Reacher. (BHER w/o prioritization only uses bootstrap princi- ple on the basis of HER, but does not use any priority replay)
can explore more goals in shorter period of time. This shows that bootstrap principle has improved the agent's exploration. Fig. ((C) that we can make the agent preferentially select more valuable goals to learn by using counterintuitive prioritization. This effectively improves the exploitation of agent, which thus learns the optimal strategy faster.
15


Reviews:
[{"review_id": "tUPED9JTAJ_", "reviewer": "Reviewer_nX4W", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Quality\n* #### Strengths\n    * The authors seem to have picked up on something interesting in paying attention to how transitions are sampled during the course of HER. Empirically, their approach seems to provide reasonable to modest gains across various continuous control tasks. \n* #### Weaknesses\n    * ##### Major\n        * The authors have some fundamental issues and misconceptions about the BootstrappedDQN (BootDQN) algorithm. Section 2 mentions that BootDQN addresses exploration by running multiple behavior policies in the environment; if so, then it would be true that the method would have no reason to solve sparse reward tasks a priori as described by the authors. However, the reason for BootDQN's success in exploration is because of posterior sampling [8] and the fact that the ensemble is an approximate posterior over the optimal action-value function of the MDP, conditioned on all agent interactions observed thus far. It is precisely this principle that allows the algorithm to address sparse-reward tasks [2]. The authors' description of Thompson sampling in BootDQN as an ad-hoc heuristic seems incorrect given the rigorous theoretical guarantees that accompany randomized least-squares value-iteration algorithms [3]. The authors are also confused about the use of replay buffers in BootDQN, claiming that data from each head is held separate. There is exactly one replay buffer used in BootDQN and Bernoulli masks are sampled and stored with each transition for implementing the statistical bootstrap. Overall, the connection the authors draw with BootDQN in this work seem rather disingenuous; the proposed algorithm is simply applying ensembles of actor-critic pairs with no connection to the statistical bootstrap, unlike BootDQN. Renaming the algorithm and rephrasing the contribution seem appropriate.\n        * A more critical issue concerning BootDQN and the proposed BHER algorithm is that the former is a purely value-based RL algorithm that maintains a posterior distribution over the optimal action-value function. In contrast, the latter is an off-policy actor-critic algorithm where, naturally, the critic is meant to be an estimate of the action-value function induced by the actor policy. While the empirical results of this paper confirm empirical benefits of this ensembling heuristic (the ablation in Figure 5 shows that this is responsible for most of the BHER performance), the authors have offered no real justification for this ensemble actor-critic algorithm. What is the point of representing epistemic uncertainty over the actor and critic in this manner? \n        * I don't find the so-called counterintuitive prioritization to be counterintuitive at all. It seems natural that hindsight transitions will serve the agent well only when there is little uncertainty in the associated optimal behavior under the relabeled goal. Can the authors explain why sampling based on higher variance seems to still maintain reasonable performance in three of seven environments shown in Figure 5?\n\t* ##### Minor\n\t\t* In the first paragraph of Section 4.2, the authors describe an instance of Prioritized Experience Replay (PER) [6] where the variance of the critic ensemble is used to prioritize transitions sampled from the replay buffer. They seem to confuse this technique with methods for intrinsic motivation [7] based on curiosity and Random Network Distillation.\n\n### Clarity\n* #### Strengths\n    * The authors provide ample details about their experimental setup for reproducibility of their results.\n* #### Weaknesses\n    * ##### Major\n        * Overall, the paper is not well written. There are numerous grammatical errors throughout, genuinely too many for me to sensibly list them all out here. Oftentimes, these errors are missing articles (for example, \"it uses successful trajectories generated by agent as expert demonstrates\") or incorrect phrases (\"on the contrast\", \"we inference all the Q-values\"). Normally, I wouldn't bother nitpicking at a small handful of these, but there are too many throughout the entire body for what may end up being a published conference paper.\n\t* ##### Minor\n\t\t* The authors should remove the phrase \"importance sampling\" that is used twice in the paper to, in my reading, talk about the importance of sampled goals, rather than the Monte-Carlo technique of the same name.\n\n### Originality\n* #### Strengths\n    * The authors demonstrate a good instinct in examining how other techniques used in deep reinforcement learning might further improve the efficacy of HER.\n* #### Weaknesses\n    * ##### Major\n        * Fundamentally, this paper rests on the idea of using ensembles and prioritized experience replay together, neither of which is new to (deep) reinforcement learning [1,4,5,9]. Though I do not know of any prior work that has explored this specific combination, it would not surprise me if such prior work already exists.\n        * More importantly, there are other options for leveraging such ensembles that have not been addressed in this work [1,4]. Similarly, the authors only consider variance-based prioritization schemes, rather than the traditional prioritization based on TD-error or any recent variants of PER. Demonstrating that the authors' specific choices in the proposed approach are better than these existing approaches to ensembling and PER would dramatically improve what so far seems to be a rather incremental algorithm.\n\t* ##### Minor\n\t\t* While it is appropriate for the related work section to focus on HER, it should also acknowledge the two fundamental innovations of this paper (ensembles and prioritization schemes) and provide an overview of related work for these areas as well.\n\n### Significance\n* #### Strengths\n    * The only positive I can glean from this paper are the empirical results which seem to support the use of ensembling in actor-critic algorithms. Figure 5 shows a marginal drop in performance when the proposed counterintuitive prioritization scheme is not used. That said, it is not clear that this paper advances our understanding of ensembling in deep RL any more than prior work.\n* #### Weaknesses\n    * ##### Major\n        * Given the lack of comparisons mentioned above, it's difficult to assess how impactful the proposed approach will be. With the breadth of existing work on the topic, I'm unconvinced that this will add any novel insights into how practitioners use ensemble methods in reinforcement learning. The prioritization scheme, while slightly interesting on the surface, doesn't seem to be a critical ingredient to the proposed algorithm's success based on the ablation studies shown. \n        * I don't believe any of the experiments have shown results for regular DDPG without the use of HER. Having this baseline in place is important as it communicates the extent to which HER is even necessary for achieving a reasonable level of performance in each of the examined environments.\n\t* ##### Minor\n\t\t* \n\n# References\n1. Lee, Kimin, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. \"Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning.\" In International Conference on Machine Learning, pp. 6131-6141. PMLR, 2021.\n2. Osband, Ian, and Benjamin Van Roy. \"Why is posterior sampling better than optimism for reinforcement learning?.\" In International conference on machine learning, pp. 2701-2710. PMLR, 2017.\n3. Osband, Ian, Benjamin Van Roy, Daniel J. Russo, and Zheng Wen. \"Deep Exploration via Randomized Value Functions.\" J. Mach. Learn. Res. 20, no. 124 (2019): 1-62.\n4. Peer, Oren, Chen Tessler, Nadav Merlis, and Ron Meir. \"Ensemble Bootstrapping for Q-Learning.\" arXiv preprint arXiv:2103.00445 (2021).\n5. Saphal, Rohan, Balaraman Ravindran, Dheevatsa Mudigere, Sasikant Avancha, and Bharat Kaul. \"SEERL: Sample Efficient Ensemble Reinforcement Learning.\" In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1100-1108. 2021.\n6. Schaul, Tom, John Quan, Ioannis Antonoglou, and David Silver. \"Prioritized Experience Replay.\" In ICLR, 2016.\n7. Singh, Satinder, Richard L. Lewis, Andrew G. Barto, and Jonathan Sorg. \"Intrinsically motivated reinforcement learning: An evolutionary perspective.\" IEEE Transactions on Autonomous Mental Development 2, no. 2 (2010): 70-82.\n8. Strens, Malcolm. \"A Bayesian framework for reinforcement learning.\" In ICML, vol. 2000, pp. 943-950. 2000.\n9. Wiering, Marco A., and Hado Van Hasselt. \"Ensemble algorithms in reinforcement learning.\" IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 38, no. 4 (2008): 930-936.", "summary_of_the_paper": "# Summary & Contributions\n* The authors call attention to sparse-reward, goal-based tasks where Hindsight Experience Replay, through its provision of relabeled goals for otherwise failed experiences facilitates efficient learning.\n* To further improve upon HER, the paper focuses on two algorithmic innovations: (1) maintaining multiple actor-critic pairs inspired by BootstrappedDQN and (2) applying a variant of prioritized experience replay that draws samples from the replay buffer inversely proportional to the current Q-value variance under the critic ensemble.\n* The authors support their approach with comparative experiments to other variants of HER as well as ablation studies of their own proposed method.", "summary_of_the_review": "# Final Remarks\n* The authors summarize their contributions in Section 6. While their fifth point is true, I would respond to the other points as (1) there is actually no use or examination of the statistical bootstrap in this work as the ensembles rely solely on random initializations, BootDQN also does not distinguish data sources, and there are other principled means of addressing exploration through ensembles such as UCB (2) while the proposed prioritization is new, there are many others that also do not depend on the environment (TD-error) which have not been assesssed (3) the combination is largely uninteresting based on ablation studies in this work which show near negligible impact of the proposed prioritization scheme and (4) I don't believe DDPG (with or without HER) holds state-of-the-art for these Mujoco domains; I would suspect that lies with either Soft Actor Critic or TD3.\n\nTaken together, I don't believe this paper is ready for publication at this time.\n\n\n\n======= Post Rebuttal =======\n\nI thank the authors for their response but the justifications for the utility of representing epistemic uncertainty and lack of baselines are shallow;  it's clear that substantial revisions are needed before the submission is ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dADDIHaVC45", "reviewer": "Reviewer_PtA1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## High level comments\n### Strengths\n* Incorporating the idea of multiple heads from bootstrapped DQN for goal-conditioned RL (GCRL) and then also using those multiple heads to also prioritize replay is a sensible extension of the idea to GCRL.\n* The counterintuitive prioritization needed when using the variance of the Q-values for prioritization is an interesting effect that is evaluated empirically with a possible explanation that it balances exploration and exploitation. This empirical example comparing the variance in Q-values in HER vs the original samples is helpful to see why the lower variance samples should be helpful.\n\n### Weaknesses:\n* The paper focuses on this counterintuitive prioritization in the experiments. This focus is understandable since that is the novel aspect introduced in the paper. However, it is unclear whether the improved performance by BHER is due to the deep exploration due to multiple heads or this counterintuitive prioritization. The ablation that removes the prioritization seems to still perform almost as well, suggesting that most of the improvement is due to the improved exploration.\n\nThe above reason is why I am unconvinced about the experimental validation. Perhaps an additional ablation of BHER without the deep exploration would highlight the efficacy of the counterintuitive prioritization.\n\nAlso see below for questions about other comparisons.\n\n## Detailed Comments\n\n* Since the bootstrapped heads improve the exploration of the agent, I wonder how this exploration stacks up to other exploration techniques in this domain. For example, would it be better than the directed exploration induced by [1]? I understand that this paper is very recent, but comparison to some other exploration enhancing mechanisms would be good.\n* The approach of using multiple heads and the variance in Q-values for picking samples seems related to [2]. Could authors clarify the difference, and why this method was not compared to in baselines?\n* Section 4.2, which goes over the counterintuitive prioritization and why it is needed, is enlightening but also lacking. The comparison of the variance when using HER and when considering just samples from the environment show that HER samples have lower variance. The paper points out that this discrepancy could be due to the higher proportion of successful trajectories due to hindsight. This analysis is reasonable. But then the authors suggest that prioritizing higher variance samples would be similar to sampling without hindsight. My question is: What would happen if you only sampled higher variance samples from the relabeled experience, and sampled uniformly from the  true experience? Wouldn't that ensure that original samples are not sampled out of proportion?\n* Secondly, by prioritizing lower variance samples, does BHER just sample more from relabeled experience?\n* In the experiments, the Fetch robot task used for evaluation is just the Reach environment, similar to the Reacher task. Evaluation on some actual manipulation tasks such as Push or Pick and Place would be a better evaluation in this domain. This is a small nitpick, the other experiments deal with more challenging tasks. It just seems like the FetchReach task is not bringing much to the table, except nominally adding a domain.\n* As mentioned earlier, in the ablations, BHER without deep exploration would be a good one to have. And in those figures, the baseline DDQN should also be plotted, to show the improvement.\n\n## Minor Comments\n* More preciseness in the problem setup (Section 3.1) would be appreciated. As it stands, this setup would be hard for someone not familiar with reinforcement learning literature to parse.\n* Typo: Section 4.1, line 2\n\n\n[1] - Adversarial Intrinsic Motivation for Reinforcement Learning, Durugkar et al., 2021\n[2] - Automatic Curriculum Learning through Value Disagreement , Zhang et al., 2020", "summary_of_the_paper": "This paper looks at the problem of goal-conditioned reinforcement learning, and focuses specifically on the problem of prioritizing replay in conjunction with hindsight experience replay (HER).\nIt does so by using the idea of multiple heads in the actor-critic policy and critic networks, inspired by bootstrapped DQN. These different heads are initialized separately, and are also used to induce deep exploration in the style of bootstrapped DQN.\n\nThe main idea is to use the variance in the q-values across the heads for different transitions when prioritizing transitions to use for learning. The algorithm, termed counterintuitive prioritization, prioritizes transitions with lower variance. An example on the reacher environment is used to illustrate why this counterintuitive prioritization assists learning, and hypothesizes that it is based on a tradeoff between exploration and exploitation.\n\nExperiments are performed in two simpler environments (Point 2D and Reacher) and 5 environments simulating different robots (Fetch, Sawyer, and Hand). They compare with two algorithms that prioritize transitions for replay based on some other criteria, and show that the counterintuitive prioritization along with deep exploration due to multiple heads leads to faster improvement in success rate of reaching goals in terms of interactions with the environment.\n\nAblations try to tease apart whether this improvement is due to the improved exploration due to multiple heads or the better exploitation due to the counterintuitive prioritization.", "summary_of_the_review": "The idea of enhancing exploration in goal-conditioned RL using bootstrapped DQN is reasonable. Also using the multiple heads to prioritize samples for replay is a good extension, which leads to the investigation in the paper.\nThe idea of counterintuitive prioritization is investigated using an empirical investigation that gives a good insight into the reasons why this approach can be expected to work.\n\nHowever, the paper does not sufficiently tease apart the effect of prioritization versus the enhanced exploration.\nThe comparison to other techniques that prioritize their samples does not take into account the enhanced exploration either.\nAnd the paper does not compare to other techniques that enhance exploration.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "M943SSsby_h", "reviewer": "Reviewer_BnLV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Significance**: Because the underlying ideas (bootstrap DQN, hindsight relabeling, and prioritized experience relay) are well known, the main criterion for evaluation is the empirical results. The results are quite strong. However, the complexity of the method and its sensitivity to hyperparameters make me a bit skeptical whether the paper will make a lasting impact on the field.\n\n**Correctness**: I had two minor comments about the correctness of the paper:\n\n* Much of the discussion of the paper discussed how hindsight relabeling helps with exploration. I don't think this is correct: exploration is a problem of data collection, while hindsight relabeling is a tool for performing updates using previously-collected data. Hindsight relabeling doesn't (directly) say anything about how the data should be collected.\n* I didn't see bootstrap DQN as a baseline. As prior work has found that using multiple Q functions significantly improves performance (e.g., [1, 4]), I think this is an important baseline to add.\n\n**Originality**: The proposed method is a new combination of old ingredients.\n\n**Clarity**: The paper is mostly clear.\n* I found the discussion of prioritization hard to follow. For example, I didn't understand what \"higher variance first\" means in Fig 5.\n* The related work section lacks structure. I'd recommend organizing the related work thematically, and making sure that the commonality is clear.\n\n**Minor comments**:\n\n* \"non-negative reward\" -> \"non-zero reward\" (The agent could be getting negative dense rewards)\n* \"often fail to perform well in sparse rewards environments\" -- Cite.\n* \"A brilliant ...\" -- Cite [2], which predates the HER paper by ~25 years.\n* \"uniformly sampling one of them, as HER did\" -- The HER paper actually tried many different sampling strategies, including strategies that sampled multiple goals. Check out the appendix of the HER paper.\n* When discussing prior work in the introduction and in Sec. 4.1, I'd recommend citing [1, 4] and other recent papers that use multiple Q functions.\n* \"Both these methods don't work well ...\" -- Cite.\n* \"Goal-Conditioned Supervised Learning\" -- Cite [3], too.\n* \"the rewards in goal-conditioned environments are sparse\" -- Actually define what the reward function is in an equation.\n* \"subtly applies\" -> \"applies\". The application isn't subtle.\n* I found Lines 5 and 8 of Alg 1 unclear.\n* I found Fig 2 hard to interpret. Are there alternative ways of presenting the same data that more succinctly convey the same message?\n* \"sparse and non-negative ... reward of -1\" -- Contradiction: the reward isn't non-negative if it is -1 for some states and actions.\n* \"acorss\" -> \"across\"\n\n\n[1] Lee, Kimin, et al. \"Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Kaelbling, Leslie Pack. \"Learning to achieve goals.\" IJCAI. 1993.\n\n[3] Ding, Yiming, et al. \"Goal-conditioned imitation learning.\" arXiv preprint arXiv:1906.05838 (2019).\n\n[4] Chen, Xinyue, et al. \"Randomized ensembled double q-learning: Learning fast without a model.\" arXiv preprint arXiv:2101.05982 (2021).", "summary_of_the_paper": "This paper proposes a method for goal-conditioned RL that combines three ideas from prior work: bootstrap DQN, hindsight relabeling, and prioritized experience relay. Experiments show that the proposed method outperforms variants of hindsight relabeling.", "summary_of_the_review": "While the empirical results of the paper are quite strong, the paper doesn't compare to a number of more recent methods that also use ensembles of Q functions. Moreover, the method is rather complex, and it doesn't provide a guiding explanation for why this particular combination of ideas is good. \n\n------------------------\n**After rebuttal**: Thanks to the authors for responding to some of the points raised in the review, and for running additional experiments. My two main concerns with the paper are (1) whether most of the empirical benefits are coming from the ensemble and (2) the clarity of the writing. While I am not convinced that the revised version addressed these concerns, I would encourage the authors to continue revising the paper and submit to a future conference. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EvVZunzD-S5", "reviewer": "Reviewer_yX9d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- the idea is simple and easy to implement;\n- the BHER can achieve a significant improvement across different tasks;\n\nweakness:\n- This paper claims that BHER can achieve a trade-off between exploration and exploitation about the hindsight experience, but there are not strong enough evidence (empirical or theoretical) to support this claim. Though the experiments in Appendix.D try to show this trade-off, the results can't convince me, Figure.8 only shows that the multiple head principle can have a better exploration about the goal than HER, and the BHER(multiple head principle with prioritization) have the best exploration about the goal, it doesn't show BHER scarifices the exploration so as to improve the exploitation;\n- Most of the performance improvement is due to multiple head principle, the counterintuitive prioritization can only slightly improve the performance in most environments. So the reason behind the counterintuitive prioritization should be further investigated. \n\nsome questions and typos:\n1. the caption of the last figure in Appendix should be 'Figure 8' instead os 'Figure 7';\n2. About the experiment setting, this paper first said that the reward was sparse and non-negative, however, this is contrast to the setting that the agent gets a reward of 0 only when it reaches desired goal, otherwise it receives a reward of -1. \n", "summary_of_the_paper": "This paper aims to improve learning efficiency for Hindsight Experience Replay(HER). Instead of learning with a fixed proportion of fake and original data like HER does, this paper proposes to adopt the multiple head structure used in Bootstrapped DQN and then utilize the uncertainty measured by the variance of multiple estimated Q-values so as to weight the prioritization of each transition. Specifically, the proposed BHER enhances the importance of data samples with lower uncertainty, and thus achieves a trade-off between exploration and exploitation, resulting in a higher sample efficiency.", "summary_of_the_review": "The paper is built on the observation that  achieving different goals may need different pseduo success trajectories which are unfortunately not provided by the naive HER algorithm. The paper then provides a straightforward solution to this by adding a Boostrapped DQN onto HER so as to allow it explore deeper and be able to evaluate the goodness of a pseduo trajectory for different goals.  Although this idea is shown work well compared to the naive HER, I think that further more principled solution may be needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and active participation in the discussion. The reviewers unanimously agree that this submission has significant issues, including comparison to baselines/ablations [BnLV,yX9d,PtA1], clarity [BnLV], justification of the method [nX4W]. Thus, I am recommending rejection of this paper.
Decision: Reject