Bootstrapped Hindsight Experience replay with Counterintuitive Prioritization

1 INTRODUCTION . Deep reinforcement learning ( DRL ) has gained significant achievements in solving games ( Zha et al. , 2019 ; Crespo & Wichert , 2020 ; Oh et al. , 2021 ) and robotics problems ( Rahimi et al. , 2018 ; Nguyen & La , 2019 ; Zhu et al. , 2021 ) . Despite these successes , sparse rewards problems are still challenging . In sparse reward tasks , the agent gets a non-negative reward only when it achieves some goal . As a consequence , it is very difficult for an agent to see successful trajectories . Indeed , many state-of-the-art RL algorithms , such as TRPO ( Schulman et al. , 2015 ) , PPO ( Schulman et al. , 2017 ) , DQN ( Mnih et al. , 2015 ) , and DDPG ( Lillicrap et al. , 2019 ) , etc. , often fail to perform well in sparse rewards environments . To solve sparse rewards problems , there have been rich literature devoted to reward engineering . Potential Based Reward Shaping ( PBRS ) ( Ng et al. , 1999 ) uses difference between two potential functions to design a shaping function , where the potential function comes from the expertise knowledge and evaluates the value of state . Potential Based state-action Advice ( PBA ) ( Wiewiora et al. , 2003 ) extend PBRS to use potential function to evaluates the value of state-action pair . Dynamic Potential Based ( DPB ) ( Devlin & Kudenko , 2012 ) approach takes time as one of the inputs of the potential function . However , reward engineering usually requires domain-specific knowledge and it often suffers from sub-optimal performance . A brilliant idea for solving goal-conditioned tasks is using hindsight experiences . HER ( Andrychowicz et al. , 2017 ) , which replaces the desired goal in failed experiences with some achieved state in the failed trajectory to pretend that the agent obtains a positive reward , has been demonstrated its effectiveness in a wide range of goal-conditioned appli- cations . However , HER has its own limitations that when relabeling the goal in one failed trajectory , there exists many choices of the pseudo goals that any achieved state can be selected . Generally , uniformly sampling one of them , as HER did , is not the most effective way from the perspective of importance sampling . In this paper , we focus on improving the data efficiency in HER via the following two directions : 1 ) inspired by Bootstrapped DQN ( Osband et al. , 2016 ) , we propose a bootstrapped version of HER that employs multiple Q-value heads to increase the diversity in choosing the pseudo goals for deep exploration . In this paper , we use DDPG as the basic RL algorithm , while the proposed techniques can be integrated by any off-the-shelf RL algorithms ; 2 ) in addition to the benefit on exploration by using the bootstrapped Q-learning , we further explicitly leverage the uncertainty among the multiple heads by sampling goals conditioning on the variance of the estimated Q-values . Many previous approaches have shown that data with higher value uncertainty can promote exploration , and maximizing the uncertainty by incorporating a bonus term in the reward has been widely considered , such as bandit algorithms ( Lattimore & Szepesvári , 2020 ) . To our surprise , this is totally not the case in HER . On the contrast , sampling goals with lower value uncertainty first is demonstrated to be significantly better than uniform sampling , and sampling goals with high uncertainty in prior deteriorates the performance of HER a lot . We thus call this empirical fact as counterintuitive prioritization . Indeed , this can be explained by the trade-off between exploration and exploitation when importing HER , bootstrapped Q-learning and importance sampling . We refer our algorithm to as the Bootstrapped HER with counterintuitive prioritization ( BHER ) . We experiment with a number of control tasks with continuous action space . The experimental results verify the superiority of BHER over many state-of-the-art baseline methods . 2 RELATED WORK . Sparse rewards in goal-conditioned environment is one of the major challenges of RL at present . The earlier work is UVF ( Schaul et al. , 2015 ) algorithm , which uses state and goal together as the conditions for agent to take action . Bootstrapped DQN ( Osband et al. , 2016 ) uses the bootstrap principle to expand network structure . It uses multiple randomized value functions to estimate the Q-value of the state-action pair . This is equivalent to agent having multiple policies to interact with the environment , which effectively enhances agent ’ s exploration . But these methods don ’ t work well in the environment of sparse rewards . Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) algorithm recalculates reward by randomly replacing the desired goal in the failed transition with the achieved goal . In this way , pseudo successful data can be obtained , which can improve data efficiency . HER algorithm better solves the problem of sparse rewards . Since then , there have been many studies related to HER algorithm . Dynamic Hindsight Experience Replay ( DHER ) ( Fang et al. , 2019a ) utilizes the HER algorithm to process dynamic goals in the environment . Competitive Experience Replay ( CER ) ( Liu et al. , 2019 ) uses the competition between two agents to obtain better exploration . Goal-Conditioned Supervised Learning ( GCSL ) ( Ghosh et al. , 2020 ) is a kind of imitation learning . It uses successful trajectories generated by agent as expert demonstrates , and use supervised learning to optimize a lower bound on the goal-oriented RL objective . PlanGAN ( Charlesworth & Montana , 2020 ) is a model-based RL algorithm that uses GANs ( Creswell et al. , 2018 ) to learn the model of the environment and make planning . But this method requires a huge amount of computation to select a single action . Recently , researchers have realized that not all data have the same value to learn ( Schulman et al. , 2017 ) . More of the work is to further improve data efficiency by adding priority replay on HER algorithm . Energy-Based HER ( Zhao & Tresp , 2018 ) calculates the kinetic energy , potential energy and rotational energy as the total energy of the achieved goal according to the energy principle in physics . Then it gives higher priority to transitions with larger energy . Maximum Entropybased Prioritization ( MEP ) ( Zhao et al. , 2020 ) calculates the entropy of a trajectory which is used as priority . Curriculum-guided HER ( CHER ) ( Fang et al. , 2019b ) calculates the proximity and diversity between the achieved goal and the desired goal by designing two functions , which are added together with a weight to balance exploration and exploitation . These methods can be regarded as a form of curriculum learning ( Elman , 1993 ; Bengio et al. , 2009 ; Zaremba & Sutskever , 2014 ; Graves et al. , 2017 ) . But these algorithms either make some assumptions about the environment or design some specific functions to calculate the priority . Unlike the above works , our BHER algorithm does not have any requirements on the environment when calculating the priority , and it is easy to generalize to other goal-conditioned environments . 3 PRELIMINARIES . In this section , we introduce the preliminaries , such as the challenge of goal-conditioned environment , HER ( Andrychowicz et al. , 2017 ) and Bootstrapped DQN ( Osband et al. , 2016 ) which motivate us to propose our BHER algorithm . 3.1 GOAL-CONDITIONED ENVIRONMENT . In RL , the problem considered is usually composed of an agent and an environment . At time step t , the agent takes action at conditioned on the state st of the environment . After the environment receives the action at , it transfers to the state st+1 and feeds back the reward rt . The agent learns a policy π ( a|s ) to maximize expected discounted return : G = Eτ [ T−1∑ t=0 γtrt ] , ( 1 ) where τ is a trajectory ( s0 , a0 , r0 , ... , sT−1 , aT−1 , rT−1 ) and γ is the discount factor . In goal-conditioned environment , the agent takes an action a by taking an additional input , i.e. , a goal g , and the policy is hence a function of ( s , g ) that a ∼ π ( ·|s , g ) . Generally , the rewards in goal-conditioned environments are sparse , and only when the agent reaches the goal , it gets a nonnegative feedback . Therefore , it is difficult for the agent to explore successful trajectories when its policy starts from scratch . 3.2 HER AND BOOTSTRAPPED DQN . HER replaces the desired goal g in the failed transition with some sampled achieved state g′ in the current trajectory , and then recalculates the reward based on the pseudo goal g′ . In this way , the agent can see successful trajectories much more frequently , even if the goal is fake . It has been demonstrated in rich literature that HER is very effective for solving goal-conditioned tasks and it can be adopted by any off-policy algorithms , such as DQN ( Mnih et al. , 2015 ) , DDPG ( Lillicrap et al. , 2019 ) , TD3 ( Fujimoto et al. , 2018 ) , etc . Bootstrapped DQN designs multiple value function heads which are initialized randomly in the network structure of the DQN algorithm , and all heads share the underlying network . Each time agent interacts with the environment , one of the value function heads is randomly selected as policy to generate a trajectory . Each value function head is updated according to its own target network for temporally extended exploration , and each value function head only uses data generated by itself for updates . Bootstrapped DQN subtly applies the Thompson sampling ( Russo et al. , 2017 ) heuristic to RL which allows for deep exploration . 4 METHODOLOGY . In this section , we introduce the bootstrapped version of HER with multiple Q-value heads and the counterintuitive prioritization . 4.1 BOOTSTRAPPED HER . We use DDPG as the basic RL algorithm . In goal-conditioned tasks , the policy and Q-value take both the state and goal as input . A bootstrapped version of DDPG is dipicted in Fig . 1 , where both the actor network and critic network are implemented with multiple heads . All critic heads share an underlying embedding network , similar to what was used in Bootstrapped DQN ( Osband et al. , 2016 ) , and actor heads share an underlying network for feature extraction as well . Each actor head is updated according to its corresponding critic head , for which its pseudo goals are sampled independently and it is updated by according to its own recomputed rewards . As has been demonstrated in Bootstrapped DQN , random initialization of the parameters of each head is sufficient to generate diversity for deep exploration . This is also verified in our experiments for BHER . In this way , we can benefit a lot in terms of exploration with the cost of increasing only a small amount of the network parameters from multiple heads , since they share a large proportion of parameters in the embedding network . In Appendix D , we provide an illustration of the distribution of goals the agent can arrive in a toy environment by using multiple heads . Compared to the standard HER , we can verify that the bootstrapped version of HER is much more efficient in terms of exploration .

Summary: This paper looks at the problem of goal-conditioned reinforcement learning, and focuses specifically on the problem of prioritizing replay in conjunction with hindsight experience replay (HER). It does so by using the idea of multiple heads in the actor-critic policy and critic networks, inspired by bootstrapped DQN. These different heads are initialized separately, and are also used to induce deep exploration in the style of bootstrapped DQN.  The main idea is to use the variance in the q-values across the heads for different transitions when prioritizing transitions to use for learning. The algorithm, termed counterintuitive prioritization, prioritizes transitions with lower variance. An example on the reacher environment is used to illustrate why this counterintuitive prioritization assists learning, and hypothesizes that it is based on a tradeoff between exploration and exploitation.  Experiments are performed in two simpler environments (Point 2D and Reacher) and 5 environments simulating different robots (Fetch, Sawyer, and Hand). They compare with two algorithms that prioritize transitions for replay based on some other criteria, and show that the counterintuitive prioritization along with deep exploration due to multiple heads leads to faster improvement in success rate of reaching goals in terms of interactions with the environment.  Ablations try to tease apart whether this improvement is due to the improved exploration due to multiple heads or the better exploitation due to the counterintuitive prioritization.